Softgic
Senior AI Quality Engineer (LLM Evaluation & Automation) 1754
Remote AI Quality Engineering role with clear candidate location fit.
PostedJul 8, 2026
Eligible countries1 accepted country
Seniority signalSenior
Work settingRemote
Accepted candidate locations
USA
Role overview
Senior AI Quality Engineer (LLM Evaluation & Automation) 1754
Requirements and responsibilities
Readable role content extracted into sections for faster review.
Details
- Build and maintain the MVP eval harness: golden tasks, exception tasks, scorecard metrics, and regression packs.
- Wire evals into CI so quality regressions fail builds and releases.
- Define and maintain release-gate thresholds with Product and the Tech Lead.
- Lay the path for later adversarial and drift-testing expansion without overbuilding MVP scope.
- Experience evaluating ML, LLM, or non-deterministic systems.
- Strong test and benchmark design capability.
- Comfort working with noisy metrics, thresholds, and probabilistic behavior.
- Good scripting and automation skills.
- Uses AI to generate candidate eval cases and failure hypotheses, but never confuses generated tests with validated quality.
- Approaches AI quality as an operating system, not a QA afterthought.
- The first reference agent has a published scorecard and gated eval path. • Golden and exception tests run automatically. • The team can explain what “good enough to ship” means in measurable terms.
Similar roles
Keep a backup shortlist.
Stack
Use these tags to compare similar remote roles.
Location eligibility
Candidates should apply only when their profile country is listed here.
Your profileCountry not setSign in to check your country against this role.
Hiring flow
WithMira shows the role, then sends candidates to the company application.
1Check role fit, stack, and location eligibility in WithMira.
2Open the company application page from the tracked apply link.
3Save the role or subscribe for similar opportunities before leaving.