Sentry
Senior Software Engineer, AI Evals
Vaga remota de Engineering com fit claro de localização do candidato.
Publicada28 de jan. de 2026
Países elegíveis1 país aceito
Sinal de senioridadeSenior
Modelo de trabalhoRemoto
Locais aceitos para candidatos
Estados Unidos
Resumo da vaga
Senior Software Engineer, AI Evals
Requisitos e responsabilidades
Conteúdo da vaga extraído em seções para revisão mais rápida.
In this role you will
- Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
- Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
- Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
- Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
- Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring
You’ll love this job if you
- Care deeply about correctness, rigor, and measurement in AI systems
- Enjoy turning fuzzy product goals and model behavior into concrete tests and metrics
- Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team
- Thrive in cross-functional environments and enjoy influencing model design through better evaluation
Qualifications
- Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
- Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
- Comfort writing production-quality code (we use Python and TypeScript)
- Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
- Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)
- Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer tools
Vagas similares
Mantenha uma lista reserva.
Stack
Use estas tags para comparar vagas remotas similares.
Elegibilidade de localização
Candidatos devem aplicar apenas quando o país do perfil estiver listado aqui.
Seu perfilPaís não definidoEntre para comparar seu país com esta vaga.
Fluxo de contratação
O WithMira mostra a vaga e depois envia candidatos para a aplicação da empresa.
1Confira fit da vaga, stack e elegibilidade de localização no WithMira.
2Abra a página de aplicação da empresa pelo link rastreado.
3Salve a vaga ou assine oportunidades similares antes de sair.