Ryz Labs
AI QA Analyst
Vaga remota de AI QA Analyst com fit claro de localização do candidato.
Publicada19 de jun. de 2026
Países elegíveis24 países aceitos
Sinal de senioridadeMiddle
Modelo de trabalhoRemoto
Locais aceitos para candidatos
Resumo da vaga
AI QA Analyst
Requisitos e responsabilidades
Conteúdo da vaga extraído em seções para revisão mais rápida.
Qualifications:
- Ability to spot subtle logical inconsistencies or factual inaccuracies in AI-generated content.
- Experience managing, cleaning, or labeling datasets for machine learning or NLP projects.
- Familiarity with the AI lifecycle, Prompt Engineering, and tools like Google Sheets/Excel (SQL or Python is a significant plus).
Key Responsibilities:
- Review and annotate complex conversation traces to rate response quality based on metrics such as helpfulness, honesty, and harmlessness (HHH).
- Build and maintain high-quality "Golden Datasets" and benchmarks to stress-test the model across various domains and edge cases.
- Conduct pre-deployment testing and A/B model comparisons to identify performance regressions or improvements.
- Categorize model failures (hallucinations, logic errors, tone drift) to provide actionable feedback to the Engineering and Research teams.
- Help define and refine the rubric for "what a good response looks like" as the product evolves.
Details
- Building LLM-as-a-judge system with golden prompts and automation for output scoring
- Regression monitoring: detecting quality degradation across prompt, model, and config changes
- Evaluating agent routing
- AI Safety & Guardrails checks
Our values and what to expect:
- Customer First Mentality - Every decision we make should be made through the lens of the customer.
- Bias for Action - urgency is critical, expect that the timeline to get something done is accelerated.
- Ownership - Step up if you see an opportunity to help, even if it's not your core responsibility.
- Humility and Respect - Be willing to learn, be vulnerable, and treat everyone who interacts with RYZ with respect.
- Frugality - being frugal and cost-conscious helps us do more with less
- Deliver Impact - get things done most efficiently.
- Raise our Standards - always be looking to improve our processes, our team, and our expectations. The status quo is not good enough and never should be.
Vagas similares
Mantenha uma lista reserva.
Python, SQL 5 países aceitos
Data ScientistMorgan StanleyVer vaga Python 5 países aceitos
Application Security Engineer (Tech Lead)Morgan StanleyVer vaga Python 5 países aceitos
Application Security Engineer (Middle)Morgan StanleyVer vaga Python 5 países aceitos
Application Security Engineer (Senior)Morgan StanleyVer vaga Stack
Use estas tags para comparar vagas remotas similares.
Elegibilidade de localização
Candidatos devem aplicar apenas quando o país do perfil estiver listado aqui.
Seu perfilPaís não definidoEntre para comparar seu país com esta vaga.
Fluxo de contratação
O WithMira mostra a vaga e depois envia candidatos para a aplicação da empresa.
1Confira fit da vaga, stack e elegibilidade de localização no WithMira.
2Abra a página de aplicação da empresa pelo link rastreado.
3Salve a vaga ou assine oportunidades similares antes de sair.