Ryz Labs
AI QA Analyst
Rol remoto de AI QA Analyst con fit claro de ubicación del candidato.
Publicado19 jun 2026
Países elegibles24 países aceptados
Señal de seniorityMiddle
Modelo de trabajoRemoto
Ubicaciones aceptadas para candidatos
Resumen del rol
AI QA Analyst
Requisitos y responsabilidades
Contenido del rol extraído en secciones para revisar más rápido.
Qualifications:
- Ability to spot subtle logical inconsistencies or factual inaccuracies in AI-generated content.
- Experience managing, cleaning, or labeling datasets for machine learning or NLP projects.
- Familiarity with the AI lifecycle, Prompt Engineering, and tools like Google Sheets/Excel (SQL or Python is a significant plus).
Key Responsibilities:
- Review and annotate complex conversation traces to rate response quality based on metrics such as helpfulness, honesty, and harmlessness (HHH).
- Build and maintain high-quality "Golden Datasets" and benchmarks to stress-test the model across various domains and edge cases.
- Conduct pre-deployment testing and A/B model comparisons to identify performance regressions or improvements.
- Categorize model failures (hallucinations, logic errors, tone drift) to provide actionable feedback to the Engineering and Research teams.
- Help define and refine the rubric for "what a good response looks like" as the product evolves.
Details
- Building LLM-as-a-judge system with golden prompts and automation for output scoring
- Regression monitoring: detecting quality degradation across prompt, model, and config changes
- Evaluating agent routing
- AI Safety & Guardrails checks
Our values and what to expect:
- Customer First Mentality - Every decision we make should be made through the lens of the customer.
- Bias for Action - urgency is critical, expect that the timeline to get something done is accelerated.
- Ownership - Step up if you see an opportunity to help, even if it's not your core responsibility.
- Humility and Respect - Be willing to learn, be vulnerable, and treat everyone who interacts with RYZ with respect.
- Frugality - being frugal and cost-conscious helps us do more with less
- Deliver Impact - get things done most efficiently.
- Raise our Standards - always be looking to improve our processes, our team, and our expectations. The status quo is not good enough and never should be.
Roles similares
Mantén una lista de respaldo.
Python, SQL 5 países aceptados
Data ScientistMorgan StanleyVer rol Python 5 países aceptados
Application Security Engineer (Tech Lead)Morgan StanleyVer rol Python 5 países aceptados
Application Security Engineer (Middle)Morgan StanleyVer rol Python 5 países aceptados
Application Security Engineer (Senior)Morgan StanleyVer rol Stack
Usa estas tags para comparar roles remotos similares.
Elegibilidad de ubicación
Candidatos deberían aplicar solo cuando el país del perfil aparece aquí.
Tu perfilPaís no definidoInicia sesión para comparar tu país con este rol.
Flujo de contratación
WithMira muestra el rol y luego envía candidatos a la aplicación de la empresa.
1Revisa fit del rol, stack y elegibilidad de ubicación en WithMira.
2Abre la página de aplicación de la empresa desde el link rastreado.
3Guarda el rol o suscríbete a oportunidades similares antes de salir.