Quality Assurance Automation Engineer- AI

Details

• Design, build, and maintain robust automation frameworks specifically tailored for testing AI features, including LLM-powered outputs, recommendation engines, and intelligent workflows.
• Develop and execute test strategies for validating AI model outputs, including accuracy, relevance, consistency, and bias evaluation across diverse input scenarios.
• Perform functional, regression, performance, and scalability testing on AI-driven features, including API-level testing of AI service integrations and end-to-end automated test suites.
• Collaborate with AI engineers, data scientists, product managers, and UX designers to define acceptance criteria and quality standards for AI features throughout the development lifecycle.
• Identify, document, and track defects in AI behavior, including edge cases, hallucinations, and unexpected model responses through resolution using appropriate tools and workflows.
• Champion quality and responsible AI practices throughout the development lifecycle, contributing to prompt engineering reviews, model evaluation rubrics, and AI safety checklists.
• Build and maintain CI/CD-integrated automation pipelines that continuously validate AI feature quality, using tools such as Playwright, Claude, or equivalent frameworks.
• Participate in team stand-ups, sprint planning, retrospectives, and continuous improvement activities, bringing an AI-quality-first mindset to every discussion and helping establish best practices for testing intelligent systems.
• 2-4 years of experience in QA automation engineering, with demonstrated hands-on work building and maintaining automated test frameworks.
• Proficiency with automation tools and frameworks (e.g., Playwright, Claude, or similar) and API testing tools (e.g., Postman, REST-assured).
• Demonstrated experience testing AI/ML-integrated features or services, with a solid understanding of the unique challenges of validating non-deterministic outputs, including LLM responses, probabilistic recommendations, and generative content.
• Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
• Strong analytical and critical-thinking skills, with the ability to design test cases for complex, probabilistic AI behaviors.
• Excellent verbal and written communication skills, with the ability to clearly articulate AI quality issues and their business impact to both technical and non-technical stakeholders.
• Hands-on experience with LLM evaluation techniques, including prompt testing, output scoring, and benchmarking AI model responses against defined quality criteria.
• Familiarity with Microsoft Azure AI services, Azure OpenAI, and cloud-based AI infrastructure (e.g., Azure Machine Learning, Cognitive Services).
• Experience with data-driven testing approaches, including generating and managing synthetic test datasets for training and validating AI model behavior at scale.
• Understanding of responsible AI principles, including fairness, transparency, and explainability, and experience incorporating these into QA testing practices.
• Proficiency JavaScript for scripting test utilities, data processing pipelines, and custom AI evaluation harnesses.
• Background in integrating automated QA into CI/CD pipelines (e.g., GitHub Actions, Azure DevOps) with experience in shift-left testing and continuous quality monitoring for AI-driven products.
• We’re a remote-first company with team members across the USA, Canada, UK, and India!
• OnePlan has been recognized as the Global Microsoft Partner of the Year in Project Portfolio Management in 2019, 2020, 2021, 2022 and 2023.
• We’ve been named a "Strong Performer" in the latest Forrester Strategic Portfolio Management WAVE report.
• We offer comprehensive health, dental, and vision benefits, with additional insurance options.
• Employer RRSP and 401K matching programs.
• A fun, collaborative, and diverse environment with regular health and team challenges to keep things light and enjoyable!