Senior QA Automation Engineer

About the Role We are seeking a highly skilled QA Automation Engineer to safeguard the quality, reliability, performance and safety of our AI-powered data products. In this role, you will architect intelligent, test automation frameworks, design rigorous evaluation systems for ML/LLM outputs and build quality infrastructure that scales alongside rapidly evolving AI capabilities. You'll be a key quality gatekeeper for AI products used in production, ensuring models behave safely, consistently, and as intended — before, during, and after release.Roles & ResponsibilitiesTest Automation & Framework EngineeringArchitect and build scalable, modular automation frameworks using Python, Pytest, Playwright, Selenium, or other custom-built tooling.Automate testing across UI, APIs, microservices, and AI models to enable fast, high-confidence releases.Build internal bots or agents that auto-generate test cases and run intelligent, AI-assisted exploratory testing.Champion best practices in test architecture, design patterns and maintainable, reusable test code.AI/ML & LLM EvaluationOwn end-to-end test strategy and execution for AI/ML models, GenAI, LLMs, and multi-agent systems, spanning functional, adversarial, safety, and security testing.Design evaluation frameworks to measure hallucination rate, bias, toxicity, and prompt-response consistency.Curate and maintain golden datasets; apply both deterministic rule-based checks and LLM-as-a judge techniques to evaluate output quality at scale.Data ValidationValidate data pipelines, training/inference data quality, feature stores, schema contracts to prevent data leakage or drift.Verify embedding quality, RAG retrieval relevance and the accuracy of agentic tool- and function calling.CI/CD & Production ReliabilityEmbed automated test suites into CI/CD pipelines, defining data-driven release gates based on AI quality metrics.Establish real-time production monitoring and dashboards to detect model drift, concept drift and anomalies.Continuously triage failures and tune performance to reduce pipeline execution time.Stakeholder CollaborationPartner with Data Scientists, ML Engineers, and Product Managers to define measurable acceptance criteria.Translate testing and quality metrics into actionable outcomes for both technical and business stakeholders.Who we are looking forWork QualificationsAt least 5 years of hands-on experience in QA, Test Automation, or SDET roles, including some experience in testing AI/ML, LLM or agentic systems.Technical ExperienceProgramming & FrameworksStrong proficiency in Python with solid software engineering fundamentalsHands-on experience with Pytest, unittest, or equivalent test frameworks; comfortable writing maintainable, well-documented test code.Proficient in SQL for querying, validating and profiling test data.Comfortable working with structured/semi-structured data formats and large-scale datasetsAutomation ExpertiseProven experience with API and UI test automationExperience designing end-to-end, data-driven and keyword-driven automation frameworks.Testing ProficiencyStrong grasp of test design techniquesExperience with risk-based testing and defining measurable quality/exit criteriaFunctional SkillsExcellent communication skills with the ability to translate complex technical concepts for both technical and non-technical stakeholdersComfortable working in cross-functional, fast-moving environments where priorities may evolve.