Software Quality Assurance Engineer – GenAI / LLM

Primary Focus: Software Quality Assurance / Software Testing / Test Automation – GenAI, LLM & Agentic AISecondary Exposure: Solution Analysis / Technology Solution Design / Enterprise IntegrationDomain / Project: Global Markets, Capital Markets Banking Technology & Market Risk TechnologyRole OverviewWe are looking for a Senior GenAI Quality Engineer / Solution Analyst to design, analyse, test and validate production-grade Generative AI (GenAI), Large Language Model (LLM), RAG and Agentic AI applications within a complex enterprise environment.This is not a traditional manual QA or software testing role.The role combines:Software Quality EngineeringGenAI / LLM Testing & EvaluationAgentic AI / AI Agent TestingUI & API TestingTest AutomationSolution AnalysisEnterprise Integration TestingObservability & TroubleshootingYou will work across discovery, solution design, development, testing and release, translating business requirements into clear application behaviours and validating end-to-end application quality across user interfaces, APIs, data flows, LLMs, RAG components, AI agents and enterprise integrations.Key ResponsibilitiesGenAI / LLM Quality EngineeringDefine and execute end-to-end quality engineering and test strategies covering:Web / UI workflowsREST APIsBackend servicesEnterprise integrationsGenAI applicationsLLM workflowsRAG pipelinesAgentic AI / AI Agent interfacesPerform GenAI / LLM testing and evaluation covering:Response qualityTask completionGroundingFaithfulnessRelevanceConsistencyCitation accuracyHallucination riskSafe failure behaviourTest non-deterministic / probabilistic AI systems using:Evaluation datasetsRepeat testingQuality thresholdsAcceptance criteriaRegression evaluationValidate RAG / Retrieval-Augmented Generation solutions, including retrieval quality, grounding and response accuracy.Agentic AI / AI Agent TestingTest end-to-end Agentic AI and AI Agent workflows, including:Multi-turn conversationsContext handlingAgent planningTool selectionTool calling / function callingTool inputs and outputsState transitionsMemory and stateHuman-in-the-loop approvalsHandoffsRetriesTimeoutsFallback behaviourError recoveryTermination conditionsPartial failuresValidate that AI agents behave correctly across both successful and failure scenarios.Software & API Quality EngineeringPerform:Functional TestingIntegration TestingAPI TestingRegression TestingExploratory TestingNegative TestingResilience TestingBasic Performance TestingEnd-to-End TestingDesign comprehensive REST API tests covering:API contractsAuthenticationAuthorisationInput validationError handlingIdempotencyRate limitsDownstream system failuresTest web application behaviour across browsers and realistic end-user journeys, including:Loading statesInterrupted sessionsError messagesFeedback captureAccessibility fundamentalsTest AutomationDevelop and maintain risk-based test automation that reduces:Regression testing timeManual testing effortRelease cycle timeProduction riskUse automation frameworks and tools such as:PlaywrightCypressSeleniumpytestREST AssuredPostmanEquivalent UI / API automation frameworksApply pragmatic automation principles by prioritising stable, high-value and frequently executed test scenarios.GenAI Evaluation & AI Safety TestingValidate LLM and GenAI applications for:Grounded responsesHallucinationsRetrieval qualityCitation accuracyPrompt behaviourPrompt injectionUnsupported requestsRestricted content handlingSafe failure behaviourAdversarial scenariosSupport AI evaluation / LLM evaluation using appropriate evaluation datasets, quality metrics and repeatable evaluation approaches.Exposure to AI Red Teaming / Adversarial Testing would be advantageous.Observability & TroubleshootingUse application and GenAI observability to identify the source of defects across:ApplicationLLM / ModelRAG / RetrievalDataAPI / IntegrationPlatformAnalyse:LogsDistributed tracesAPI requests / responsesPayloadsNetwork callsDatabase recordsAgent execution tracesExposure to observability and LLM evaluation tools such as:LangfuseLangSmithOpenTelemetryElastic / ElasticsearchSplunkis advantageous.Solution Analysis & DesignThe role also acts as a hands-on Solution Analyst for GenAI applications.Responsibilities include:Partner with product owners, business users, architects, engineers and GenAI specialists during discovery and solution design.Analyse proposed GenAI use cases and determine whether the requirement should use:Conventional application logicDeterministic business rulesSearch / retrievalRAGWorkflow automationAgentic AIHuman approvalTranslate business requirements into:Functional requirementsEnd-to-end solution flowsUser journeysAcceptance criteriaInterface behaviourDecision rulesNon-functional requirementsMap interactions across:User InterfacesAPIsLLMs / ModelsPromptsRAG / Retrieval componentsEnterprise data sourcesAI Agent toolsDownstream enterprise systemsAnalyse solution design trade-offs involving:QualityComplexityCostLatencySecurityData accessMaintainabilityOperational riskIdentify missing controls, integration assumptions, ownership gaps, failure scenarios and operational risks before development begins.Support the design of:Human-in-the-loop approvalFallback flowsEscalationException handlingSolution DocumentationProduce practical technical and functional artefacts including:Process FlowsSequence DiagramsContext DiagramsInterface SpecificationsDecision TablesUser StoriesAcceptance CriteriaTest ScenariosTraceability DocumentationMaintain traceability across:Business Requirement → Solution Design → Implementation → Test / Evaluation Scenario → Release EvidenceRelease Quality & GovernanceCreate and maintain:Test scenariosTest datasetsReusable regression scenariosTest evidenceDefect reportsQuality metricsRelease quality reportsProvide evidence-based release recommendations identifying:Known defectsKnown limitationsResidual risksQuality concernsAreas requiring production monitoringCore RequirementsExperience5–8 years of experience in Software Quality Engineering, Test Engineering, Test Automation, SDET or similar hands-on software testing roles.Strong experience testing complex enterprise applications.Strong experience testing:Web applicationsREST APIsBackend servicesEnterprise integrationsTest Automation / ProgrammingHands-on experience with one or more of:PlaywrightCypressSeleniumpytestREST AssuredPostmanEquivalent automation frameworksWorking programming knowledge of:PythonJavaJavaScriptTypeScriptCandidates should be capable of developing, reviewing and troubleshooting test automation.Software Engineering / DevOpsExperience with:GitPull RequestsCI/CDAutomated TestingTest ReportingDefect ManagementExperience validating distributed systems including:Asynchronous ProcessingQueuesBatch ProcessingAPIsDownstream DependenciesEnterprise IntegrationsGenAI / LLM RequirementsPractical understanding of:Generative AI / GenAILarge Language Models / LLMLLM EvaluationLLM TestingRetrieval-Augmented Generation / RAGRAG EvaluationAgentic AIAI AgentsMulti-Agent WorkflowsPrompts / Prompt EngineeringContext WindowsEmbeddingsTool CallingAgent Memory & StateLLM ObservabilityCandidates should understand how GenAI applications differ from conventional deterministic software and how to validate probabilistic AI behaviour.Security & Risk TestingUnderstanding of software and GenAI security fundamentals including:Access ControlAuthentication / AuthorisationSensitive Data HandlingInput ValidationAuditabilityPrompt InjectionAI Safety TestingAdversarial TestingNice to HaveExperience with:Banking / Financial ServicesRegulated enterprise environmentsContract TestingService VirtualisationSynthetic MonitoringPerformance TestingAI Red TeamingAccessibility Testing / WCAGKubernetesOpenShiftAWSContainerised Application DeploymentKey Domain / Technical Skills1. Software Quality Engineering, API Testing & Test Automation2. GenAI / LLM Evaluation, RAG & Agentic AI Testing3. Solution Analysis, Observability & Enterprise IntegrationKey Search KeywordsGenAI Quality Engineer,AI Quality Engineer,LLM Quality Engineer,Generative AI Testing,GenAI Testing,LLM Testing,LLM Evaluation,AI Evaluation,Agentic AI Testing,AI Agent Testing,RAG Testing,RAG Evaluation,Retrieval-Augmented Generation,Software Quality Engineering,Quality Engineering,Software QA,Test Automation,SDET,Automation Testing,API Testing,REST API Testing,UI Testing,Integration Testing,Regression Testing,End-to-End Testing,Playwright,Cypress,Selenium,pytest,REST Assured,Postman,Python,Java,JavaScript,TypeScript,CI/CD,Git,Prompt Testing,Prompt Injection,Hallucination Testing,Grounding,Faithfulness,AI Safety Testing,Adversarial Testing,AI Red Teaming,Langfuse,LangSmith,OpenTelemetry,Elastic,Splunk,Observability,Distributed Systems,Kubernetes,OpenShift,Solution Analysis