Senior AI Data Engineer

Be a Part of Something BIG! Responsible for designing, building, and operating scalable data ingestion, transformation, and serving capabilities across a modern hybrid cloud data platform, ensuring solutions are reliable, secure, reusable, and aligned to enterprise architecture and governance standards.Develop and optimise batch and streaming pipelines using cloud tools such as Databricks and Kafka, applying sound engineering practices to ensure performance, resilience, and maintainability.Contribute to the delivery of reliable, secure, and high-quality data for analytics, reporting, and machine learning use casesLead the implementation and operationalisation of knowledge base and retrieval-augmented generation solution stacks to support scalable GenAI and agentic use cases across business domainsMake an Impact by:Design, build, optimise, and maintain batch and streaming data ingestion pipelines using platforms such as Databricks and Kafka, ensuring scalability, reliability, observability, and alignment with enterprise data architecture standards.Perform data transformation and cleansing using PySpark or SQL based on business and technical requirementsMonitor and troubleshoot data workflows to ensure data quality and pipeline reliabilityProvide technical guidance to engineers and delivery partners on data platform patterns, reusable components, code quality, deployment readiness, and production support practices.Lead integration of data from diverse source systems including files, APIs, databases, and streaming platforms, working with source-system owners and consuming teams to define fit-for-purpose ingestion patterns and delivery timelines.Help maintain metadata and pipeline documentation for transparency and traceabilityOwn production readiness for assigned data and AI platform components, including observability, incident triage, root-cause analysis, release coordination, and continuous improvement of operational runbooks.Participate in integrating pipelines with tools such as Microsoft Fabric, Databricks, Delta Lake, and other platform componentsBuild and maintain knowledge base and RAG solution on variety of hosting platformsImplement and operate knowledge base storage, lifecycle management and embedding/vectorizationContribute to automation efforts using version control and CI/CD workflowsApply data governance, security, access control, and operational risk policies during solution design and implementation, ensuring pipelines and knowledge platforms meet enterprise compliance requirements.Skills for Success: Bachelor’s degree in Computer Science, Engineering, or a related field5–8 years of experience in data engineering, data platform engineering, or cloud-scale analytics solution delivery, with demonstrated ownership of production pipelines and platform components.Proven ability to independently design, build, optimise, and operate production-grade batch or streaming data pipelines, including orchestration, observability, error handling, performance tuning, and operational support.Hands-on experience with Python and SQL for data transformation and validationFamiliarity with Apache Spark (especially PySpark) and large-scale data processing conceptsExperience with implementing knowledge base and RAG solutions for agentic AI use casesSelf-starter with strong problem-solving skills and a keen attention to detailAble to work independently and lead technical discussions with engineers, architects, product owners, source-system teams, and business stakeholders to translate requirements into secure and maintainable platform solutions.Strong documentation and communication skillsStrong understanding of enterprise data architecture, cloud security, access control, CI/CD, release management, and production operations for data and AI platform solutions.