Senior AI Data Engineer

NCS is a leading AI Tech Services company. With a 15,000-strong team across the Asia Pacific, NCS scales its platforms and capabilities to provide clients with greater agility and AI expertise across a range of Industries. Embracing a strong ecosystem of global partners, NCS transforms technology services delivery combining AI with digital resilience to drive real business impact. NCS is a subsidiary of the Singtel Group.Job DescriptionRole SummaryYou are an autonomous AI Engineer, responsible for creating the harness for designing and building the AI harness, agent framework, and reusable engineering components required to operate a modern data platform based on Apache Iceberg, Trino, and Parquet.Your primary responsibility is not only to build data pipelines, but also to establish the agentic execution framework (AI Harness) that enables multiple specialized agents to collaborate, execute tasks, maintain state, orchestrate workflows, and deliver end-to-end data engineering capabilities.Depending on your level of experience, your focus in the team will be on either task execution, independent delivery, or technical leadership.Key ResponsibilitiesAI Harness DevelopmentDesign and implement the foundational agent framework that provides:Agent orchestration and coordinationWorkflow execution and state managementMemory and context handlingTool and function callingAgent-to-agent communicationHuman approval workflowsLogging and audit trailsError handling and recoverySecurity and access controlsObservability and monitoringThe harness must support autonomous and multi-agent execution patterns.Specialized Agents CreationThe agents will need to perform the following:Discover Iceberg catalogs and schemasProfile datasetsGenerate metadata and lineageIngest data from databases, APIs, files, and streamsCreate Parquet datasetsRegister and maintain Iceberg tablesGenerate Trino SQLBuild transformation workflowsCreate curated and business-ready datasetsValidate schemas and data integrityDetect anomaliesGenerate reconciliation reportsAnalyze Trino workloadsRecommend partitioning strategiesOptimize Iceberg tables and Parquet filesClassify sensitive informationManage metadata and lineageApply retention and compliance policiesMonitor pipelines and agent activitiesDetect failures and bottlenecksGenerate alerts and operational insightsExpected DeliverablesIceberg table definitions and schemasTrino SQL scripts and viewsPython data pipeline codeData quality validation suitesMetadata and lineage documentationCI/CD deployment artifactsMonitoring and observability configurationsTechnical documentation and runbooksQualificationsSkills and ExperienceThe candidate should demonstrate expertise in:Experience with Generative AI, Agentic AI, or LLM-based solutions.Proficiency in Python, including FastAPI, Pydantic and Pandas.Experience designing and implementing AI Harness or multi-agent frameworks.Hands-on experience with LangChain & LangGraph.Experience with Apache Iceberg, Trino, and Parquet-based lakehouse architectures.Experience building data ingestion, transformation, and orchestration pipelines.Strong knowledge of metadata management, data lineage, and data governance practices.Experience developing REST APIs, SDKs, and reusable engineering frameworks.Experience with observability, monitoring, tracing, and operational support practices.Strong understanding of distributed systems, event-driven architectures, and microservices patterns.Proven ability to design, develop, and operationalize enterprise-grade AI agents for data discovery, ingestion, transformation, optimization, governance, and observability.Familiar with modern data engineering technologies, including DuckDB, dbt, and related engineering tools for data transformation, modelling, and pipeline development.Senior engineers will be expected to have demonstrated evidence of leading technical designs, guiding junior engineering team members, working independently with clients and delivery teams, and translating AI/data requirements into scalable, secure, and deployable solutions.We are driven by our AEIOU beliefs—Adventure, Excellence, Integrity, Ownership, and Unity—and we seek individuals who embody these values in both their professional and personal lives. We are committed to our Impact: Valuing our clients, Growing our people, and Creating our future.Together, we make the extraordinary happen.Learn more about us at ncs.co and visit our LinkedIn career site.Scam AlertWe are aware of fraudulent job offers and impersonations of NCS recruiters. Phishing emails using convincing-looking but fake addresses are also commonly used to trick you into thinking that they come from official NCS sources.Please note that all official communications from NCS Group will only be sent from verified corporate email addresses. Always check that the sender’s email address ends with the genuine NCS domain, @ncs.com.sg and beware of extra letters, symbols or misspellings. When in doubt, verify the sender’s identity by contacting us at [email protected].