Senior Data Engineer, Databricks & AI Automation

Flintex Consulting · Singapore

Sector
AI
Function
Product & Engineering
Level
Mid-Level
Employment type
Contract
Posted
2026-11-19
Source
mycareersfuture

Senior Data Engineer Databricks and AutomationData and AI Innovation TeamAbout the roleBuild reliable data pipelines and trusted datasets for analytics, business applications and AI use cases. This hands-on role centres on Databricks, Python, PySpark and SQL. You will also simplify recurring engineering and support activities through automation, using AI where it provides a clear benefit.Key responsibilities• Design, build and maintain production ETL/ELT pipelines using Databricks, PySpark, SQL and Delta Lake.• Integrate data from databases, APIs and files, with streaming ingestion where required.• Develop curated datasets and data models that meet agreed business definitions, access requirements and consumption needs.• Implement data quality checks, reconciliation, incremental processing, schema handling and recovery procedures.• Monitor pipeline reliability and data freshness; investigate failures, resolve root causes and improve recovery.• Optimise processing performance and job compute usage, partnering with the Platform Engineer on shared settings and controls.• Apply version control, automated testing, CI/CD and documentation to support reliable releases and maintainable solutions.• Automate repetitive engineering and support tasks; assess AI-assisted approaches against accuracy, security and manual effort saved.Essential skills and experience• Typically 5 or more years of hands-on data engineering experience, including building and supporting production pipelines.• Strong Python, SQL and Spark or PySpark skills.• Practical Databricks and Delta Lake experience, including job orchestration and troubleshooting. Strong production Spark and lakehouse experience with less Databricks exposure will also be considered.• Experience with data modelling, incremental ingestion, data quality, reconciliation and performance tuning.• Experience with a cloud data platform and practical use of Git, testing and deployment pipelines.• Ability to translate business requirements into maintainable data solutions and collaborate with business and technical teams.Preferred experience• Azure data services and enterprise data integration.• Unity Catalog, Auto Loader, Structured Streaming or Databricks deployment tooling.• Streaming technologies such as Kafka or Event Hubs.• Automation of monitoring, reconciliation or operational support.• Practical use of LLMs, AI agents or Databricks AI capabilities to improve engineering workflows.Role ownershipOwn data pipelines, data quality, curated datasets and job-level performance. Partner with the Platform Engineer on shared environments, deployment tooling, access controls and platform monitoring.What success looks likeAccurate and timely data, reliable pipelines, efficient processing, faster recovery from failures and less manual support effort.

Apply on mycareersfuture →
AI Business Data Analysis Collaborate With Engineers PySpark Design AI Agents Streaming Video Product Catalogues