Senior Data Engineer (Big Data, Data Lake & GenAI)

Job OverviewWe are seeking an experienced Senior Data Engineer with strong expertise in enterprise Data Warehousing, Big Data platforms, Data Lake architectures, and modern cloud technologies. The ideal candidate will have hands-on experience designing and implementing large-scale ETL/ELT pipelines, data transformation frameworks, and analytics solutions within banking, financial services, or large enterprise environments.The role requires deep technical knowledge of Apache Spark, PySpark, Hadoop ecosystems, Teradata, Cloud Data Platforms, and emerging Generative AI technologies for intelligent data processing and automation. The candidate should be capable of driving end-to-end data engineering initiatives, supporting business intelligence, analytics, regulatory reporting, and AI-enabled data solutions. Key ResponsibilitiesDesign, develop, and optimize enterprise Data Warehouse and Data Lake solutions.Build scalable ETL/ELT pipelines using Python, Spark, and PySpark.Develop and maintain data ingestion, transformation, and analytical data processing frameworks.Implement data integration solutions across structured, semi-structured, and unstructured data sources.Manage large-scale data platforms using Hadoop, AWS S3, Azure Databricks, and Cloud Data Services.Develop data marts and business intelligence datasets for analytics and reporting.Implement metadata management, data lineage, and governance frameworks using tools such as Collibra and Enterprise Data Catalog.Design and optimize Spark SQL workloads for high-performance analytics.Build automated workflow orchestration solutions using Airflow, Control-M, or equivalent scheduling platforms.Integrate Generative AI capabilities into data engineering workflows for metadata tagging, data classification, anomaly detection, and intelligent reporting.Collaborate with business stakeholders, architects, data scientists, and analytics teams to deliver scalable data solutions.Ensure data quality, governance, security, and compliance requirements are met.Support production systems, troubleshoot performance issues, and maintain SLA compliance. Required SkillsData EngineeringApache SparkPySparkSpark SQLPythonSQL / PL-SQLHadoop HDFSData Warehouse DesignData Lake ArchitectureETL / ELT DevelopmentDimensional ModelingStar Schema & Snowflake SchemaCloud & Big DataAWS S3Azure DatabricksGCP BigQueryCloudera Data PlatformKafkaParquetAlluxioData GovernanceCollibraEnterprise Data Catalog (EDC)Metadata ManagementData LineageData Quality ManagementDevOps & AutomationAirflowControl-MJenkinsGitLabBitbucketCI/CD PipelinesGenerative AIOpenAI APIClaude APILangChainPrompt EngineeringRetrieval-Augmented Generation (RAG)Vector Databases (Pinecone, ChromaDB) Preferred Experience10+ years of Data Engineering experience.Banking, Financial Services, or Insurance domain experience.Experience working with Teradata Data Warehouses.Hands-on experience in Data Lake modernization initiatives.Experience delivering regulatory reporting and compliance data solutions.Exposure to AI-enabled data platforms and intelligent analytics solutions.Experience supporting enterprise-scale production environment