Databricks Data Engineer

Responsibilities

The Databricks Data Engineer Core AI & Data practice helps organizations modernize data platforms, strengthen enterprise data foundations, and scale analytics and artificial intelligence capabilities across the business. The team works with clients to architect, engineer, and deploy cloud-based data solutions that improve decision-making, enable innovation, and support large-scale transformation. As a hands-on Databricks Data Engineer with deep expertise in Azure/AWS Databricks (AKS/EKS as a backbone) and MLOps, this role will have the opportunity to migrate and translate legacy SSIS ETL logic into scalable, cloud-native data pipelines in Databricks. This role will partner with data engineers, data scientists, and product manager to design features, train/evaluate models, and deploy them to production using MLflow, Databricks and Workflows—with rigorous observability, governance (Unity Catalog), and CI/CD automation.

Data Pipeline Engineering

Design, build, and maintain high-performance, scalable ETL/ELT pipelines using Azure Databricks, Delta Lake, and PySpark.

• Convert and modernize existing SSIS package logic into cloud-native Databricks pipelines using PySpark notebooks, Delta Live Tables (DLT), and Databricks Workflows.
• Implement reliable batch and streaming pipelines with robust data quality and validation frameworks.
• Optimize pipeline performance using Photon, efficient file formats, partitioning, Z-ordering, and caching strategies.

Lakehouse Platform Development

Develop and manage datasets within Delta Lake, ensuring ACID reliability, schema evolution, versioning, and time travel.

• Architect feature-rich data layers including:
• Bronze (raw ingestion)
• Silver (validated, conformed)
• Gold (analytics-ready and ML-ready)
• Implement data governance using Unity Catalog for fine-grained access control, lineage, auditability, and metadata management.

MLOps & ML-Enabled Data Pipelines

Partner with...