Data Engineer - Hybrid / Remote

Data Engineer - Hybrid / Remote Opportunity

• Hybrid for candidates in Nashville and surrounding areas.
• Remote option available for candidates outside of surrounding areas.

This role requires a highly technical Data Engineer with expert-level proficiency in Azure Databricks, distributed data pipelines, and large-scale healthcare data processing. This role focuses on designing and implementing high-throughput ingestion pipelines, transactional lakehouse layers, and secure PHI data flows using Azure-native services and Databricks runtime optimizations.

You will build and operate production-grade data pipelines that meet rigorous requirements for security, lineage, compliance (HIPAA), observability, and operational SLAs, supporting analytics, AI, and clinical insights across the organization.

Core Responsibilities

Platform & Architecture

* Architect and implement scalable data processing pipelines using: * + Databricks Runtime (Apache Spark, Spark SQL, MLflow, Delta Lake) + Delta Lake ACID transactions, Z-Ordering, OPTIMIZE, and Change Data Feed (CDF) + Unity Catalog for governance, lineage, RBAC, and audit controls * Design and enforce a medallion (Bronze/Silver/Gold) architecture with schema evolution, Delta Live Tables (DLT), and robust error-handling patterns * Build high-performance ingestion frameworks for: * + FHIR and HL7 message streams + X12 837/835 healthcare claims data + EHR/EMR source systems + Batch, real-time, and event-driven data sources

Azure Cloud Engineering

* Develop and operate data pipelines leveraging: * + Azure Data Lake Storage Gen2 (hierarchical namespace, ACLs, POSIX permissions) + Azure Data Factory or Synapse Pipelines (parameterization, dynamic pipelines, triggers) + Azure Event Hubs and/or Service Bus for streaming ingestion + Azure SQL Database and Azure Synapse (Dedicated and Serverless pools) + Azure Functions for lightweight orchestration and automation + Azure Monitor, Log Ana...