Databricks Data Engineer - Remote

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, PySpark, Python, and SQL.
• Implement enterprise lakehouse architectures using Delta Lake, Delta Live Tables, Unity Catalog, and Databricks SQL.
• Build batch and real-time data-processing solutions using technologies such as Structured Streaming, Kafka, Event Hubs, or Kinesis.
• Develop reusable ingestion and transformation frameworks for structured, semi-structured, and unstructured data.
• Design and implement medallion architectures using Bronze, Silver, and Gold data layers.
• Develop agentic AI applications capable of planning tasks, retrieving information, calling tools, executing workflows, and validating results.
• Build AI agents using frameworks such as Databricks Mosaic AI Agent Framework, LangChain, LangGraph, LlamaIndex, AutoGen, or Semantic Kernel.
• Develop retrieval-augmented generation solutions using vector embeddings, semantic search, metadata filtering, and enterprise knowledge repositories.
• Implement vector-search solutions using Databricks Vector Search or comparable vector databases.
• Integrate large language models with enterprise applications, APIs, databases, document repositories, and business workflows.
• Implement model and agent evaluation processes covering accuracy, groundedness, relevance, hallucination, safety, latency, and cost.
• Develop agent memory, prompt-management, context-management, tool-calling, guardrail, and human-in-the-loop capabilities.
• Use MLflow for experiment tracking, model registration, evaluation, deployment, and lifecycle management.
• Build and support machine-learning and generative-AI deployment pipelines using Databricks Model Serving.
• Implement CI/CD processes for notebooks, pipelines, infrastructure, models, prompts, and AI agents.
• Automate Databricks deployments using **Databricks Asset Bundles, Terraform, Azure DevOps, GitHub Actions, or...