Senior Data Engineer

Company Overview

At Agilite, we harness the power of the Microsoft Azure ecosystem to deliver transformative data solutions for our healthcare clients. Our mission is to drive data-driven insights that improve outcomes, and we’re looking for a skilled Azure Data Engineer to join our team. Visit us at www.agilite.tech .

Job Title: Senior Data Engineer

Location: Remote

Job Type: Full-Time

Overview

We are seeking an experienced Azure Data Engineer to design, develop, and maintain scalable data pipelines and infrastructure within the Microsoft Azure ecosystem. The ideal candidate will have expertise in Databricks (Python or Scala), C#, SQL, and a strong background in data modeling, ETL processes, and data warehousing. With a focus on healthcare datasets, this role requires proficiency in Azure services, data visualization tools, and experience with SCM and CI/CD pipelines to deliver secure, efficient, and high-quality data solutions.

Key Responsibilities

• Data Pipeline Development : Build and optimize ETL/ELT pipelines using Azure Databricks (Python or Scala) and Apache Spark to process large-scale healthcare datasets.
• Data Modeling and Warehousing : Design and maintain data models, schemas, and data warehouses using Azure Synapse Analytics or Azure SQL Database to support analytics and reporting.
• SQL-Based Analytics : Write and optimize complex SQL queries for data transformation, analytics, and reporting within Azure SQL Database or Azure Synapse.
• C# Development : Develop data processing applications and APIs using C# to integrate with Azure services and enhance data workflows
• Healthcare Data Expertise : Handle sensitive healthcare datasets, ensuring compliance with HIPAA and other data privacy regulations while maintaining high standards of data security.
• SCM and CI/CD Implementation : Use Git for source control management and implement CI/CD pipelines with Azure DevOps to automate data pipeline deployments and ensure code quality.
• Collaboration : Work closely with data analysts, and business stakeholders to understand requirements and deliver data solutions that support advanced analytics
• Performance Optimization : Monitor and optimize Azure-based data systems for performance, reliability, and scalability, including tuning Spark jobs and Delta Lake configurations in Databricks.
• Documentation : Maintain detailed documentation for data pipelines, processes, and workflows to ensure transparency and reproducibility.

Required Qualifications

• Education : Bachelor’s or Master’s degree in Computer Science, Data Science, Engineering, or a related field.
• Experience :

+ 5+ years of experience in data engineering, with a focus on building scalable data pipelines. + Proven expertise in Azure Databricks using Python or Scala for data processing and analytics. + Strong proficiency in C# for developing data applications and APIs. + Extensive experience with SQL and SQL-based analytics, including query optimization in Azure SQL Database or Azure Synapse. + Demonstrated experience in data modeling, ETL/ELT processes, and data warehousing using Azure services. + Hands-on experience with Azure cloud services, including Azure Data Factory, Azure Data Lake Storage, and Azure Synapse Analytics. + Experience working with healthcare datasets and familiarity with HIPAA compliance. + Proficiency in Power BI for data visualization and reporting. + Experience with Git for SCM and Azure DevOps for CI/CD pipelines. * Technical Skills :

+ Expertise in Apache Spark, Delta Lake, and Azure Databricks for data processing. + Strong programming skills in Python or Scala and C#. + Advanced SQL skills for database management and analytics in Azure environments. + Knowledge of Azure security and governance practices (e.g., Azure Active Directory, role-based access control). + Familiarity with Azure-based big data technologies is a plus. * Soft Skills :

+ Strong problem-solving and analytical skills. + Excellent communication skills for collaboration with technical and non-technical stakeholders. + Ability to thrive in a fast-paced, agile environment. + Detail-oriented with a focus on delivering high-quality solutions.

Preferred Qualifications

• Certifications such as Microsoft Certified: Azure Data Engineer Associate or Databricks Certified Associate Developer.
• Experience with real-time data processing using Azure Stream Analytics or Databricks Structured Streaming.
• Familiarity with Azure Machine Learning or MLflow for integrating machine learning models into data pipelines.
• Knowledge of Azure Cosmos DB or other NoSQL databases.
• Experience in Agile/Scrum methodologies.

Benefits

• Competitive salary and performance-based bonuses.
• Comprehensive health insurance coverage.
• Flexible remote work options.
• Collaborative and innovative work environment