Data Engineer

Job Overview

We are seeking a dynamic and innovative Data Engineer to join our team and drive the development of scalable, high-performance data solutions. In this role, you will be at the forefront of designing, building, and maintaining robust data pipelines and architectures that empower data-driven decision-making across the organization. Your expertise will enable us to harness the full potential of big data technologies and cloud platforms, transforming complex data into actionable insights. If you thrive in a fast-paced environment and are passionate about creating efficient data systems, this is your opportunity to make a significant impact.

Duties

• Design, develop, and optimize scalable data pipelines using ETL (Extract, Transform, Load) processes to ensure seamless data flow across systems.
• Build and maintain data warehouses and lakes utilizing cloud platforms such as AWS, Azure Data Lake, and Hadoop ecosystems to support analytics and reporting needs.
• Collaborate with cross-functional teams to understand data requirements and translate them into technical solutions using SQL, Python, Java, and shell scripting.
• Implement data integration solutions with tools like Informatica, Talend, and Apache Hive to facilitate comprehensive data analysis.
• Develop RESTful APIs for secure data sharing between applications while ensuring compliance with best practices for data security and privacy.
• Support model training and analytics initiatives by preparing clean datasets and optimizing database design for performance.
• Monitor system performance, troubleshoot issues promptly, and continuously improve data infrastructure for efficiency and scalability.
• Stay current with emerging big data technologies such as Spark, Hadoop, and Looker to incorporate innovative solutions into our architecture.

Skills

• Proficiency in cloud platforms including AWS and Azure Data Lake for scalable storage solutions.
• Strong programming skills in Java, Python, Bash (Unix shell), Shell Scripting, and VBA for automation and customization tasks.
• Extensive experience with big data frameworks such as Hadoop, Spark, Apache Hive, and related tools for processing large datasets.
• Expertise in SQL databases including Microsoft SQL Server, Oracle, and Data Warehouse design principles for structured data management.
• Knowledge of ETL tools like Informatica and Talend for efficient data integration workflows.
• Familiarity with Looker or similar BI tools for creating insightful dashboards and reports.
• Understanding of Linked Data principles to connect disparate datasets across platforms effectively.
• Experience with analysis skills including statistical methods and model training techniques to derive insights from complex datasets.
• Ability to design robust database schemas that support high-volume transaction processing while maintaining integrity.
• Agile methodology experience to facilitate iterative development cycles in collaborative environments.

Join us if you're eager to leverage your expertise in big data technologies to shape innovative solutions that empower smarter decisions!

Pay: $120,123.83 - $144,665.25 per year

Work Location: Remote