Data Engineer

Piper Companies is looking for a Data Engineer for a hybrid, Contract-to-hire opportunity with a Health Information Exchange organization in Columbia, MD.

Responsibilities of the Data Engineer:

• Design, build, and maintain large, complex data processing pipelines that meet business requirements.
• Design, develop, and maintain a data analytics platform solution, focusing on creating new use cases through Databricks programming within the healthcare data domain.
• Ensure the quality of deliverables by developing automated controls and performing unit, integration, and user acceptance testing.
• Develop scalable and reusable frameworks for ingestion and transformation of large data sets.
• Troubleshoot and perform root cause analysis on data pipeline issues.
• Collaborate with source system owners and business owners to incorporate changes into data pipelines.
• Create comprehensive documentation of data workflows, processes, and infrastructure.

Qualifications for the Data Engineer:

• 4+ years of experience in data integration.
• Advanced programming skills in PySpark, Python, Spark SQL.
• Thorough understanding of Data Lakes, raw/enriched/curated layer concepts, and ETL within the Azure framework.
• Proven experience in building complex data pipelines for lakehouse/data warehouses using Agile methodology.
• Experience in architecture, design, and implementation using Databricks and Azure Data Factory.
• Experience working with Databricks (preferred).
• 4+ year college degree (required).

Compensation/Benefits for the Data Engineer:

• $45-$55 per hour pay rate
• Contract-to-hire opportunity
• Comprehensive Benefits package: Medical, Dental, Vision, 401k, ect.

Keywords: Data Engineer, Data Developer, Data Manager, Pyspark, Python, Apache Spark, Azure, Azure Data Lake 2, Azure Data Factory, Spark, Databricks, Python, Spark SQL, Data Lakes, Data warehouse, Data Pipelines, Agile Methodology, Data Processing Pipeline, Troubleshooting, Automation, Unit testing, Integration Testing, #LI-zs1, #LI-hyrbid