Lead Data Engineer

Position Overview: 

We are seeking a highly skilled and experienced Lead Data Engineer to join our team. 

The ideal candidate will be responsible for designing, building, and maintaining data pipelines, ensuring they adhere to best practices and reference architecture. 

The Lead Data Engineer will also play a crucial role in guiding business analysts, defending project designs, and providing post-production support.

Key Responsibilities:

Pipeline Design and Best Practices:

Ensure all data pipelines are built and designed following best practices of the reference architecture.

Utilize the reference architecture to propose data flow from source systems to the Enterprise Data Platform (EDP) and its consumption by the business.

Development and Tool Selection:

Identify the most appropriate services/tools for project needs based on data size and transformation complexity.

Guide business analysts in documenting work, explaining project goals, and detailing source and destination table structures for future use.

Architectural Defense:

Defend project architecture designs to technical architects and project owners.

Explain the integration and efficiency of chosen tools to business stakeholders and management.

Post-Production Support:

Assist support teams with issues beyond their scope, providing guidance and troubleshooting.

Document common issues and create resources to reduce repetitive support requests.

Business Interaction:

Lead discussions with business teams, translating technical language into understandable terms.

Provide project status updates, including schedules, delays, and necessary modifications.

Communicate project progress and recommendations to internal management.

Technical Knowledge:

Familiarity with all technologies in the reference architecture, including AWS services (IAM, VPC, Security Groups, Redshift, S3, Glue), ETL processes, API development, SQL, CI/CD environments, and BI tools (PBi, AWS Quicksight).

Ability to learn new tools as required for projects and quickly adapt to new technologies.

Security:

Ensure data security according to business rules and interpret security requirements for implementation.

Manage data access for developers and end-users, ensuring appropriate security measures are in place.

Qualifications:

Proven experience in data engineering, with a strong focus on pipeline design and best practices.

Expertise in AWS services, ETL processes, API development, SQL, and CI/CD environments.

Strong communication skills, with the ability to translate technical concepts for business stakeholders.

Experience in leading discussions and providing project updates to both business and internal management.

Ability to troubleshoot complex issues and provide post-production support.

Familiarity with data security practices and the ability to implement them effectively.

Primary Skills: 

7+ years experience in ETL and Data Engineering 

1+ years RECENT experience in Team Leadership/Mentorship 

5+ years experience in AWS Services: Glue, Redshift, Lambdas, and S3 

Good communication skills

Education: Bachelors Degree