Lead Data Engineer
Atlanta, Georgia
Employer: Insight Global
Industry: Software Engineering
Salary: $65 per hour
Job type: Part-Time
Compensation:
$65/hr to $80/hr. - Exact compensation may vary based on several factors, including skills, experience, and education.
Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.
Job Description:
A Fortune 500 client of Insight Global is looking for a Lead Data Engineer to join their team. This person will be helping develop a new marketing data platform that measures digital marketing effectiveness. The platform is already architected on Azure and they predominantly use scala running on spark. The bulk of this person's role will be hands-on development, but they will also be responsible for interviewing and hiring 3 additional engineers to work alongside them to complete the project.
Job Responsibilities:
Lead end to end design and development of the W+ data solution.
Develop data pipelines to ingest and transform data using Scala/PySpark and SQL on Azure Synapse Spark.
Setup orchestration of the pipelines using Azure Synapse pipeline.
Source and version control code/configuration artifacts using GitHub.
Create GitHub actions to deploy code/configuration artifacts.
Create tables/views (using SQL) in Synapse SQL Serverless pool for consumption.
Investigate and remediate data pipeline errors or performance issues.
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com .
To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/ .
Required Skills & Experience
at least 5-7 years' experience in data engineering
In depth understanding of distributed computing (Spark)
Proficient in Scala/PySpark and SQL.
Knowledge of source code repository like GIT/Azure Repos & build tools like Maven.
Experience working with Azure Synapse.
Nice to Have Skills & Experience
PySpark
Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.
$65/hr to $80/hr. - Exact compensation may vary based on several factors, including skills, experience, and education.
Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.
Job Description:
A Fortune 500 client of Insight Global is looking for a Lead Data Engineer to join their team. This person will be helping develop a new marketing data platform that measures digital marketing effectiveness. The platform is already architected on Azure and they predominantly use scala running on spark. The bulk of this person's role will be hands-on development, but they will also be responsible for interviewing and hiring 3 additional engineers to work alongside them to complete the project.
Job Responsibilities:
Lead end to end design and development of the W+ data solution.
Develop data pipelines to ingest and transform data using Scala/PySpark and SQL on Azure Synapse Spark.
Setup orchestration of the pipelines using Azure Synapse pipeline.
Source and version control code/configuration artifacts using GitHub.
Create GitHub actions to deploy code/configuration artifacts.
Create tables/views (using SQL) in Synapse SQL Serverless pool for consumption.
Investigate and remediate data pipeline errors or performance issues.
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com .
To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/ .
Required Skills & Experience
at least 5-7 years' experience in data engineering
In depth understanding of distributed computing (Spark)
Proficient in Scala/PySpark and SQL.
Knowledge of source code repository like GIT/Azure Repos & build tools like Maven.
Experience working with Azure Synapse.
Nice to Have Skills & Experience
PySpark
Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.
Created: 2024-07-04
Reference: 366727
Country: United States
State: Georgia
City: Atlanta
ZIP: 30334
Similar jobs:
-
Sr. Data Engineer - Remote
UnitedHealth Group in Atlanta, Georgia💸 $85000 - $167300. per year -
Sr. Data Engineer
NLB Services in Atlanta, Georgia -
Sr. Data Engineer
TK Elevator in Atlanta, Georgia -
Manager, Data Engineer, Global Health & Wellbeing
Delta Air Lines in Atlanta, Georgia -
Senior Data Engineer
INTELLISWIFT INC in Atlanta, Georgia -
Sr. Data Engineer
TK Elevator in Atlanta, Georgia -
Practice Fusion Data Engineer
Medix in United States of America, Georgia -
Senior Data Engineer
Koch Industries in Atlanta, Georgia💸 $100000 - $140000 per year -
Data Engineer (Remote)
Cisco Systems Inc in Atlanta, Georgia -
Project Engineer, Data Center , DCC Communities
Amazon in Atlanta, Georgia -
Manager, Data Engineer, Global Health and Wellbeing
Delta Air Lines in Atlanta, Georgia -
SiteOps Data Center Production Operations Engineer
Meta in Newton County, Georgia -
Senior Data Engineer
System One Holdings, LLC in Atlanta, Georgia -
Data Engineer
AEG in Atlanta, Georgia -
Sr. Manager Enterprise Data of Science & Engineering
Leaf Home in Atlanta, Georgia -
Data Engineer 2 (Mailchimp)
Intuit in Atlanta, Georgia -
Cloud Data Engineer
Apex Systems in Atlanta, Georgia -
Data Engineer
Koch Industries in Atlanta, Georgia -
Lead Data Engineer
System One Holdings, LLC in Atlanta, Georgia -
Project Delivery Specialist - Cloud Data Engineer
Deloitte in Atlanta, Georgia