Data Engineer/ Analytical Engineer

New York, New York

Employer: Tata Consultancy Service Limited
Industry: Technology
Salary: Competitive
Job type: Full-Time
  • Programmed user-defined functions using Python packages like Pandas, NumPy, JSON, and Itertools for the optimized ingestion flow with a data orchestration tool called Prefect.
  • Performed Python Unit testing of developed code snippets using pytest and mock.
  • Maintained data Pipelines up to 99.8% while ingesting streaming and transactional data across various data sources using Python and Prefect.
  • Ingested data from disparate data sources using a combination of SQL, and Google Analytics API using Python to create data views to be used in BI tools like Tableau and PowerBI.
  • Created multiple code snippets of DBT for Data Transformation, creating transformed tables and views in Big Query Production datasets.
  • Performing joins, group by, and other aggregate operations as per Business rules on GCP Big query.
  • Data Modelling of the raw ingested data using SQL, Python, and DBT tools while increasing the quality and consistency of the business applications.
  • Migration from atomic layer to DBT, while automating the ETL process across billions of rows of data.
  • Built an ETL that ingests historical and transactional data from a Web API with about 12,000 daily actives users.
  • Analyzed data of developers in the SecAPI where the issues are about a million records in a day and ingested this data in Big Query through the Google Storage Buckets while using Python CICD.
  • Worked with event-based Google Cloud functions which trigger the ingestion pipelines when there is new data in the storage buckets.
  • Used Kanban framework to implement agile and DevOps software development which requires real time communication and full transparency of work.
Created: 2024-05-03
Reference: 310896
Country: United States
State: New York
City: New York
ZIP: 10036