Job Description
- Create ADF pipelines with good exposure of SQL knowledge.
- Create LLDSTM documents, develop, test, and deploy data integration processes using PythonPySpark on Azure
- Collaborate with onshoreoffshore lead designers, analyze LLDSTM, translate & apply business rules to data transformations
- Adopt integration standards, best practices, ABC framework etc while creating ETL jobs.
- Performs data validation, cleansing and analysis.
- Tests, debugs, and documents ETL processes, SQL queries, and stored procedures
- Analyzes data from various sources, including databases and flat files
- ETL development, deployment, optimization, support and defect fixes.
- Develop error handling processes
- Create and implement scheduling strategy
- Possess good knowledge of Agile methodologies using JIRA or DevOps
Detailed Requirement :
- The Data Engineer is responsible for the development, unit testing and implementation of data integration solutions .
- Responsible for importing, cleansing, transforming, validating and analyzing data with the purpose of understanding or making conclusions from the data for data modeling, data integration and decision making purposes.
- The primary role will be developing ADF Pipelines and unit testing ETL codes based on the requirement document (HLDLLDSTM) provided by Lead or Client and fixing defects in QA and UAT process, sometime may develop the ETL codes based on requirements.
- He should also possess good knowledge on Azure cloud platform, PythonPySpark is good to have and should have exposure to data model and concepts of Data warehouse to create ETL jobs.