Job Title - ETL Data Automation Engineer
Description - Job Summary:
This position is for an Offshore ETL Data Tester to support FordDirect's CC+ Product. . This individual will be responsible for creating and executing various test cases that support the processing of data related to the Ford and Lincoln consumer. The primary focus on this testing is to verify the accuracy of this data as various business rules are applied against it in an Azure Data Factory. This individual will be expected to understand the data flow, validate the content, and build an automation framework that supports a number of data quality checks.
Essential Job Functions:
This position is for an Offshore ETL Data Tester to support FordDirect's CC+ Product. . This individual will be responsible for creating and executing various test cases that support the processing of data related to the Ford and Lincoln consumer. The primary focus on this testing is to verify the accuracy of this data as various business rules are applied against it in an Azure Data Factory. This individual will be expected to understand the data flow, validate the content, and build an automation framework that supports a number of data quality checks.
Essential Job Functions:
- Create and execute scenarios to test various API. Preparing request block and analyzing the responses in JSON/XML formats.
- Validate the flow of data from disparate sources ingested into multiple databases inside MySQL/Databricks, post which data is transformed by workflows built inside Azure Data Factory followed by Azure Databricks (ETL process).
- Thoroughly test the ETL rules built for data transformation.
- Test different source and target tables available in MySQL and Azure Databricks that are sourced, cleansed, transformed, joined, aggregated and final data sent in files or in Synapse tables
- Develop test queries using SQL or Python or PySpark as per the user stories to validate the data inside various databases including Databricks.
- Automate recurring QA processes through the use of SQL scripts or advanced languages such as Java or Python.
- Help design and build out an automation framework and automation suite that continually validates data.
- Experience building basic workflows or data pipelines inside Azure Data Factory would be a plus. This would be a great add as certain repeated testing of data flows will need to be automated.
- Create and execute detailed manual test cases from information as related to functional requirements and technical specifications documentation to assure quality and accuracy.
- Log appropriate defects within Jira when product does not conform to specifications.
- Participate in daily stand-ups with project team as part of the agile methodology.
- Coordinate with development team members regarding defect validation and assist development team members with re-creating defects.
- Create appropriate test cases within TestRail Test Management tool.
- Update tasks information in Jira as appropriate to communicate progress with onshore test lead.
- Maintain technology expertise, keeping current with evolving testing tools, techniques, and strategies to improve the overall testing efficiency, processes, and best practices.
- Maintain a focus on customer-service, efficiency, quality, and growth.
- Safeguard the company's assets.
- Adhere to the company's compliance program.
- Maintain comprehensive knowledge of industry standards, methodologies, processes, and best practices.
- Excellent SQL skills and the ability to write complex queries.
- 2+ years working with Azure Data Factory
- Experience building basic workflows or data pipelines inside Azure Data Factory would be a plus.
- 1+ year experience with Azure Databricks
- Experience in Python programming to deal with data or PySpark experience would be a plus.
- Good understanding of file formats including JSON, Parquet, Avro, and others
- Ability to learn new technologies quickly
- Excellent problem-solving skills
- Basic understanding of software development principles.
- Knowledge of Jira
- Ability to handle multiple tasks/projects concurrently and meet deadlines.
- Ability to work in a fast-paced team environment. Expectations include a high level of initiative and a strong commitment to job knowledge, productivity, and attention to detail
- Solid software engineering skills - participated in full lifecycle development on large projects.