Senior Data Consultant

Position Summary:

The Senior Data Engineer will play a critical role in designing, developing, and maintaining our data infrastructure. The ideal candidate will have extensive experience with SQL, Python, Azure, Azure Data Factory (ADF), Databricks and Power BI.

Key Responsibilities:

Data Pipeline Development: Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks to ensure efficient data flow and transformation.

Data Integration: Integrate data from various sources, including APIs, databases, and flat files, into our data warehouse and data lake solutions.

Database Management: Utilize SQL for database design, development, and optimization to support data processing and analytics needs.

Data Modeling: Develop and maintain robust data models that support business requirements and enable effective data analysis.

Automation & Scripting: Write efficient and reusable Python scripts to automate data processing tasks and support data engineering workflows.

Reporting & Visualization: Collaborate with business intelligence teams to create interactive dashboards and reports in Power BI, providing valuable insights to stakeholders.

Performance Tuning: Monitor and optimize the performance of data pipelines, databases, and reporting solutions to ensure high availability and scalability.

Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data needs and deliver solutions that meet business objectives.

Documentation: Maintain comprehensive documentation of data engineering processes, data models, and integration workflows.

Technical Skills:

Proficiency in SQL for data querying, manipulation, and database management.

Strong programming skills in Python for data processing and automation.

Extensive experience with Microsoft Azure, particularly Azure Data Factory (ADF) and Databricks.

Experience with Power BI for data visualization and report development.

Analytical Skills: Strong problem-solving skills and the ability to analyze complex data requirements and develop appropriate solutions.

Communication Skills: Work closely with team members and stakeholders to understand data needs and deliver solutions that meet business objectives.

Team Player: Ability to work collaboratively in a team environment and manage multiple priorities in a fast-paced setting.

Qualifications:

Education: Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.

Experience: Minimum of 5 years of experience in data engineering or a related role.