описание
We are looking for an experienced Azure Data Engineer with strong expertise in Azure Databricks, PySpark, Python, and Azure Data Factory to design, develop, and maintain scalable data pipelines and ETL/ELT solutions. The candidate should have hands-on experience in big data processing, cloud data engineering, pipeline optimization, and CI/CD implementation using Azure DevOps.
задачи
- Design, develop, and maintain data pipelines using Azure Data Factory and Azure Databricks.
- Build and optimize large-scale data processing solutions using PySpark and Python.
- Develop ETL/ELT workflows for data extraction, transformation, and loading.
- Work with Azure Data Lake Storage (ADLS) and Delta Lake.
- Implement CI/CD pipelines using Azure DevOps.
- Write complex SQL queries and optimize data processing performance.
- Monitor, troubleshoot, and optimize data pipelines.
- Ensure data quality, security, and compliance.
- Collaborate with cross-functional teams to deliver effective data solutions.
требования
- Strong hands-on experience with Azure Databricks.
- Expertise in PySpark and Python programming.
- Experience with Azure Data Factory (ADF) and pipeline orchestration.
- Strong knowledge of SQL and Spark SQL.
- Experience with Azure DevOps and CI/CD pipelines.
- Understanding of ETL/ELT processes and data integration.
- Good problem-solving, communication, and documentation skills.
- Будет плюсом: Azure Data Lake Storage (ADLS Gen2); Delta Lake and Delta Tables; Azure Synapse Analytics; Databricks Workflows and Jobs; Git and version control; Medallion Architecture (Bronze, Silver, Gold)
условия
- Условий в вакансии нет
навыки
компания
Лондонская компания IT-рекрутинга и консалтинга с 2000 года.