описание
We are looking for a Data Engineer with solid Databricks experience to design, build and operate data pipelines on a Lakehouse architecture. You will join a data team and work closely with analysts, data scientists and business teams. The focus is the Databricks platform: ingesting, transforming and serving reliable, governed and well-performing data. Experience with Azure and Microsoft Fabric is a secondary requirement, relevant for integration with the wider data ecosystem.
задачи
- Design and implement batch and streaming data pipelines on Databricks, using PySpark and Spark SQL.
- Build and maintain a layered Lakehouse architecture (medallion: bronze, silver, gold) on Delta Lake.
- Develop declarative pipelines and orchestrate workloads with Lakeflow (Delta Live Tables, Jobs/Workflows).
- Implement data governance with Unity Catalog: catalogs, permissions, lineage and auditing.
- Model data for analytical consumption and make it available to BI and Data Science teams.
- Document solutions and contribute to the team's engineering standards and best practices.
требования
- 3+ years of experience as a Data Engineer, including at least 2 years on Databricks projects in production.
- Strong command of Apache Spark (PySpark and Spark SQL) and Python.
- Advanced SQL, including query optimization and data modeling (dimensional and Lakehouse).
- Hands-on experience with Delta Lake and the medallion architecture.
- Experience with Unity Catalog, access management and governance.
- Experience orchestrating pipelines on Databricks (Jobs/Workflows, Delta Live Tables).
- Working practice with Git, CI/CD and test-driven development in data projects.
- Knowledge of stream processing (Structured Streaming, Auto Loader).
- Azure
- Azure Databricks and Azure Data Lake Storage Gen2.
- Azure Data Factory for ingestion and orchestration.
- Security and identity: Microsoft Entra ID, Key Vault, managed identities and private networking.
- Azure DevOps (Repos and Pipelines).
- Будет плюсом: Databricks certifications (Data Engineer Associate or Professional); Experience with dbt, Terraform or other infrastructure-as-code tools; Experience migrating to Databricks from legacy platforms (SQL Server, Synapse, Hadoop); Knowledge of MLflow and of supporting Machine Learning and AI use cases
условия
- Remote work
- Food ticket
- Health insurance
- Training and access to certification vouchers
- Work-life balance initiatives
- Referral program
- Birthday off
- Give back day – enjoy a day to give back to society!
