data engineer in data warehousing

The team develops and operates Databricks and Spark data pipelines and Delta Lake lakehouse layers on Azure for a client, with accountability for pipeline reliability, data freshness, and dataset quality against agreed SLAs.

• Develop and operate Databricks / Spark pipelines using PySpark, SQL, Delta Live Tables, or Workflows;
• Design Delta Lake / lakehouse layers, including bronze–silver–gold architecture, partitioning, and Unity Catalog governance;
• Build ETL/ELT jobs and orchestration with Azure Data Factory and/or Airflow, managing dependencies and retries;
• Implement data-quality checks and validation, including expectations, reconciliation, and anomaly alerts;
• Tune Spark job performance and cluster cost using autoscaling, Photon, job clusters, and spot instances;
• Manage schema evolution and change control;
• Document data lineage and transformations;
• Ensure pipeline reliability and data freshness against agreed SLAs;
• Ensure the accuracy and completeness of curated datasets;
• Manage schema changes and backward compatibility for downstream consumers;
• Document data lineage and transformations using Unity Catalog and a data catalogue.

• Bachelor’s degree in Computer Science, Engineering, Information Systems, or a related field, or equivalent practical experience;
• 7+ Years of experience in data engineering, including 3+ years with Databricks / Apache Spark and 2+ years with Azure data services;
• Advanced knowledge of Databricks, including Workflows, Delta Live Tables, and Unity Catalog;
• Advanced knowledge of Apache Spark, including PySpark and Spark SQL;
• Experience with Delta Lake;
• Experience with Azure Data Factory, Data Lake Storage Gen2, Key Vault, Event Hubs, Synapse or SQL DB, and Azure DevOps CI/CD for notebooks and jobs;
• Expert-level Python and SQL skills;
• Knowledge of dimensional and data vault modelling and ELT design;
• Experience with orchestration using ADF or Airflow;
• Experience with data-quality frameworks, monitoring, and alerting;
• Experience with Spark workload performance and cost tuning;
• Experience with Git-based development and pipeline testing;
• English at C1 Advanced level;
• Nice to have: Databricks Certified Data Engineer Professional, Azure DP-203, Structured Streaming, Kafka / Event Hubs, dbt, Power BI semantic models, MLflow, experience in financial services, sovereign wealth / investment holding or other regulated enterprise environments, experience working with distributed teams across onsite UAE, nearshore India, and offshore Poland squads.

• Relocation options are available;
• Benefits include annual holiday, occasional leave, childcare leave, maternity leave, parental leave, paternity leave, private healthcare insurance, dental support, travel insurance, life insurance, corrective glasses reimbursement, a Multisport card, preferential banking and car leasing offers, cafeteria discounts, and financial support through the Luxoft Social Benefit Fund;
• Access to Luxoft Training Center courses, external learning libraries, cloud academies, custom learning programs, technical mentorship, and leadership programs;
• Rotation between projects and accounts and new career opportunities;
• Benefits depend on the form of cooperation and apply to employees under an employment contract;
• Relocation is not available for all open positions.