Data Engineer

The client is an independent, active global asset manager with over R3 trillion in assets under management, executing a firm-wide data strategy to govern, manage and engineer data as a product across more than 70 business-owned sub-domains.

• Perform the end-to-end delivery of one or more priority data domains on Azure Databricks;
• Design and build medallion (Bronze / Silver / Gold) pipelines using Lakeflow Declarative Pipelines, Auto Loader, Structured Streaming and Delta Lake with Liquid Clustering;
• Register curated data products in Unity Catalog with the correct tags, masks, row filters, lineage and access policies;
• Implement data quality gates at the Silver-to-Gold boundary, and refine rules with domain stewards;
• Package and deploy work using Databricks Asset Bundles through Azure DevOps CI/CD;
• Write automated tests and documentation;
• Work at AI-as-Collaborator level today with a path to AI-as-Orchestrator, using Claude Code, Databricks Assistant, GitHub Copilot and MCP-connected agent tooling to accelerate the recurring parts of pipeline build, test and deploy.

• Have 6+ years of data engineering experience, with at least the last three years on Databricks in production;
• Possess deep hands-on experience with medallion / Lakehouse architecture, Delta Lake, Unity Catalog, Lakeflow Declarative Pipelines and Databricks Asset Bundles;
• Demonstrate strong python, PySpark and SQL skills, being comfortable diagnosing performance issues on large workloads through Spark UI and Photon;
• Bring real production experience with streaming ingestion (Kafka Structured Streaming, Auto Loader) and CDC patterns;
• Have experience implementing data quality at scale (Great Expectations or equivalent), plus lineage, cataloguing and access control;
• Build CI/CD on data platforms through Azure DevOps or GitHub Actions, with infrastructure as code (Terraform or equivalent);
• Fluent working English;
• Nice to have: Active Databricks certifications, Prior exposure to investment management platforms, asset management operations data, market data feeds from major providers, CDMP DAMA certification or equivalent data governance credential, Experience applying agentic engineering tooling to production data engineering (not just personal productivity), including MCP-connected Databricks or cloud servers.

• No conditions specified