описание
The London Borough of Havering is modernising how we manage, connect and use data across our services, with Microsoft Fabric at the heart of our future data and analytics architecture. We're looking for an experienced, hands-on Lead Data Engineer to help develop and mature our Microsoft data environment, drive the phased migration and consolidation of legacy data platforms, and embed modern engineering and governance practices.
задачи
- Developing, optimising and maintaining our Microsoft Fabric environment, including OneLake, Data Factory pipelines, lakehouses, warehouses, Spark notebooks, Delta tables and Power BI semantic models.
- Assessing and prioritising existing SQL Server data warehouses, SSIS packages, data marts and integrations; designing appropriate migration and modernisation approaches; and delivering phased improvements with robust testing, data validation and service continuity.
- Helping develop consistent approaches to workspace organisation, development, test and production environments, access management, source control, automated deployment, monitoring and operational management.
- Developing, maintaining and improving SQL Server data warehouses and ETL/ELT processes, integrating data from cloud and on-premises databases, Dynamics 365, Dataverse, APIs, SFTP and other business systems.
- Writing and tuning complex T-SQL, stored procedures and views, improving performance and investigating data issues through root-cause analysis.
- Supporting person-matching services, golden-record management, UPRN-based address matching and the Local Land and Property Gazetteer, alongside existing ASP.NET/C# applications and opportunities for modernisation.
- Embedding data quality, ownership, metadata management, lineage and appropriate access controls into engineering solutions. Working with Information Governance and Data Protection colleagues to protect sensitive information, support DPIAs and meet UK GDPR requirements, while developing the use of Microsoft Fabric and Purview governance capabilities.
- Monitoring and optimising data pipelines, workloads and platform performance, supporting capacity and cost efficiency, troubleshooting production issues and implementing reliable operational processes.
- Improving engineering practices through Git, Azure DevOps, CI/CD, automated testing and Microsoft Fabric deployment pipelines.
- Taking ownership of engineering deliverables, contributing to technical decisions, mentoring colleagues, reviewing technical work and promoting consistent, maintainable engineering practices across the data estate.
- Establishing trusted, reusable data foundations for analytics, Power BI, Power Platform, joined-up digital services, automation and responsible AI adoption.
требования
- Strong experience developing and supporting SQL Server data warehouses, data marts and ETL/ELT processes in production environments.
- Advanced T-SQL skills, including query optimisation, stored procedures and troubleshooting.
- Experience building and maintaining reliable data pipelines, including SSIS or comparable integration technologies.
- Experience with Microsoft Fabric, Azure Data Factory, Databricks or another modern cloud data engineering platform.
- Experience modernising or migrating legacy data solutions to cloud-based environments.
- Good knowledge of data modelling, integration patterns, data quality and performance optimisation.
- Experience integrating data from databases, business applications, APIs and file-based sources.
- Familiarity with Git, Azure DevOps or equivalent source control and deployment practices.
- An understanding of data governance, information security and UK GDPR, including handling sensitive information.
- Strong problem-solving and communication skills, with the ability to work collaboratively across technical and service teams.
- Будет плюсом: Hands-on Microsoft Fabric experience, including lakehouses, warehouses, PySpark, Spark SQL and Delta Lake; Microsoft Purview and data governance tooling; Dynamics 365, Dataverse, Power Apps or Power Automate; ASP.NET/C# applications connected to SQL Server; Master data management, person matching, UPRN and address data management; Python, PowerShell or Azure Logic Apps; Local government or other public sector data environments; Relevant Microsoft certifications, such as DP-600 or DP-700
условия
- £56,058 –£60,393 per annum (Grade 9)
- You'll have the opportunity to help shape Havering's future data estate, working with modern Microsoft technologies while contributing to the migration and modernisation of existing systems.
- Your work will support essential public services, improve how information is shared and used, and help establish the foundations for better analytics, digital services and responsible AI adoption.
- You'll be part of a collaborative Digital function, working alongside engineers, architects, analysts and service teams, with opportunities to develop your skills, influence technical decisions and deliver meaningful improvements.