Data platform engineer in lakehouse

At LUZA Group, passion, perseverance, and the desire to push boundaries define our path to success. Founded in 2006, we are a Portuguese multinational with more than 1,200 talented professionals and a turnover exceeding €30M. We offer innovative solutions in engineering, IT, design, consulting, Industry 4.0, training, and recruitment. We are looking for a Senior Data Platform Engineer with experience in Databricks, data engineering, and architecture, capable of designing and developing scalable, secure, and governed data platforms that promote reuse, data quality, and good engineering practices.

• Develop and maintain modular data ingestion standards, supporting batch sources, API-based and event-driven sources (for example, JDBC, SaaS/ERP connectors, Kafka, and CDC).
• Create reusable, preferably metadata-driven ingestion frameworks that enable the rapid and consistent integration of new data sources.
• Develop and optimize data pipelines in Databricks (Spark, Delta Live Tables/Lakeflow) across the Medallion architecture (Bronze, Silver, and Gold).
• Implement and evolve data governance mechanisms, including access controls, lineage, cataloging, auditing, data classification, and compliance with privacy requirements.
• Define and ensure compliance with data quality standards, data contracts, monitoring mechanisms, and alerts across the different ingestion and transformation layers.
• Implement data protection and privacy solutions, including masking, tokenization, pseudonymization, and other appropriate anonymization techniques, to meet regulatory requirements, namely GDPR.
• Develop, manage, and evolve data products aligned with a canonical data model, ensuring clear definition of responsibilities, SLAs, and documentation.
• Collaborate with domain teams to integrate source data into the canonical model while preserving each domain's specific characteristics.
• Promote a Data as a Product approach, ensuring data discoverability, self-service access, version control, and the definition of consumer-oriented data contracts.
• Develop and maintain data delivery and consumption layers, including datasets for BI and analytics, APIs (REST/GraphQL), and extensions for specific use cases, such as streaming outputs or knowledge graph representations (Knowledge Graphs).
• Define and promote technical standards and platform evolution in the areas of CI/CD, Infrastructure as Code, testing, observability, and cost optimization.
• Support and guide other engineers, collaborating with governance, security, and architecture teams and with different business stakeholders.

• Around 6 or more years of experience in Data Engineering, including at least 2 years designing and operating Databricks-based Lakehouse platforms in production and enterprise-scale environments.
• Solid hands-on experience with Apache Spark and Delta Lake. Experience with Delta Live Tables/Lakeflow Declarative Pipelines will be an asset.
• Hands-on experience with Unity Catalog (or equivalent tools) for managing access controls, lineage, and data cataloging.
• Proven experience implementing data anonymization and pseudonymization techniques in regulated environments.
• Good knowledge of canonical and conformed data modeling, namely Data Vault, dimensional modeling, and applying Domain-Driven Design (DDD) principles to data.
• Experience delivering data through APIs (REST/GraphQL), including schema design, data contracts, and version management for internal and external consumers.
• Good knowledge of Python and/or Scala, SQL, and Infrastructure as Code, preferably Terraform.
• Strong knowledge of data governance frameworks, privacy regulations (GDPR and CCPA), and data security practices.
• Experience with cloud platforms (Azure, AWS, or GCP) and their native data and event-processing services.
• Будет плюсом: Experience with Delta Live Tables/Lakeflow Declarative Pipelines will be an asset

• Условий в вакансии нет