Data platform engineer for regulated environments

грейд
senior
формат
офис Лиссабон
язык
Английский B1
опубликована
12 ноя

описание

This client is building a modular, enterprise-grade data platform powered by Databricks, designed to deliver trusted, governed data through multiple consumption channels, including analytics, APIs, streaming, and knowledge graph solutions. The platform is built on a shared canonical data model and emphasizes scalability, governance, privacy, and a Data as a Product mindset. As a Senior Data Platform Engineer, you will play a key role in designing and evolving the platform across ingestion, governance, data modeling, and data products.

задачи

  • Develop and maintain modular ingestion patterns supporting batch, API-based, and event-driven data sources (e.g., JDBC, SaaS/ERP connectors, Kafka, CDC).
  • Establish reusable ingestion frameworks (metadata driven when possible) that enable rapid and consistent onboarding of new data sources.
  • Build and optimize Databricks-based data pipelines (Spark, Delta Live Tables/Lakeflow) across the medallion architecture (Bronze, Silver, Gold).
  • Implement and evolve data governance capabilities, including access controls, lineage, cataloging, auditing, data classification, and privacy compliance.
  • Define and enforce data quality standards, data contracts, monitoring, and alerting across ingestion and transformation layers.
  • Implement data protection and privacy solutions, including masking, tokenization, pseudonymization, and other appropriate anonymization techniques to meet regulatory requirements (e.g. GDPR).
  • Build, own, and evolve data products aligned to a canonical data model, ensuring clear ownership, SLAs and documentation.
  • Collaborate with domain teams to integrate source data into the canonical model while preserving domain-specific nuance.
  • Champion a Data as a Product approach: discoverability, self-service, versioning, and consumer-facing contracts.
  • Build and maintain data consumption and serving layers, including BI/analytics datasets, APIs (REST/GraphQL), and use case-driven extensions such as streaming outputs or knowledge graph representations.
  • Drive technical standards and platform direction across CI/CD, Infrastructure as Code, testing, observability, and cost optimization
  • Mentor engineers and partner with governance, security, architecture, and business stakeholders.

требования

  • Around 6+ years in data engineering, with 2+ years designing/operating Databricks-based Lakehouse platforms at production scale.
  • Hands-on expertise with Apache Spark and Delta Lake; experience with Delta Live Tables/Lakeflow Declarative Pipelines is a strong plus.
  • Practical experience with Unity Catalog (or equivalent) for access control, lineage, and cataloguing.
  • Demonstrated experience implementing data anonymisation/pseudonymisation techniques in a regulated environment.
  • Solid understanding of canonical/conformed data modelling (e.g., data vault, dimensional modelling, domain-driven design for data).
  • Track record exposing data via APIs (REST/GraphQL) - designing schemas, contracts, and versioning for external/internal consumers.
  • Proficient in Python and/or Scala, SQL, and infrastructure-as-code (Terraform preferred).
  • Strong grasp of data governance frameworks, data privacy regulation (GDPR/CCPA), and data security practices.
  • Experience with cloud platforms (Azure, AWS, or GCP) and their native data/event services.
  • Будет плюсом: Exposure to Structured Streaming or event-driven ingestion (Kafka, CDC tools such as Debezium) - not a core requirement, but useful as the platform matures; Familiarity with knowledge graph concepts (e.g., RDF/property graphs, Neo4j, graph layers on Spark) - a plus rather than an expectation; Experience with data mesh or federated data product architectures; Experience with metadata management/catalog tools beyond Unity Catalog (e.g., OpenMetadata, Collibra, Purview); Exposure to MLOps or feature store patterns on Databricks; Relevant certifications (Databricks Data Engineer Professional, cloud architecture certs)

условия

  • You will find here not only a challenging and interesting workplace, but also a rewarding work experience, with competitive compensation and benefit packages:
  • Private health insurance;
  • Career development: opportunity to grow within the team;
  • Special referral bonuses for recommending your friends;
  • Friendly team who is eager to meet you;

навыки

компания

похожие вакансии

статьи для разработчиков