Data engineer in observability platforms

The Data Engineer (Senior) is a core contributor to the Data Observability RFQ — DEPOT platform. In this senior-level role, you will architect and implement scalable data pipelines, data models, and transformation logic that power the DEPOT observability platform, enabling real-time data quality monitoring and anomaly detection across enterprise data ecosystems. You will drive data architecture decisions, establish engineering best practices, and serve as a technical mentor to junior engineers within the team.

• Design, build, and optimize scalable data ingestion pipelines and ELT/ETL workflows that feed the DEPOT observability platform, ensuring high throughput, reliability, and data quality at scale.
• Develop and maintain data transformation logic using dbt and SQL, implementing dimensional models, data vault patterns, and incremental load strategies across cloud data warehouses and lakehouses.
• Architect and manage real-time and batch data streaming workflows using Apache Kafka and Apache Spark, integrating diverse enterprise data sources into the DEPOT pipeline ecosystem.
• Own pipeline orchestration using Apache Airflow — designing DAG structures, managing dependencies, implementing alerting, and continuously improving pipeline observability and resiliency.
• Collaborate with platform engineers, AI engineers, and DevOps teams to integrate data engineering components into the broader DEPOT observability platform and ensure seamless data availability.
• Lead data architecture discussions, contribute to technical design documents, and participate in code reviews to uphold engineering standards and data modeling best practices across the team.
• Instrument data quality checks, freshness monitors, and lineage tracking across pipelines, partnering with observability tooling (e.g., Great Expectations, Monte Carlo) to proactively detect and resolve data issues.
• Mentor junior data engineers through code reviews, pair programming, and knowledge-sharing sessions; contribute to internal documentation, reusable templates, and DataOps tooling.

• Proven experience designing and implementing production-grade data pipelines, ELT/ETL workflows, and data models in enterprise environments
• Deep proficiency in SQL and Python for data transformation, pipeline scripting, and automation
• Strong expertise with Apache Spark (PySpark/Scala)
• Strong expertise with Apache Airflow (DAG design, operators, sensors)
• Strong expertise with dbt (models, tests, macros, incremental strategies)
• Strong expertise with Apache Kafka (producers/consumers, topic management, streaming patterns)
• Hands-on experience with cloud-native data platforms such as Snowflake, Databricks, BigQuery, or Azure Synapse
• Familiarity with cloud infrastructure services (AWS, Azure, or GCP) for storage, compute, and data integration
• Experience with version control (Git)
• Experience with CI/CD practices for data workflows
• Experience with containerized environments (Docker/Kubernetes)
• Experience with data observability or data quality tooling (Great Expectations, Monte Carlo, Soda, or similar)
• Будет плюсом: Experience working within a data observability platform context — instrumenting metadata, lineage, freshness, and quality metrics across pipelines; Familiarity with DataOps frameworks, automated data testing, and schema evolution management in production environments; Exposure to advanced Kafka ecosystem tooling (Kafka Streams, ksqlDB, Confluent Platform), event-driven architecture patterns, and real-time analytics use cases; Knowledge of data lakehouse architectures (Delta Lake, Apache Iceberg, or Apache Hudi); Experience with infrastructure-as-code (Terraform, Pulumi) for data platform provisioning

• Contract
• Fully remote work