Data platform engineer

грейд
senior
формат
удалённо
язык
Английский B2
опубликована
сегодня

описание

We run many datasets that share the same shape, and we need them to answer one question as if they were one table. You will own the Trino platform that fans queries out across those shards, keeps them fast and predictable, and sits under a shared semantic layer so every metric means the same thing everywhere. This is a hands-on infrastructure role. You will deploy, tune, monitor and be on call for the query layer, not just write SQL against it.

задачи

  • Deploy and operate Trino clusters on Kubernetes (Helm, autoscaling workers, coordinator HA, upgrades with no downtime)
  • Design how shards are exposed to Trino: one catalog per shard, unified views over identical schemas, or a partitioned Iceberg table, and know when each breaks down
  • Tune partitioning, bucketing, file sizing and table statistics so queries prune shards instead of scanning them
  • Read EXPLAIN ANALYZE output to find skew, spills and missed pushdowns, then fix them in config, layout or SQL
  • Automate shard onboarding so a new shard is queryable within minutes, via Terraform, GitOps and catalog management
  • Set up resource groups, query limits and tenant isolation so one heavy user can't stall the clusters
  • Build observability for query latency, queue time, worker utilization and per-shard cost (Prometheus, Grafana, OpenTelemetry)
  • Maintain the semantic layer that sits above Trino, defining metrics and dimensions once in an open format and converting them for downstream BI and AI tools
  • Join the on-call rotation for the query platform and write the runbooks that make it boring

требования

  • Professional experience in running data infrastructure in production, including on-call ownership
  • Hands-on operation of at least one distributed query or storage system where data is sharded or partitioned across nodes: Trino, Presto, Athena, Vitess, Citus, ClickHouse (Distributed), Druid, Pinot or similar
  • A clear understanding of scatter-gather execution: partition pruning, predicate and aggregation pushdown, data skew and straggler workers
  • Table formats and catalogs: Apache Iceberg, Delta Lake or Hive, plus a metastore or REST catalog (Polaris, Glue, Nessie, Unity)
  • Kubernetes and infrastructure as code (Terraform, Helm, Argo CD or Flux)
  • Strong SQL, including reading and acting on query plans
  • A scripting language for automation (Python or Go preferred)
  • English Language Upper-Intermediate
  • Будет плюсом: Trino specifically: custom connectors, dynamic catalogs, resource groups, fault-tolerant execution, Starburst or Galaxy; Semantic-layer experience with dbt Semantic Layer / MetricFlow, Snowflake semantic views, Cube, LookML or AtScale; Exposure to Apache Ossie (incubating), the open spec for sharing metric and dimension definitions across tools. It is new, so we don't expect production experience. Reading the spec, trying a converter or following the working groups counts; Multi-tenant SaaS data platforms where each customer's data is a shard; Query-cost attribution and chargeback per tenant or shard; Contributions to Trino, Iceberg, Polaris, Ossie or another Apache data project

условия

  • International projects
  • In-office, hybrid, or remote flexibility
  • Medical healthcare
  • Recognition program
  • Ongoing learning & reimbursement
  • Well-being program
  • Team events & local benefits
  • Sports compensation
  • Referral bonuses
  • Top-tier equipment provision

навыки

компания

похожие вакансии

статьи для разработчиков