описание
In short: Data engineer for the data platform of a B2B multi-tenant transaction platform. Turn high-volume financial events (balances, transactions, payments and rewards) from many brands into data that our clients, finance and compliance teams can trust, on Kafka, ClickHouse, PostgreSQL and S3, running on AWS. Remote, mid to senior.
задачи
- Event ingestion: Kafka topics from the transactional outbox, loaded into ClickHouse by our Go ingester today, CDC from PostgreSQL next, with events that arrive late, twice or out of order, deduplicated and replayable.
- ClickHouse as the analytical core: MergeTree table engines, sort keys, materialized views, projections, partitioning and TTL-based retention.
- Financial reporting that reconciles: revenue and fees, pay-ins and payouts, rewards cost, per brand, per partner and per day in each brand's time zone, checked against the ledger to the cent.
- Regulatory and audit data: reporting for regulated markets, audit trails, data retention and deletion under privacy laws such as GDPR.
- Data products for the business: client back-office reports, CRM segments, partner and referral commissions, risk signals, and fraud and incentive-abuse features.
- Multi-tenancy: per-brand isolation in the warehouse with row policies and scoped access, never a leak between brands.
- Scaling the data stores: PostgreSQL partitioning and retention, read replicas, the sharding plan on tenant_id, Kafka (MSK) partitions and lag, ClickHouse capacity and cost.
- Backfilling and verifying the ScyllaDB and Aerospike read models before reads switch over to them.
- Building the data lake: Parquet on S3 and tiering of old partitions (not in production yet).
- Moving data safely: account and region moves with logical replication, moving one brand's data, and erasing a customer's data on request (privacy laws such as LGPD).
- Design and build batch and streaming pipelines that stay correct under replays, backfills and late data: idempotent by design, never double counting.
- Write the tests that prove it: data contracts, model tests and reconciliation checks against the ledger, written to fail first.
- Change live tables safely: schema evolution, partition swaps, backfills on a copy compared before promotion.
- Own your pipelines in production: freshness and volume SLOs, consumer lag alerts, lineage, data incident response.
- Define metrics with product, finance and analysts so revenue means one thing everywhere.
- Review code, models and designs with the team, with a strong focus on money numbers.
требования
- 5+ years building data systems; 3+ years running production pipelines in Python or Go, with SQL as a first language.
- You have built data for systems that move money or value: wallets, payments, trading or fintech, and you know why reports must reconcile.
- Streaming in practice: event time vs processing time, watermarks, deduplication, exactly-once effects over at-least-once delivery, CDC.
- Kafka in production: partitions, consumer groups, offsets, rebalances, lag.
- Strong ClickHouse or another columnar store (Druid, Pinot, Snowflake, BigQuery): table design, sort keys, materialized views, query profiling.
- Strong SQL and PostgreSQL: query plans, logical replication and replication slots, partitioning.
- Orchestration and transformation tools such as Airflow or Dagster, and dbt.
- You read a query profile and a pipeline lag graph, and you know when a pre-aggregation is the wrong fix.
- Будет плюсом: Clear written English; you explain trade-offs in a few lines; Go, to work closely with the core money services (our ingestion is written in Go today); Revenue and unit-economics metrics, customer lifetime value, partner, referral and CRM data; Debezium, Kafka Connect, Flink, schema registry, protobuf/buf (CDC and the lake are next on our plan); ScyllaDB/Cassandra, Aerospike or another distributed store in production; Rust; PostgreSQL sharding (Citus or similar) and multi-region data; Data quality and lineage: Great Expectations or Soda, OpenLineage; Kubernetes (EKS), Argo CD, Terraform, AWS with several accounts; Regulated markets, regulatory reporting, GDPR and similar privacy laws, audit trails; Feature pipelines for fraud or risk models
условия
- Fully remote work.
- A competitive salary, above the market average, set by your experience level.
- Real ownership: you design, ship and run what you build, and you see it handle real money in production.
- Room to grow: as the platform and the team grow, your scope grows with them, towards technical leadership or leading people if that is where you want to go.
- A young, fast-moving team that ships every day, keeps meetings short and writes things down.
- Commitment that gets noticed: people who care about the product and own their work are trusted with the bigger problems.
- Hard problems worth solving: real money, many brands and high volume, where correctness matters.