Data engineer in payments

We are seeking a Data Engineer (Databricks / Spark) to join a lean, engineering-focused team building a new data platform for a global payments technology company. In this role, you will work closely with our Tech Lead and the client’s engineers to build Databricks on AWS pipelines across a medallion architecture, develop ClickHouse structures for real-time analytics, and own the quality of your code end to end, from automated tests to data quality checks, while giving pragmatic support to a legacy Hadoop platform.

• Build pipelines and data transformations on Databricks on AWS, developing Bronze, Silver and Gold layers as part of a platform that unifies the engineering approach across many consumers and use cases.
• Develop and optimize Spark / PySpark workloads, including those behind monthly reports for around 40,000 merchants.
• Build ClickHouse ingestion and analytical structures for low-latency use cases.
• Implement automated tests, integration tests and data quality checks yourself, with no dedicated QA on the team, and take part in code reviews.
• Troubleshoot production issues and deliver targeted fixes in a legacy Hadoop platform with file-based sources and Kafka streams (around 10% of your time).
• Contribute to shared engineering patterns that become the standard for the new platform, with potential growth into AI/ML and MLOps.

• Commercial experience in data engineering, building production data pipelines and transformations.
• Hands-on experience building pipelines and workloads on Databricks.
• Strong practical skills in developing and optimizing Spark workloads.
• Strong programming skills in Python and SQL.
• Ability to own automated testing and data quality validation as part of everyday delivery.
• Будет плюсом: Strong experience building automated testing for data platforms; Knowledge of the Hadoop ecosystem, Kafka and streaming data; Experience with ClickHouse for real-time or low-latency analytics; Fundamentals of AI/ML and MLOps, including Databricks MLOps; Experience with AWS, Delta Lake, CI/CD and performance optimization

• Opportunity to work on bleeding-edge projects
• Work with a highly motivated and dedicated team
• Competitive salary
• Flexible schedule
• Benefits package - medical insurance, sports
• Corporate social events
• Professional development opportunities
• Well-equipped office