Data engineer

грейд
senior
формат
гибрид Лондон
язык
Английский B1
опубликована
сегодня

описание

We are looking for a Senior Data Engineer to be a key technical contributor to our Data Surface programme. You will implement critical components of our AWS Lakehouse infrastructure, working with modern cloud-native technologies including Apache Iceberg, Apache Spark, and AWS services to build scalable, reliable data pipelines.

задачи

  • As a Senior Data Engineer, you will design and implement data ingestion pipelines from DynamoDB and Oracle into Apache Iceberg tables on S3 using no-code capabilities.
  • Use Claude Code to generate specifications and design documents, and review specifications produced by other team members to ensure AI-generated code accounts for all operational nuances of the technology
  • Develop Bronze-Silver-Gold data transformation layers using Apache Spark and AWS Glue
  • Build streaming data ingestion from Kinesis/Kafka into the Lakehouse with near-real-time processing
  • Implement data quality frameworks, schema validation, and automated monitoring systems
  • Create RESTful APIs and data access layers for consuming lakehouse data in downstream applications
  • Optimize query performance using Iceberg features (partitioning, clustering, compaction)
  • Build the data access and serving layers that allow AWS AI/ML services, including Amazon Bedrock and SageMaker, to consume Iceberg data directly
  • Integrate data governance capabilities including metadata management, lineage tracking, and access control
  • Collaborate on architectural decisions and contribute to technical design documentation
  • Mentor mid-level and junior engineers through code reviews and technical guidance
  • Implement infrastructure as code using Terraform, CloudFormation, or similar tools
  • Design and deploy containerized applications and data processing workloads on EKS

требования

  • Senior Data Engineer with 5+ years of software/data engineering experience with strong focus on data engineering and distributed systems
  • Deep expertise in Python and/or Java/Scala for building production data pipelines
  • Extensive hands-on experience with AWS services (S3, EKS, Glue, Kinesis, Lambda, EMR, Athena), or equivalent Lakehouse architecture builds.
  • Strong experience with Apache Spark for batch and streaming data processing
  • Hands-on experience with Apache Iceberg, or an equivalent open table format such as Delta Lake or Hudi, including partitioning, compaction, and table maintenance
  • Proficiency in SQL and experience optimizing complex analytical queries
  • Solid understanding of data modeling, schema design, and database performance tuning
  • Experience with CI/CD pipelines, automated testing, and modern DevOps practices.
  • Strong problem-solving skills and ability to debug complex distributed systems
  • Bachelor's in Computer Science, Engineering, or related field
  • Будет плюсом: Familiarity with data governance tools (Apache Atlas, AWS Glue Data Catalog); Experience working as an ML engineer, or exposure to AWS AI/ML services such as Amazon Bedrock and SageMaker; Understanding of SQL query engines (Trino, Presto, Athena, Oracle, Postgres); Experience in financial services, commodities, or energy markets, and/or working with timeseries data; AWS certifications (Solutions Architect, Developer, etc.); Background in migrating Oracle workloads to cloud-native architectures; Experience with Kubernetes and EKS for container orchestration; Track record in scale-up or fast-growth technology companies with product-focused delivery

условия

  • Shape the future of energy, commodities, and financial markets through innovative technology.
  • Work with cutting-edge software, advanced analytics, and complex data that solve real-world challenges.
  • Join a fast-growing global organization where your ideas, development, and career progression are valued.
  • Collaborate with talented colleagues and industry experts across North America, Europe, and Asia.
  • Take ownership of your work, influence strategic initiatives, and make a measurable impact on our customers and business
  • Be part of a collaborative culture built on trust, respect, and our values: Think Big. Make It Happen. Win as One.

навыки

похожие вакансии

статьи для разработчиков