описание
We are looking for a Senior Data Engineer to be a key technical contributor to our Data Surface programme. You will implement critical components of our AWS Lakehouse infrastructure, working with modern cloud-native technologies including Apache Iceberg, Apache Spark, and AWS services to build scalable, reliable data pipelines.
задачи
- As a Senior Data Engineer, you will design and implement data ingestion pipelines from DynamoDB and Oracle into Apache Iceberg tables on S3 using no-code capabilities.
- Use Claude Code to generate specifications and design documents, and review specifications produced by other team members to ensure AI-generated code accounts for all operational nuances of the technology
- Develop Bronze-Silver-Gold data transformation layers using Apache Spark and AWS Glue
- Build streaming data ingestion from Kinesis/Kafka into the Lakehouse with near-real-time processing
- Implement data quality frameworks, schema validation, and automated monitoring systems
- Create RESTful APIs and data access layers for consuming lakehouse data in downstream applications
- Optimize query performance using Iceberg features (partitioning, clustering, compaction)
- Build the data access and serving layers that allow AWS AI/ML services, including Amazon Bedrock and SageMaker, to consume Iceberg data directly
- Integrate data governance capabilities including metadata management, lineage tracking, and access control
- Collaborate on architectural decisions and contribute to technical design documentation
- Mentor mid-level and junior engineers through code reviews and technical guidance
- Implement infrastructure as code using Terraform, CloudFormation, or similar tools
- Design and deploy containerized applications and data processing workloads on EKS
требования
- Senior Data Engineer with 5+ years of software/data engineering experience with strong focus on data engineering and distributed systems
- Deep expertise in Python and/or Java/Scala for building production data pipelines
- Extensive hands-on experience with AWS services (S3, EKS, Glue, Kinesis, Lambda, EMR, Athena), or equivalent Lakehouse architecture builds.
- Strong experience with Apache Spark for batch and streaming data processing
- Hands-on experience with Apache Iceberg, or an equivalent open table format such as Delta Lake or Hudi, including partitioning, compaction, and table maintenance
- Proficiency in SQL and experience optimizing complex analytical queries
- Solid understanding of data modeling, schema design, and database performance tuning
- Experience with CI/CD pipelines, automated testing, and modern DevOps practices.
- Strong problem-solving skills and ability to debug complex distributed systems
- Bachelor's in Computer Science, Engineering, or related field
- Будет плюсом: Familiarity with data governance tools (Apache Atlas, AWS Glue Data Catalog); Experience working as an ML engineer, or exposure to AWS AI/ML services such as Amazon Bedrock and SageMaker; Understanding of SQL query engines (Trino, Presto, Athena, Oracle, Postgres); Experience in financial services, commodities, or energy markets, and/or working with timeseries data; AWS certifications (Solutions Architect, Developer, etc.); Background in migrating Oracle workloads to cloud-native architectures; Experience with Kubernetes and EKS for container orchestration; Track record in scale-up or fast-growth technology companies with product-focused delivery
условия
- Shape the future of energy, commodities, and financial markets through innovative technology.
- Work with cutting-edge software, advanced analytics, and complex data that solve real-world challenges.
- Join a fast-growing global organization where your ideas, development, and career progression are valued.
- Collaborate with talented colleagues and industry experts across North America, Europe, and Asia.
- Take ownership of your work, influence strategic initiatives, and make a measurable impact on our customers and business
- Be part of a collaborative culture built on trust, respect, and our values: Think Big. Make It Happen. Win as One.