Data engineer

грейд
senior
формат
удалённо
язык
Английский B1
опубликована
вчера

описание

Удалённый формат рассматривают для кандидатов с особенно релевантным опытом.

Join the AI team at Prosus, a global technology group with businesses in food delivery, ecommerce, classifieds, travel and payments. We build LCM, the Large Commerce Model: our own foundation model, trained on billions of transactions and interactions from proprietary data across the portfolio. LCM already powers search, recommendations and marketing in production at iFood.

задачи

  • Build acquisition and processing pipelines for portfolio data and relevant public and licensed sources, including connectors and content extraction.
  • Turn commerce records into ordered histories and link behaviour to outcomes without leaking future information.
  • Develop the shared platform for inspecting sources, configuring processing and publishing training datasets.
  • Operate distributed workflows, monitor reliability and cost, and implement access controls and retention requirements.
  • Work with scientists, engineers and the relevant teams on data provenance, permitted use across companies and privacy-aware handling of customer histories.
  • Implement dataset versioning and lineage across source snapshots, processing code and dataset configurations, so releases can be reproduced and traced to the training runs that used them.
  • Build quality checks for completeness, freshness, duplication, extraction accuracy and coverage.
  • Combine automated checks with sampled review to investigate problems that aggregate metrics can miss.
  • Provide tools for source tagging, sampling and weighting, and maintain evaluation datasets with temporal splits and overlap checks.
  • Work with scientists on controlled comparisons to establish whether data changes improve model quality.
  • Build synthetic data pipelines supporting pre- and post-training, including instruction examples, preference data and agent trajectories.
  • Implement generation, validation, scoring and filtering.
  • Work with the research team to measure how the generated data affects model performance, and manage generation throughput and cost.
  • Build agents that turn catalogue sources into training datasets: propose tasks based on available data and target model capabilities, configure processing and generation, curate outputs and investigate quality failures.
  • Keep agent actions traceable and validate proposed changes before dataset release.

требования

  • At least five years of relevant data engineering experience, including ownership of substantial production pipelines.
  • Strong Python and SQL, with experience writing tested, maintainable code.
  • Hands-on experience with distributed processing using Spark, Ray, Beam or a comparable framework.
  • Experience with cloud storage, workflow orchestration and operating pipelines through source changes, failures and backfills.
  • Practical knowledge of data quality, versioning and lineage, and the ability to translate evolving research requirements into reliable implementations.
  • Будет плюсом: Experience with LLM datasets, commerce data, content extraction, synthetic data generation or agents for data tasks; Experience with AWS, Databricks or EMR; Strong distributed-data engineering experience is welcome without prior experience on an LLM team

условия

  • Ownership of the platform supplying LCM’s training and evaluation data, with direct collaboration with researchers and ML engineers.
  • Access to proprietary commerce datasets, distributed processing infrastructure and models for generation and evaluation.
  • Scope to make technical decisions and build reusable tools across portfolio companies.
  • Colleagues who have released Hugging Face models with more than 100 million downloads, authored papers at NeurIPS, created benchmarks and built multiple production AI systems.
  • Current ML frameworks and unlimited access to coding assistants.
  • A base at our Amsterdam office and AI House, with relocation support. We also consider remote arrangements for candidates with particularly strong experience relevant to the role.
  • Competitive compensation and a top-spec MacBook Pro.
Удалённый формат рассматривают для кандидатов с особенно релевантным опытом. Удалённый формат рассматривают для кандидатов с особенно релевантным опытом.

навыки

компания

похожие вакансии

статьи для разработчиков