описание
Join the AI team at Prosus, a global technology group with businesses in food delivery, ecommerce, classifieds, travel and payments. We build LCM, the Large Commerce Model: our own foundation model, trained on billions of transactions and interactions from proprietary data across the portfolio. LCM already powers search, recommendations and marketing in production at iFood.
задачи
- Build acquisition and processing pipelines for portfolio data and relevant public and licensed sources, including connectors and content extraction.
- Turn commerce records into ordered histories and link behaviour to outcomes without leaking future information.
- Develop the shared platform for inspecting sources, configuring processing and publishing training datasets.
- Operate distributed workflows, monitor reliability and cost, and implement access controls and retention requirements.
- Work with scientists, engineers and the relevant teams on data provenance, permitted use across companies and privacy-aware handling of customer histories.
- Implement dataset versioning and lineage across source snapshots, processing code and dataset configurations, so releases can be reproduced and traced to the training runs that used them.
- Build quality checks for completeness, freshness, duplication, extraction accuracy and coverage.
- Combine automated checks with sampled review to investigate problems that aggregate metrics can miss.
- Provide tools for source tagging, sampling and weighting, and maintain evaluation datasets with temporal splits and overlap checks.
- Work with scientists on controlled comparisons to establish whether data changes improve model quality.
- Build synthetic data pipelines supporting pre- and post-training, including instruction examples, preference data and agent trajectories.
- Implement generation, validation, scoring and filtering.
- Work with the research team to measure how the generated data affects model performance, and manage generation throughput and cost.
- Build agents that turn catalogue sources into training datasets: propose tasks based on available data and target model capabilities, configure processing and generation, curate outputs and investigate quality failures.
- Keep agent actions traceable and validate proposed changes before dataset release.
требования
- At least five years of relevant data engineering experience, including ownership of substantial production pipelines.
- Strong Python and SQL, with experience writing tested, maintainable code.
- Hands-on experience with distributed processing using Spark, Ray, Beam or a comparable framework.
- Experience with cloud storage, workflow orchestration and operating pipelines through source changes, failures and backfills.
- Practical knowledge of data quality, versioning and lineage, and the ability to translate evolving research requirements into reliable implementations.
- Будет плюсом: Experience with LLM datasets, commerce data, content extraction, synthetic data generation or agents for data tasks; Experience with AWS, Databricks or EMR; Strong distributed-data engineering experience is welcome without prior experience on an LLM team
условия
- Ownership of the platform supplying LCM’s training and evaluation data, with direct collaboration with researchers and ML engineers.
- Access to proprietary commerce datasets, distributed processing infrastructure and models for generation and evaluation.
- Scope to make technical decisions and build reusable tools across portfolio companies.
- Colleagues who have released Hugging Face models with more than 100 million downloads, authored papers at NeurIPS, created benchmarks and built multiple production AI systems.
- Current ML frameworks and unlimited access to coding assistants.
- A base at our Amsterdam office and AI House, with relocation support. We also consider remote arrangements for candidates with particularly strong experience relevant to the role.
- Competitive compensation and a top-spec MacBook Pro.