описание
Join a highly technical AI research and engineering organisation building the data foundations for next-generation AI models. As a Senior ML Data Processing Engineer, you’ll develop and optimise large-scale data pipelines that transform raw, web-scale data into high-quality training datasets. You’ll work at the intersection of data engineering, machine learning and NLP, improving data quality through automated filtering, deduplication, ML-based evaluation and privacy controls.
задачи
- Build and optimise distributed pipelines for processing large-scale, unstructured datasets for ML training.
- Implement data curation and quality controls, including deduplication, filtering, PII removal, toxicity detection and metadata extraction.
- Develop automated data-quality systems using ML classifiers, heuristics and LLM-based evaluators.
- Establish dataset versioning, lineage and reproducibility, including mechanisms to detect data leakage and evaluation contamination.
- Monitor pipeline performance and data quality, identifying bottlenecks, regressions and opportunities to improve efficiency.
- Collaborate with Research, Engineering and Governance teams on dataset requirements, privacy, licensing and compliance.
требования
- PhD in a relevant technical discipline.
- At least 3 years of relevant experience in data processing, ML engineering, NLP or a closely related field.
- Experience working with massive, unstructured text datasets and distributed processing frameworks such as Apache Spark, Ray or Flink.
- Strong Python skills and experience building production-grade data-processing pipelines.
- Practical experience with data curation, deduplication, filtering, dataset quality or large-scale dataset construction.
- Understanding of data privacy, PII removal, content filtering, evaluation contamination and dataset reproducibility.
- Experience with workflow orchestration tools such as Airflow, Prefect or Dagster.
- Будет плюсом: Experience with LLM training or fine-tuning, LLM inference tools such as vLLM or SGLang, Docker/Kubernetes, web-scale data acquisition, data licensing or open-source NLP projects
условия
- Условий в вакансии нет