Data processing engineer

грейд
senior
формат
гибрид Берлин
страна
Германия
язык
Английский B1
опубликована
вчера

описание

Join a highly technical AI research and engineering organisation building the data foundations for next-generation AI models. As a Senior ML Data Processing Engineer, you’ll develop and optimise large-scale data pipelines that transform raw, web-scale data into high-quality training datasets. You’ll work at the intersection of data engineering, machine learning and NLP, improving data quality through automated filtering, deduplication, ML-based evaluation and privacy controls.

задачи

  • Build and optimise distributed pipelines for processing large-scale, unstructured datasets for ML training.
  • Implement data curation and quality controls, including deduplication, filtering, PII removal, toxicity detection and metadata extraction.
  • Develop automated data-quality systems using ML classifiers, heuristics and LLM-based evaluators.
  • Establish dataset versioning, lineage and reproducibility, including mechanisms to detect data leakage and evaluation contamination.
  • Monitor pipeline performance and data quality, identifying bottlenecks, regressions and opportunities to improve efficiency.
  • Collaborate with Research, Engineering and Governance teams on dataset requirements, privacy, licensing and compliance.

требования

  • PhD in a relevant technical discipline.
  • At least 3 years of relevant experience in data processing, ML engineering, NLP or a closely related field.
  • Experience working with massive, unstructured text datasets and distributed processing frameworks such as Apache Spark, Ray or Flink.
  • Strong Python skills and experience building production-grade data-processing pipelines.
  • Practical experience with data curation, deduplication, filtering, dataset quality or large-scale dataset construction.
  • Understanding of data privacy, PII removal, content filtering, evaluation contamination and dataset reproducibility.
  • Experience with workflow orchestration tools such as Airflow, Prefect or Dagster.
  • Будет плюсом: Experience with LLM training or fine-tuning, LLM inference tools such as vLLM or SGLang, Docker/Kubernetes, web-scale data acquisition, data licensing or open-source NLP projects

условия

  • Условий в вакансии нет

навыки

похожие вакансии

статьи для разработчиков