search crawler analyst

Perplexity’s crawling and storage system processes URLs through multiple stages, including discovery, crawling, parsing, and indexing. The company’s search and answer systems rely on this pipeline.

• Find and diagnose quality issues in the crawling pipeline
• Train small models to optimize specific aspects of the pipeline, such as parsing quality
• Build datasets for model training, including LLM-as-a-judge labeling pipelines
• Improve page selection algorithms for indexing
• Design and analyze experiments to validate improvements

• 4+ Years of experience as a data analyst, ML engineer, or in a related role
• Strong coding skills and ability to write production-grade code at the level of a mid-level backend engineer
• Experience designing metrics from scratch
• Experience training ML models that have shipped to production with measurable metric improvements
• Будет плюсом: Direct experience working on web crawling or indexing pipelines

• Условий в вакансии нет