описание
Nebius is leading a new era in cloud infrastructure for the global AI economy. We are building a full-stack AI cloud platform that supports developers and enterprises from data and model training through to production deployment, without the cost and complexity of building large in-house AI/ML infrastructure. In a rapidly evolving world, trust in AI depends on AI agents being grounded in fresh, verified real-world data. Search is the foundation that makes this possible.
задачи
- Design, implement, and operate web-scale crawling systems for acquiring content from the internet
- Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations
- Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies
- Build systems for URL discovery, deduplication, content extraction, and crawl orchestration
- Ensure reliable operation of crawling infrastructure under high-throughput conditions
- Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality
- Monitor resource usage, bandwidth consumption, and infrastructure cost
- Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements
- Enable safe experimentation with crawling strategies and content acquisition policies
требования
- 8+ years of experience building backend or distributed systems
- Strong Go or C++ expertise
- Experience with large-scale distributed systems (10k+ RPS, billions of URLs, high-throughput pipelines)
- Understanding of web protocols (HTTP, DNS, TLS), crawling, scraping, and content extraction
- Experience operating production systems and debugging failures in distributed environments
- Strong understanding of scalability, fault tolerance, and resource management
- Будет плюсом: Web crawling; Building streaming data pipelines and event-driven systems; Kafka, Pulsar, NATS, RabbitMQ, or similar messaging platforms; Designing distributed schedulers, queues, and asynchronous processing systems; Spark, Flink, Beam, or MapReduce; Ad tech, social networks, search engines, or other large-scale content platforms
условия
- Competitive compensation
- Career growth and learning opportunities
- Flexibility and ownership
- Collaborative and innovative culture
- Opportunity to work on impactful AI projects
- International environment and talented teams
- Fast moving - Bold thinking - Constant growth - Meaningful impact - Trust and real ownership - Opportunity to shape the future of AI