ai engineer in AI evaluation

Plexus has partnered with an early-stage AI company building an evaluation platform that helps enterprise teams validate generative AI models and agents before they go live. The product is live and includes a public leaderboard benchmarking leading LLMs.

• Build scalable pipelines for benchmark execution, scoring, and results aggregation across multiple LLM providers and custom endpoints
• Develop systems that generate custom evaluations on demand from customer data
• Design secure, multi-tenant services and APIs that meet enterprise privacy standards
• Own orchestration, observability, performance, and cost as usage grows

• Proven experience building AI systems and AI infrastructure in production
• Strong Python or similar skills, with backend or distributed systems experience
• Hands-on experience with LLM APIs and open or self-hosted models
• Experience with AWS, GCP, or Azure, Docker/Kubernetes, and infrastructure as code
• Comfortable working in an early-stage, fast-moving environment
• Будет плюсом: Experience with LLM evaluation, benchmarking, or agent frameworks; startup or scale-up background

• Base salary plus equity