ml engineer for AI agents

EPAM is building a cloud-native Enterprise Agent Development Platform that enables engineering teams to develop, deploy, and operate AI agents in production. The platform combines agent frameworks, AWS infrastructure, CI/CD, observability, and governance to make AI development faster and more reliable.

• Design and implement evaluation frameworks for LLM-based applications and AI agents;
• Develop LLM-as-a-Judge and deterministic, code-based evaluators;
• Build custom Python evaluators for quality and behavioral checks;
• Define evaluation criteria, metrics, thresholds, and acceptance rules;
• Evaluate agent behavior across individual responses, tool calls, and complete workflows;
• Work with OpenTelemetry traces and spans as evaluation data;
• Integrate evaluations into CI/CD pipelines and automated deployment gates;
• Enable continuous quality monitoring of solutions in production;
• Establish reusable evaluation patterns and engineering standards;
• Collaborate with AI Engineers, Architects, and Platform Engineers to embed quality into the development process.

• 5+ Years of experience in ML Engineering, AI Engineering, or AI Platform Engineering;
• Strong Python development experience;
• Hands-on experience with LLM/GenAI evaluation;
• Experience designing and implementing evaluation frameworks;
• Experience developing custom or deterministic evaluators;
• Experience integrating AI/ML quality checks into CI/CD;
• Good understanding of LLM and AI agent architectures;
• Nice to have: Hands-on experience with AWS AgentCore Evaluation, AWS Bedrock Guardrails including PII detection, knowledge of CloudWatch metrics and production monitoring, experience with OpenTelemetry, familiarity with LangGraph, Strands Agents, or similar agent frameworks.

• Location-specific conditions and benefits may apply.