AI engineer

You will own the measurement layer: the Python evaluation harnesses that determine whether complex, non-deterministic agent workflows are genuinely improving or simply gaming metrics. This is not prompt engineering or subjective vibe-checking; it is rigorous experimental design, building calibrated LLM judges, isolating signal from probabilistic noise across repeated runs, and turning failure analyses into concrete upgrades across agent scaffolding, tool loops, and guardrails.

• Build and maintain the core Python evaluation harnesses scoring task completion, trajectory quality, tool-use correctness, cost, latency, and variance across repeated runs of non-deterministic agents.
• Apply statistical methods including hypothesis testing, confidence intervals, bootstrapping, and power analysis to reliably distinguish genuine performance gains from random variance.
• Develop, audit, and calibrate LLM-as-a-judge pipelines against human annotations, systematically detecting and mitigating biases in position, verbosity, and self-preference.
• Create golden evaluation datasets, scoring rubrics, and annotation guidelines from scratch while managing inter-annotator agreement and label noise.
• Run systematic root-cause analyses on agent failure modes and translate your findings into architectural improvements across agent scaffolding, context management, retry logic, and safety guardrails.

• Strong background in statistics and experimental design applied to probabilistic machine learning systems, with the ability to design outcome-driven metrics and detect benchmark gaming.
• Proven track record evaluating multi-step, non-deterministic agent systems across repeated runs, including systematic error analysis and failure clustering.
• Hands-on experience developing and calibrating LLM-as-a-judge pipelines against human ground truth, with deep attention to position, verbosity, and self-preference biases.
• Production-grade Python engineering skills working directly with LLM APIs, structured outputs, tool orchestration, retrieval, and agent memory architectures.

• Условий в вакансии нет