описание
We’re looking for a hands-on Applied AI Engineer to own the quality of Handle’s AI agent. You’ll expand and run the agent’s evals and benchmarks, drive the hill-climb loop on its skills, and help make the agent reliable, useful, and competitive.
задачи
- Build and own the eval suite: scenarios, rubrics, graders, benchmarks, and quality gates.
- Form research hypotheses, run controlled experiments, and separate model effects from prompt, tool, skill, and agent-structure effects.
- Version skills and changes, rerun the full evaluation suite, diagnose regressions, and follow improvements through to verified results.
- Evaluate conversation quality, style, user understanding, and task reliability.
- Test meaningful new models and assess portability, failure modes, and when a simpler single-model strategy is better.
- Recommend agent or skill restructuring when repeated failures show that local fixes are not enough.
требования
- Proven experience in building and running evals for an LLM product in production, ability to write the code.
- Deep interest in agent behavior, model capabilities, and evaluation.
- Enough technical fluency to run experiments, inspect failures, and collaborate closely with engineers.
- Ability to move between theory and concrete product problems.
- Proactive taste: you notice patterns, propose better approaches, and follow them through.
- Будет плюсом: Hands-on post-training or fine-tuning of open-weight models
условия
- At On The Spot, we’re creating a workplace people would want to stay in: supportive, well-organized, and fair in terms of conditions and benefits.
- We’ll share more about our benefit package during the interviews.