AI engineer

The AI Team, part of Front Office Technology, builds the AI tools and platforms used across the front office: assistants and agents for investment teams, secure access to firm data and knowledge, and the controls that let AI be used safely in a regulated trading environment. As an AI Engineer with a focus on evals, you will help build the AI tools and platforms used across the front office, and the evidence that tells us whether they are accurate, safe and worth the spend.

• Extend our evaluation framework to cover new assistants, agents and use cases, improving its design, reliability and ease of use.
• Set the direction for AI evaluation, balancing new capability against keeping the framework simple to adopt.
• Educate engineers and the business on why evals matter, through documentation, workshops, pairing and code review.
• Help teams define what good looks like for their use cases and turn it into measurable criteria.
• Add evaluation of RAG, agents and multi-step tool use: tool-call correctness, task completion, groundedness and citation accuracy.
• Evaluate agent behaviour end to end through traces, not just final answers.
• Evaluate AI coding tools (e.g. Claude Code, Codex, GitHub Copilot) on the firm’s own codebases and tasks.
• Measure how effective these tools are in practice: task success, code quality, review effort and delivery speed, and where they help or hinder.
• Turn results into practical guidance on tool choice, configuration and ways of working across engineering.
• Build and maintain ground-truth datasets with PMs, quants and researchers.
• Keep datasets versioned, governed and reusable across teams.
• Run experiment tracking and integrate evals into CI/CD, so model, prompt and retrieval changes are gated on measured quality.
• Make the framework easy for other engineers to adopt by default.
• Support red-teaming, failure analysis and the evidence needed for governance approval before deployment.
• Monitor quality in production and catch regressions early.
• Pick up other priorities across the team as they arise, from new front office tools to knowledge services, MCP integrations and data access.
• Contribute to shared engineering standards, code review and production support.
• Measure performance, accuracy and cost together, giving clear evidence for model, provider and adoption decisions.

• You are a senior engineer with a scientist’s instinct. You want to know whether something actually works, and you design tests that would show you if it didn’t.
• You understand how LLMs fail, and you are rigorous about what a result does and does not prove.
• You turn complex results into clear recommendations for engineers, governance forums and senior stakeholders, and you are comfortable saying when a tool or model is not good enough.
• You enjoy teaching others why evals matter, and are happy to switch to whatever matters most to the team when priorities change.
• 5+ years in software engineering, with strong Python and SQL.
• Hands-on experience evaluating LLM systems: LLM-as-judge, human-labelled datasets, retrieval metrics and agent trace analysis.
• Hands-on experience with eval and observability tooling (e.g. Azure AI Foundry evals, Langfuse, LangSmith, MLflow, promptfoo, Inspect).
• Strong testing discipline and an experimental mindset: baselines, controls, significance and honest reporting of results.
• Experience building LLM applications: retrieval (RAG), tool use and agents.
• Heavy daily use of agentic coding tools, with a clear view of where they fail and how to measure them.
• A track record of shipping production systems: APIs, containers, CI/CD and cloud (Azure or Kubernetes preferred).
• The ability to explain evaluation concepts and results clearly, and to bring teams with you.
• Будет плюсом: Measuring developer productivity or the impact of engineering tooling; Benchmarking coding agents on real-world software tasks; Red-teaming and AI safety testing; Front office, market data or research experience at a hedge fund or bank

• Условий в вакансии нет