AI call quality analyst (manual) in health-tech

A stealth-mode AI-powered Cloud-Native Health-Tech company is looking for an AI Call Quality Analyst. The Business Domain revolves around providing Better Care for the US Patients; meaning, it’s about Treating Patients well. The company was founded more than five years ago. It’s not vaporware, their platform supports US Physician Networks (IPAs) by enabling Smarter, Risk-Adjusted, and more Predictive Care that improves real patient outcomes.

• Own the quality of our AI-driven patient communications - define what "good" means, catch where it breaks, and drive the fixes across the agent, the flow, and the checks.
• review AI voice/chat interactions and score them against a defined behavioural rubric.
• own the automated checks and the human review loop; keep the quality signal trustworthy.
• turn findings into concrete changes to the agent, the conversation flow, and the test suite.
• Define and maintain the quality rubric - the behavioural checks (off-script, hallucination, resolution, emergency handling, "cringe"/weird behaviour) and their pass/fail criteria.
• Monitor production calls - review a smart sample, run human annotation, and measure & correct the drift of the automated LLM judge.
• Root-cause and route every defect - decide whether it belongs to the agent prompt, the conversation flow, the data pipeline, or the telephony config.
• Drive fixes with the AI team - bring prioritized, reproducible findings to engineers, propose the prompt change where the defect sits in a prompt, and add regression/autotest cases before each release.
• Align changes with the business - agree flow and script changes with clinical/ops stakeholders, balancing patient experience, compliance, and outcomes.
• Report quality in plain language - trends, dashboards, and release-readiness calls for non-technical stakeholders.

• 3+ years in QA / quality analysis of customer communications (contact-center, CX, or conversational AI) - monitoring and scoring live interactions against rubrics, giving actionablefeedback, and owning it through to verified implementation (not just reporting – confirming the fix shipped and worked).
• Experience running an evaluation loop for LLM/agent quality - LLM-as-a-judge, human annotation, regression on golden sets.
• Comfortable reading transcripts and structured logs (JSON) and using basic SQL / data literacy to slice metrics.
• Hands-on prompt engineering - writing, versioning and iterating production prompts: system prompts for LLM voice/chat agents, and evaluator prompts for LLM-as-a-judge checks.
• Hands-on experience evaluating voice and/or chat AI agents (RetellAI or similar LLM-agent platforms).
• Practical understanding of conversation flows, intents, and failure modes - barge-in, ASR/TTS errors, voicemail handling, escalation and fallback paths.
• Strong analytical and communication skills - translate technical findings into business impact and back and own the quality bar across teams without formal authority (bridge business ↔ AI team).
• English - C1 or higher; ability to present work and lead discussions with US-based teammates, customers, and stakeholders.
• Fluent Russian for collaboration with Russian-speaking teammates.
• Будет плюсом: Experience in Healthcare, Health-Tech, and MedTech is a major advantage; Spanish - a plus; our patients are served in English and Spanish

• The company has Unlimited PTOs Policy and compensated New Years Holidays on top of that. The misuse of the policy isn’t welcomed, though it’s definitely possible to take at least two weeks – and fully compensated – vacation, or more.
• The company provides Corporate Hardware for employees who completed their Probationary Period, as well as proven their value.
• We use Google Workspace, Slack, Zoom, and similar collaboration tools for messaging, meetings, and document sharing across the organization. We don’t use Microsoft Outlook, Microsoft Team and similar software.
• There’s real ability to make most solutions in a Cloud-Native and Third-Party Integrations manner, rarely spinning something self-hosted (e.g., over GKE). Fully Serverless Approach, based on Cloud Functions and Cloud Run is definitely not welcomed.
• The company partially compensates Team Building events, when multiple teammates are located in nearby countries.