product analyst for conversational AI

MEDvidi is an AI-powered mental healthcare platform providing safe, effective, and scalable psychiatric care in the United States. Its proprietary AI tools automate charting, follow-ups, and treatment planning, while the AI Receptionist Team develops Voice AI Agents that interact with patients and support their healthcare journey.

• Analyze production behavior of Voice AI Agents and conversational AI workflows;
• Investigate hallucinations, inconsistencies, silent failures, and quality degradation in AI-driven conversations;
• Analyze call transcripts, logs, generated outputs, and operational datasets to uncover patterns and actionable insights;
• Design evaluation frameworks for measuring AI quality, reliability, and business impact;
• Build monitoring systems and dashboards for AI performance visibility;
• Measure the impact of AI systems on operational efficiency, business outcomes, and patient experience;
• Detect quality shifts, anomalies, and emerging failure modes in production environments;
• Partner with Product, Engineering, QA, and Operations teams on AI initiatives;
• Support rapid decision-making in fast-changing and ambiguous environments.

• 3+ Years of experience in Product Analytics, Data Analytics, Operational Analytics, or analytics roles within complex production systems;
• Strong SQL skills;
• Python for analytical investigations and working with large-scale text datasets;
• Understanding of LLM limitations, hallucinations, evaluation challenges, prompt sensitivity, and AI failure modes;
• Experience building monitoring systems, analytical frameworks, or operational dashboards;
• Strong communication skills and ability to translate analysis into product decisions;
• Ability to work effectively in ambiguous and fast-changing environments;
• Fluent Russian;
• English level B2 or higher;
• Ability to work CET-overlapping hours;
• Not suitable for candidates who prefer only structured datasets and predefined metrics, are uncomfortable analyzing raw conversations and AI-generated outputs, expect clearly defined answers instead of investigating ambiguous problems, or are not interested in understanding how AI systems fail in production;
• Nice to have: experience with conversational AI or AI agents, experience evaluating LLM outputs, experience with AI quality metrics and evaluation methodologies, experience with prompt iteration and AI evaluation frameworks, experience working with ML pipelines or AI-powered products.

• Remote-first, fully remote, long-term B2B collaboration;
• Competitive compensation and clear growth opportunities;
• Health insurance after the probation period;
• Sports and wellness compensation;
• Career growth, including personalized English lessons via Preply;
• 19 Working days of vacation per year;
• 3 Additional wellness days;
• Paid sick leave for the first 5 working days;
• Thoughtful gifts for important life events.