automation qa

Ciklum is a custom product engineering company that supports multinational organizations and scaling startups in solving complex business challenges. The company engineers technology that redefines industries and shapes the way people live.

• Build and run automated validation test suites using Python frameworks like RAGAS to continuously score pipeline faithfulness, answer relevance, and context precision;
• Integrate automated LLM evaluation scorecards directly into Azure DevOps pipelines to prevent regression in code-generation quality;
• Develop automated checks to monitor the performance of documentation ingestion pipelines, chunking consistency, and vector store retrieval accuracy;
• Generate programmatic quality scorecards and regression reports to satisfy technical criteria at each project gate approval milestone;
• Work closely with senior and mid-level ML implementation engineers to isolate, reproduce, and patch chunking or retrieval flaws exposed by the validation metrics.

• 3+ Years of experience in backend test automation, data quality engineering, or Python software development within structured delivery environments;
• Mastery of Python for backend testing, scripting, and data manipulation;
• Direct, hands-on experience or deep technical familiarity with LLM validation frameworks, specifically RAGAS or directly equivalent automated validation engines;
• Practical experience configuring automated script execution, regression test triggers, and quality gates inside Git-based pipelines, preferably Azure DevOps;
• Basic functional understanding of relational databases (PostgreSQL), data profiling techniques, and search indexing concepts;
• Must be physically located in Poland or Eastern Europe.

• Strong community in a friendly, open-door environment;
• Large-scale projects with global impact;
• Internal events including meetups, conferences, and workshops;
• Access to Udemy, language courses, and company-paid certifications;
• Internal mobility opportunities;
• Company-paid medical insurance.