automation qa

EPAM provides digital platform engineering and software development services, focusing on complex technology solutions and enterprise-grade product development.

• Design and automate evaluation of RAG-based and LLM-driven features including grounding, answer accuracy, determinism/reproducibility, precision, recall and hallucination rate;
• Build test harnesses to scale evaluation beyond human-in-the-loop processes;
• Create data-driven test suites for underwriting rules and validate rule execution via SPARQL evaluator, arithmetic/threshold logic and multi-condition scenarios;
• Verify ontology schema correctness and instance accuracy against source data, perform reasoner consistency checks and SHACL validation;
• Define and drive end-to-end QA strategy covering data ingestion, ontology/graph layer, rule evaluation engine, AI/LLM layer and integration with underwriting solutions;
• Establish quality gates, acceptance criteria and test coverage models;
• Maintain automation test suites across knowledge graph, data ontology layer, backend services and frontend layers;
• Validate end-to-end flows across ontology, rule engine and decision output, perform contract testing and cross-system data consistency validation;
• Integrate test suites into CI/CD pipelines and define release quality gates;
• Champion automation-first and shift-left quality practices;
• Leverage agentic AI and Gen AI tooling in testing and framework development.

• 8+ Years of experience in software development, testing automation and DevOps;
• Proficiency in Python test automation using pytest or equivalent, scripting and AI/ML library integration;
• Expertise in testing Knowledge Graph or Data Ontology solutions using OWL2, RDF and SPARQL;
• Skills in SHACL validation and graph databases such as GraphDB, including ontology validation, entity/relationship integrity, reasoner consistency checks and graph query correctness;
• Proven ability to define and drive QA strategy independently for complex data-heavy systems, including building automation frameworks and implementing quality gates;
• Skills in data validation using SQL and graph query languages such as SPARQL and Cypher, along with understanding of deterministic vs probabilistic systems;
• Background in backend and API testing (REST) covering data validation, integration and E2E testing;
• Proficiency in frontend web application test automation using Playwright or equivalent Python-compatible framework;
• Hands-on experience using coding agents and agentic development daily;
• Demonstrated experience testing and evaluating RAG-based Gen AI / LLM applications including grounding, answer accuracy and hallucination/determinism checks;
• Applied knowledge of Gen AI / LLM evaluation frameworks and metrics such as precision, recall, criteria recall and efficiency, with proven ability to automate Gen AI/RAG evaluation at scale;
• Experience in Agile delivery using Git and Azure DevOps or JIRA;
• English proficiency at B2 level or higher;
• Nice to have: Skills in semantic search testing and evaluation, familiarity with Vector Database integration and retrieval validation, background in data science covering ML concepts, data pipelines or data engineering collaboration, proficiency in API tooling such as Postman and Bruno, experience with quality-gate implementation in CI/CD pipelines such as Azure DevOps or Jenkins.

• No conditions specified