Спроси Хайрика про вакансию
Сверит с твоим резюме, подскажет вилку и вопросы на собесе.
At Pencil, we are driving innovation in advertising technology through our state-of-the-art SaaS product, which harnesses Generative AI to redefine content creation. Our mission is to make AI the default in advertising without replacing creative people. To achieve this, we need to make sure that our technology isn't just in the hands of big brands - we need to try to help small businesses and creative individuals too.
Pencil's agents produce advertising creative at scale - video, adaptation and multiple formats, orchestrated by our agentic system, Scribble. The product only works if the output is right first time. This role owns measuring quality and improving it.
You'll own evals end to end. Internally, that means the systems that measure whether agents, skills and workflows produce work a client would ship. Externally, it means how we demonstrate quality to customers and the market: the evidence behind client RFP responses, creative enablement packages, and adevals.ai, our public evals site.
Measuring creative quality is hard because there is no single right answer. Your job is to build reliable measurement anyway: rubrics that define what good means, calibration between human reviewers and automated judges, and a golden dataset that stays representative as the product and client base evolve. You'll then make those metrics central to how the business operates and sells.
