[AGNTCon MCPCon Japan] From Clicks To Context: Building an Open-Source Evaluation Pipeline for AI Agents

The AI industry has moved so fast that we are still evaluating probabilistic software using the same deterministic metrics we applied to traditional code. As a Product Analyst working on AI agents at PagerDuty, I saw a critical need for a new observability standard, one that moves beyond clicks to measure true reasoning and reliability. To address this, I’ve developed and open-sourced a specialized framework designed to help teams decide, with data, when to hire, train, or fire an AI agent. In this session, I will walk through the H.I.R.E. Framework methodology and share the technical architecture of an evaluation pipeline that turns qualitative conversational data into structured, actionable product insights. I will share the open-source repository containing these metric definitions and templates, providing resources for the community to move past agent-washing and toward building verifiable, trustworthy agentic systems.

Inês Bolaños focuses on the intersection of AI, product strategy and data reliability. With almost a decade of experience, she specializes in turning complex data into actionable product decisions. Combining a background in Communication with a Master’s in Big Data, Inês is helping teams move past vanity metrics to find focus through better questions. She is currently dedicated to defining how product analytics and storytelling must evolve to meet the demands of an AI-driven future of work.