The Hard Part Isn’t Building the Agent: Measuring Effectiveness

Joshua Saxe, AI Security Technical Lead, Meta, speaks at [un]prompted 2026 on: The Hard Part Isn't Building the Agent: On Measuring Agent Effectiveness to Improve It. As AI coding tools drive the cost of building security agents toward zero, the har...

Joshua Saxe, AI Security Technical Lead, Meta, speaks at [un]prompted 2026 on: The Hard Part Isn't Building the Agent: On Measuring Agent Effectiveness to Improve It.

As AI coding tools drive the cost of building security agents toward zero, the hard problem becomes knowing whether they'll actually work in the wild against real attacks and vulnerabilities we haven't seen before. This talk shares a practical journey from naive precision/recall metrics on old data toward multi-dimensional evaluation that captures reasoning quality, evidence gathering, and tool-calling logic --and shows how proper measurement unlocks automated agent improvement using genetic algorithms and AI coding tools. Live demo included.