hebel. today · 12 November 202624 / 74
Agentic Engineering · AI engineering
Deterministic verification outpaces model confidence
Verification over confidence. OpenAI recently processed 2.8 million lines of the Linux kernel to flag 2,308 potential bugs, yet the team discarded most of these after a sandboxed machine failed to reproduce them. Only…
From 5 newsletters · 1 min read · one thing to try
Verification over confidence. OpenAI recently processed 2.8 million lines of the Linux kernel to flag 2,308 potential bugs, yet the team discarded most of these after a sandboxed machine failed to reproduce them (Cloud Girl). Only 948 bugs were confirmed through actual crashes, demonstrating that deterministic reproduction is more reliable than model-generated warnings. Netflix and Google similarly adopt this approach by requiring non-AI code to validate performance fixes or security flaws before deployment.
Harness-first benchmarking. Developers now use tools like Harness Arena to evaluate how different harnesses affect agent performance, cost, and runtime (Daily Dose of Data Science). In one test, an expensive configuration failed two out of five runs while a cheaper pairing succeeded every time, proving that the harness personality often dictates success. GitHub now automates this selection process through Copilot Fusion, which routes tasks to models based on efficiency and cost metrics (ByteByteGo Newsletter).
Tracing failure chains. Researchers at Heidelberg University and Singapore Management University analyzed 36,487 repositories and found that only 0.59% of agents included essential reliability features like execution limits or persistent state (Super Intelligence School). When agents are chained together, failures often hide in the gaps between services where standard logging fails to capture the full context (O'Reilly Radar The latest on the trends…). Engineers now prioritize trace context that survives the entire path to prevent silent output degradation.
- Cemri and colleagues identified 14 recurring failure modes in multi-agent frameworks by analyzing 1,600 execution traces (O'Reilly Radar The latest on the trends…).
- GitHub Copilot Fusion scored within two points of Claude Opus 5.5 Max on coding benchmarks while using half the cost (ByteByteGo Newsletter).
Try today · 45 min
Build a deterministic verification script for your agent's output
Create a sandbox test that executes the agent's proposed code and checks for a specific exit code or file state.
You need
- A sandboxed environment
- A script that runs the agent's proposed solution
- A validation function that checks for a successful outcome
Steps
- Isolate the agent's output into a standalone execution file.
- Write a test script that runs this file in a restricted sandbox.
- Define a pass condition based on a concrete outcome like a file creation or a return code.
- Configure the agent to only commit results that trigger this pass condition.