hebel. today · 13 November 20261 / 80
Agentic Engineering · AI engineering
Build production agents or buy off-the-shelf tools
Build for custom workflows. Forward deployed engineers at AI labs integrate bespoke sorters directly into customer systems to ensure production reliability. These engineers treat model swaps as component changes, maintaining agent identity through externalized…
From 9 newsletters · 1 min read · one thing to try
Build for custom workflows. Forward deployed engineers at AI labs integrate bespoke sorters directly into customer systems to ensure production reliability (The AI Engineer, The AI Engineer). These engineers treat model swaps as component changes, maintaining agent identity through externalized persona, memory, and verification rules (Cobus Greyling on AI, Agents, Loop &…). By embedding agents into existing tools like claims software, teams achieve genuine production adoption rather than relying on isolated demos. This approach prioritizes control over the entire stack, allowing for per-model routing and rigorous evaluation against real-world data.
Buy for operational speed. Building custom software often consumes excessive time and resources, yielding half-baked tools that require constant maintenance (Cam Trew from Artificial Wisdom). If a suitable commercial tool exists at a reasonable price, paying for it allows teams to focus on their core business rather than infrastructure. Platforms like Dots or Framer MCP offer immediate integration with existing environments, letting users leverage frontier models without the burden of managing custom codebases (Ilia Karelin from Prosper, Product Market Fit).
The trade-off. The choice hinges on whether the task demands unique, proprietary logic or standard functionality. Custom builds provide the precision required for complex, high-stakes environments, while off-the-shelf solutions maximize velocity for generic operations.
Try today · 45 min
Build an evaluation script that uses a held-out dataset to score your agent's performance.
Split your test data into a fixed set of 6 examples for iteration and a held-out set to validate the final build.
You need
- A CSV containing at least 11 labeled examples
- A scoring script that compares model output against the ground truth
Steps
- Select 5 examples to hold out for final validation.
- Fix your agent's logic against the remaining 6 examples.
- Run your script on the 6 examples to ensure the fix works.
- Run the script on the 5 held-out examples to confirm the score meets your threshold.