hebel. today · 11 November 202636 / 73
Agentic Engineering · AI engineering
Custom agent harnesses versus framework-based orchestration
The case for custom harnesses. Alexandr Wang at Meta maintained a meticulous spreadsheet of over 100 specific behaviors to validate the Muse agent, blocking any release that failed even 20 of these rows. RevenueCat…
From 4 newsletters · 1 min read · one thing to try
The case for custom harnesses. Alexandr Wang at Meta maintained a meticulous spreadsheet of over 100 specific behaviors to validate the Muse agent, blocking any release that failed even 20 of these rows (Ruben Dominguez from The AI Corner). RevenueCat found that their custom-built paywall agent, Astra, consistently produced substandard layouts despite repeated model upgrades and prompt tuning, forcing the team to confront a massive quality gap between task fulfillment and actual design utility (Azat Valiev from RevenueCat Blog).
The case for framework adoption. Developers at RevenueCat observed that Claude Code, a pre-built agent harness, produced superior paywall designs compared to their internal Astra system, highlighting the difficulty of matching frontier-level performance with custom orchestration. Meanwhile, engineers use frameworks like LangChain, LlamaIndex, or Haystack to manage the underlying plumbing of tool integration and retrieval, allowing them to focus on application-specific logic rather than building infrastructure from scratch (AI & Tech Insights).
The synthesis of approaches. Success often hinges on the distance between a working demo and a reliable business, a gap that requires both rigorous, custom-defined evaluation scorecards and the use of established tools to handle complex state management (The Tech Bridge). Whether using a custom harness or an off-the-shelf framework, engineers must define success in writing—such as a 95% threshold for booking accuracy—to ensure the agent remains functional after the initial build phase.
Try today · 30 min
Build a 10-row launch scorecard for your current agent.
Start with the most critical behavior your agent must perform reliably, then define the success threshold.
You need
- A spreadsheet or markdown table
- A clear definition of one core agent behavior
- A specific success metric (e.g., 90% accuracy)
Steps
- List 10 specific behaviors your agent must perform.
- Define a 'launch-blocking' failure threshold for each.
- Define a 'good' performance threshold for each.
- Create a simple eval script to test one row today.
- Record today's score for that row.