Automated gates versus human oversight in agentic workflows

Mechanical gates for quality. Cate’s team at her software factory enforces a 70% coverage threshold via CI and uses mutation testing to ensure tests are not just present but effective (Andrew Zigler from Dev Interrupted). By integrating jscpd as a mechanical gate, developers block code duplication before it reaches review, as teams typically see duplication rates climb above 10% without such automated checks. This strategy treats evaluation as a continuous integration process, where automated test suites catch regressions before they reach human reviewers (Jean Lee @Exaltitude Newsletter).

Human judgment for context. Exploratory data analysis requires human eyes because models cannot know what a team intends to achieve in a vacuum (TLDR Data). Even when judges like Claude Haiku 4.5 and GPT-5.2 match each other's scores at a kappa of 0.75, they may still fail to catch what human-executed tests identify (The AI Engineer). Teams that rely solely on automated judges risk missing the nuances of product behavior, as 144 security patches showed that judges matched each other but diverged significantly from actual test results.

The synthesis of both. Savvy engineering leaders scale with AI by pairing data-driven discovery with mechanical automation to free human attention for judgment calls. While machines handle the mechanical verification that code meets basic quality standards, human experts maintain the accountability for domain-specific definitions like what constitutes an active project (Modern Data 101).

• Claude Haiku 4.5 and GPT-5.2 judges showed a 0.75 agreement on security patches, yet their agreement against actual tests was as low as 0.19 (The AI Engineer).
• Text-to-SQL models improve coverage by 29-63 percentage points when they answer multiple interpretations of ambiguous questions separately rather than choosing one (TLDR Data).