Testing software readiness with messy customer data

Shikshita Juyal of Crework Labs requires that her team’s generative AI systems pass a ten-minute stress test before reaching paying customers, specifically checking how the software handles messy data and whether it explicitly signals when it is guessing (Kim Doyal 18+ years building online. Now…). This validation process addresses the gap between a developer’s curated test cases and the unpredictable inputs of a new user who lacks the internal knowledge to avoid specific errors. Kim Doyal identifies that a two-sided marketplace for beauty professionals required two months of integration work for payments, scheduling, and verification, despite the matching logic taking only two days to build (Kim Doyal 18+ years building online. Now…).

To determine if a build is ready for sale, Doyal lists every account, login, and data format a customer must configure before the system functions, counting these requirements to identify if the setup process itself is a significant part of the product offering. This approach mirrors the strategy at ConnectWise, where Austin’s team ran a low-code pilot in a controlled environment to observe how representatives used the tool and where costs exceeded initial projections (Gong Blog). Organizations now allocate approximately 80 percent of their effort to standardizing core workflows and best practices across teams, leaving the remaining capacity for custom development that provides unique differentiation (Gong Blog).

Product teams must distinguish between a prototype built to learn and software intended for production, as the latter requires adherence to a formal Definition of Done to ensure reliability (Hands-on Agile by Stefan Wolpers AI…). The 2025 DORA report indicates that while AI adoption increases delivery throughput, it also correlates with rising software delivery instability, often due to the increased cognitive load placed on reviewers.