hebel. today · 15 November 202651 / 79
Career · Product Management Evolution
Designers replace static mockups with model evaluation sets
Designers at firms like Stripe now build internal playgrounds, such as ProtoDash, which bake design systems into tools so teammates can generate realistic prototypes in minutes. Instead of relying on static mockups, product teams…
From 3 newsletters · 1 min read · one thing to try
Designers at firms like Stripe now build internal playgrounds, such as ProtoDash, which bake design systems into tools so teammates can generate realistic prototypes in minutes (The Designer's Field Guide, by Kai Wong.). Instead of relying on static mockups, product teams are shifting toward evaluation sets—collections of real-world inputs used to stress-test model behavior systematically (UX Collective, Patrick Neeman from AI for UX Substack). Designers call models directly using scripts in Claude Code or Cursor to send product inputs and print answers before drawing a single screen.
Teams run each prompt 50 times to capture a distribution of outcomes, sorting results by length, tone, and accuracy. They bring the two worst-performing outputs into design reviews to define the specific failure cases the interface must survive. This practice forces designers to act as technologists who observe model behavior across thousands of inputs, as the material for AI products is not found on a static canvas.
Parrott spent a month encoding brand judgment into an agent to ensure consistent output quality, focusing on non-negotiable rules rather than interface layout. When models change with provider updates, engineering teams rerun these saved failure cases to detect regressions without needing manual re-design.
Try today · 45 min
Create an evaluation set for your current AI feature by running 50 variations of a single prompt.
Use a script to output results into a text file, then sort them by accuracy to identify the two worst responses.
You need
- Access to Claude Code or Cursor
- A list of 50 representative user inputs
- A shared document for tracking model failures
Steps
- Select one core prompt your AI feature uses to generate user-facing content.
- Write a simple script to execute this prompt 50 times using varied inputs.
- Print all 50 outputs into a single text file.
- Sort the outputs by tone and accuracy to find the two most problematic results.
- Document these failures in a shared file for your engineering team to use as a regression test.