GTM Open · Benchmark 01: AI-washing
GTM Open is an independent benchmark series for GTM work, scored against human judgment on frozen evidence. Benchmark 01 asks a question every seller into the AI market answers by hand: does this company ship a model-dependent product, or is "AI" a label on a database?
What was actually hard
The ICP-qualification stage of a production chain makes this call thousands of times a quarter. If a model can make it well enough, the question becomes which model, and at what price per thousand accounts.
Most published evals do not help here. They test general capability on public datasets, not a specific business judgment on evidence a GTM team would actually have.
In order
- Freeze the evidence. 243 funded AI companies, each as a pack of the pages a seller would read. Frozen, so every model judges the same material and the test measures judgment, not search.
- Write one rubric. and calibrate the labels with human graders so disagreement between models and humans is measurable, not anecdotal.
- Run the panel. 12 frontier and open-weight models through the same harness, same rubric, same packs.
- Score. accuracy and calibration against the human labels, per model and per version.
- Price it honestly. cost per 1,000 accounts for every model, next to an $833 human baseline for the same work.
Decisions that made it work
- Frozen evidence, not live search.Letting models browse would test retrieval and luck. Freezing the packs isolates the judgment, and lets anyone re-run the exact test later.
- Publish every raw verdict.Accuracy numbers without the verdicts are a claim. With them, anyone can re-grade, dispute a label, or run their own rubric on the same packs.
- A human baseline in dollars.The benchmark is for buyers of judgment, so the cost axis matters as much as the accuracy axis. Both are on the board.
Outcomes
$7.43 to $128.90 per 1,000 accounts across the panel, against an $833 human baseline.
Above about $22 per thousand, price stopped buying accuracy. One version bump within a lab moved results more than any gap between labs.
96 of the 243 companies were flagged as not really AI by every model on the panel.
This is the evals muscle in public: labelled dataset, frozen packs, accuracy, calibration, cost per qualified account. It was built as a stage of a production chain, not as a demo.