Every price on this site went through the same four stages, and one of them is a person reading the provider's own pricing page. This is what happens between a crawl starting and a number appearing on a listing.
Stage one: crawl
A crawl starts with a guess at where pricing lives, then stops guessing.
The first attempts are the obvious paths, plus a couple chosen by category, because a virtual server provider is far more likely to price at /cloud than at /pricing. Whatever those pages return is then scanned for links that look like pricing, and those are followed too.
That second half matters more than it sounds. Two examples from testing:
| Provider | What the guess found | What link following found |
|---|---|---|
| Cloudflare Workers | The company-wide plans page, listing fifteen products | Two plans, both actually Workers |
| Hetzner | Three empty stub pages and a homepage | The plan table, with 25 real servers |
Hetzner answers /pricing, /plans and /features with about a hundred characters each. Anything under four hundred characters is now treated as a miss rather than as a source, because a near-empty page pollutes what the model reads and hides the fact that the guess failed.
Stage two: extract
The crawled markdown goes to a model with a typed schema and a prompt written to make omission cheap and invention expensive.
Every field is nullable on purpose:
price_monthly: z
.number()
.nullable()
.describe("Monthly price as a number, null if the page does not state one."),
The model has to be able to say "the page did not state it". A null reaches the reviewer as "not verified", which is true. A guess reaches the reviewer looking exactly like a fact, which is the failure this whole pipeline exists to prevent.
A few rules earn their place:
- A starting-at figure is not a plan price. It goes in the notes.
- An hourly rate is only converted to a monthly one if the page itself does the conversion. Never multiply on the model's behalf.
- Usage-based pricing is still pricing: the platform fee is the price, and the metered rates go in the notes.
- Take only the named product. A large vendor's pricing page lists everything it sells.
Stage three: review
Nothing published so far. The extraction lands in a pending changes table, and there is no code path from the crawler to a live listing. That is structural rather than a policy: the function that writes to a listing is only reachable from the admin panel.
A reviewer sees current and proposed values side by side, plans in their own table, and the source URLs as links, because checking them is the job. Then they approve or reject.
Approval refuses to apply an empty value, which sounds pedantic and is not. A pricing page states none of the sovereignty fields, so an extraction comes back with an empty datacenter list and unknown CLOUD Act exposure. Applying those would have quietly stripped verified European data from a listing every time somebody approved a price.
Stage four: publish
The listing shows the plans, and the page revalidates so the change is live in seconds rather than at the next build.
Until all four stages have happened, the pricing row says "not verified". That is the honest state, and it is why most listings here still show no price:
- Not verified means nobody has checked it. The provider's own page is linked.
- Verified means a person read the source and approved what the model extracted.
There is no third state where a plausible number sits on the page hoping to be right.
What this costs
Coverage, and visibly. A directory that generates plausible figures looks finished on day one. This one looks patchy until the crawls have run and somebody has read them.
The trade still seems obviously right. A reader who finds an empty pricing row learns something true. A reader who finds a wrong number learns something false, and has no way to tell the difference.