Read the page before you pay a model to read it
The reflex when you need data out of a web page is to send it to a model. Check what the page already publishes first, because a lot of the web hands you the answer for free.
We needed the contents of a page pulled apart into fields. The obvious approach is to fetch it, strip the tags and hand it to a language model with a schema. That works. It also costs a fraction of a penny per page, takes several seconds, and returns something approximately right.
Then we looked at the page. The whole thing was already there, in a script tag, in a defined shape, exact.
Why it is there
Search engines reward structured data. A page that wants a rich result publishes schema.org JSON-LD describing itself: the fields, their names, their values. That is not a courtesy to you, it is how the page earns its listing, which is exactly why it is reliable. The commercial incentive to keep it accurate is stronger than the incentive to keep the visible page tidy.
Recipes, products, events, job postings, articles, reviews, organisations and opening hours all have well-defined types, and any page that competes on search results for them will usually carry one.
What that buys you
- Exactness. The quantities are the publisher's quantities, not a paraphrase. Nothing is silently rounded, reworded or invented.
- Speed. A regex and a JSON parse against a network call.
- No cost, and no key. It works when your API provider is down and it works on a machine with no credentials on it.
- It cannot hallucinate. Either the field is there or it is not.
Walk it, do not assume a shape
The one real difficulty is that publishers nest this every imaginable way. A bare object. An array of objects. Everything wrapped in an @graph. The thing you want sitting under mainEntity of a WebPage. Inside an itemListElement.
So do not reach into a fixed path. Walk whatever you parsed, recursively, and return the first node whose @type matches what you are after. Thirty lines, and it stops mattering which of the five shapes the site chose.
Two details that catch people out. @type is sometimes a string and sometimes an array. Durations are ISO 8601, so an hour and a half is PT1H30M and needs parsing rather than reading.
Keep the model as the fallback
This is not an argument against the model. It is an argument about ordering. Try the structured data; if it is absent or empty, fall back to reading the prose. You get the exact answer where the page offers one and a decent answer everywhere else, and you only pay on the second path.
Log which path each fetch took. It tells you how much of your traffic is actually costing you anything, and it makes a surprising extraction failure easy to explain later.
We briefly concluded a site published nothing, having looked at its category listing rather than one of its item pages. Listing pages carry different markup, or none. Test against the page you will actually be fetching.
What it will not fix
Plenty of sites block server-side fetches outright. You will get a 403 from a page that loads perfectly in a browser, because bot protection is looking at the request and not the page. No parsing strategy helps there; that is a different problem, and the honest answer is usually to give the user another way in rather than to escalate an arms race.
Need help with any of this?
These notes are free and always will be. If you would rather someone just set it up, or you are stuck on something similar, get in touch at hello@opsira.io.