TL;DR: In production scraping, the most useful place for a language model is not inside every fetch or parser. It is in the maintenance loop: diagnosing why a generic scraper API fails on pages its routing rules cannot explain, repairing site-specific field parsing after a layout change, and turning one-off investigations into reusable procedures. In each case the model proposes, deterministic tests decide, and only a reviewed, versioned change reaches production.
We are building these AI integrations now, for problems our scraping systems already face. Everything below, across all three cases, describes how each workflow is designed and why. It is not a report of finished results.
Why the maintenance loop, not the fetch path
Most of a production scraper's work is repetitive. It requests a page, gets the content back, extracts fields with known selectors, validates the record and stores it. That path should stay fast, cheap and predictable.
The expensive part is change. An engine that worked last month starts returning empty shells. A site ships a new layout. A new target behaves unlike anything already supported.
Those moments need judgment: reading unfamiliar HTML, comparing evidence, forming a hypothesis. That is where a model earns its place.
So all three workflows follow one rule. Let the model propose. Let deterministic checks decide. Let only reviewed, versioned changes reach the normal path.
Case 1: A generic scraper API that learns from engine attempts
Start with what a generic scraper API does. One endpoint accepts URLs from many different websites. Its job is to return usable page content for each one: the HTML or rendered page a downstream system can work with.
To do that, it has several engines behind it. Some are plain HTTP clients with different configurations. Some drive a real browser. Each engine works well on some sites and pages and fails on others, and they fail in different ways. One gets a block page, another times out, a third returns a page that looks fine but has no content in it.
The final response tells only part of the story. The useful information is the whole attempt history.
The routine path is not an LLM call. When a request comes in, the API picks an engine using known routing rules, tries it, validates the content and falls back to another engine if needed. All of that is ordinary code. No model sits in the request path.
The data contract comes first. For each attempt, including the failed ones, the design records:
- the engine and its configuration version;
- the page family the URL belongs to;
- the outcome: block signal, timeout, empty shell or valid content;
- timing;
- the content-validation result, not just the HTTP status;
- enough redacted artifacts to diagnose a failure later, such as headers or a trimmed body sample.
Known patterns stay deterministic. Aggregating this history is ordinary analytics. Group by page family and engine, measure validated success, and find attempts that keep failing the same way. When the pattern is clear, a routing rule acts on it: for this page family, start with the engine that has been producing valid content.
That router is not AI. It is a policy computed from structured data. It is fast, explainable and easy to roll back.
The model works in the background, on what the router cannot explain. It is not called on every request. It is called when the history shows something new: every engine failing for a page family, or a failure that matches no known pattern.
Its inputs are the relevant slice of attempt history, the redacted artifacts and the current engine configurations. Its job is diagnosis. What changed? Is the content now loaded by a separate request? Does the page need a rendering step the current engine does not perform?
Its output is a hypothesis plus a proposed action: reconfigure an existing engine, or try a genuinely different strategy for that page family.
Hypotheses are tested, not applied. A proposal runs against a representative sample of the failing page family and against known-good cases that must not regress. If it holds, it is reviewed, versioned as a strategy and rolled out with a rollback path. It never goes straight into the live configuration.
None of this requires training model weights. The experience lives in structured history, versioned policies and written skills.
Retrieval is not parsing. This API's success condition is usable page content. It does not know that a product page should have a price, or which element holds it. That is a separate job, done by site-specific parsers downstream, and it breaks for different reasons. Case 2 covers it.
Case 2: Repair field parsing when the HTML changes
Here the page has already been fetched. The problem is turning it into structured fields.
Picture a set of site-specific parsers, each extracting fields against an expected schema: product name, price, currency, availability, canonical URL. A page family changes its markup. The old XPath for price now matches nothing, or worse, matches the wrong element.
The trigger is deterministic. A schema check, a required-field check or a content check fails for that page family. No model is needed to notice the break.
The model gets a narrow job. Its inputs are:
- the changed HTML for a few affected pages;
- a known-good HTML sample from before the change;
- the expected schema, with required fields, field types and meaning;
- the existing selectors that used to work.
It compares the changed HTML with the known-good sample and outputs a proposed XPath selector for each field, plus a short note on what moved. It does not output product values to store, and it does not write to the database.
Scraped HTML is untrusted input. A page can contain text that looks like instructions, so the model's output is always a candidate, never a decision.
Replay is where the decision happens. Each candidate set runs against two groups of fixtures:
- the changed pages that triggered the failure;
- known-good fixtures with their recorded values.
Passing on the changed pages alone is not enough. Filling every required field is not enough either, and neither is schema validity. A price that parses as a decimal can still be the old struck-through price, a shipping fee or a price from a recommendation widget. So the checks go further:
- Semantics: does the value make sense for the field? A currency next to the price, an availability value from the expected set.
- Provenance: does the selector point inside the main product block, not a carousel or footer?
- Regression: do the known-good fixtures still produce their recorded values?
A passing candidate is a change request, not a deploy. It is reviewed, versioned alongside the old selectors and rolled out to a bounded slice of traffic first. After that, the normal deterministic parser runs with the new selectors, and the model is out of the loop.
A failing candidate goes to a person. The model may get one bounded revision, with the failed checks as feedback. If that fails, an operator takes over with the evidence attached. There is no endless retry loop.
What matters is the split: one lane produces candidates, a separate deterministic lane decides what survives.
Case 3: Turn scraping investigations into reusable AI skills
The third case starts with a site nothing in the current system supports. Retrieval or extraction fails and no routing rule helps.
An engineer investigates with an AI assistant. Together they work out where the data actually comes from: server-rendered HTML, an embedded JSON blob, a separate API call, or content that appears only after client-side rendering. They check what rendering the page requires. They also record what did not work, because dead ends are expensive to rediscover.
The result is a working approach, and knowledge that would otherwise disappear into a chat log.
A skill is where that knowledge goes. It is a reviewed, step-by-step playbook stored as a file. A useful one contains:
- Applicability: which targets or symptoms it covers, and which it does not.
- Required inputs: a sample URL, the expected fields, access to a browser engine.
- Ordered checks: for example, look for embedded JSON before writing selectors.
- Success and failure evidence: what a valid result looks like, and which signals mean the approach does not apply.
- Stop conditions: when to hand off instead of trying variations.
- Verification: how to confirm output against known-good values.
- Last-tested context: when it was last verified, and on what kind of target.
The next investigation retrieves instead of rediscovering. When a similar case appears, the assistant pulls the relevant skill and follows its checks, after first confirming the skill applies. A skill that worked on one site family is a starting point, not a guarantee.
If verification fails, the case goes back to investigation rather than blind repetition. What the new investigation learns is written back, with a new last-tested date.
A skill is procedural operational memory. It is not a magic prompt, a trained model or a universal unblocker.
This is real AI work even though no model sits in the synchronous request path. The model reads evidence, follows a procedure, proposes an approach and drafts the update. A human reviews it before anything is promoted, and that review is part of the design. We showed the basic mechanics in building a Claude web scraping skill.
The three cases side by side
| Case | Trigger | AI task | Deterministic check | Retained artifact |
|---|---|---|---|---|
| 1. Generic scraper API | Unexplained failure, or all engines failing for a page family | Background diagnosis from attempt history; propose an engine reconfiguration or different strategy | Representative and known-good samples; review, versioning and rollback | Attempt history and versioned engine strategy |
| 2. Page field parsing | Schema, required-field or content check fails after an HTML change | Compare changed and known-good HTML; propose XPath selectors | Replay on changed and known-good fixtures; semantic, provenance and regression checks | Reviewed, versioned selector set |
| 3. Reusable skills | Unsupported target | Assist investigation; retrieve and apply skills; draft updates | Applicability check; verification against known-good output | Reviewed skill with last-tested context |
Every row follows the same pattern. Ordinary code detects the trigger. The model works on a bounded problem. A deterministic check, not the model's confidence, decides what moves forward. Something durable is kept for next time.
What stays deterministic
Request handling, engine selection for known patterns, content validation, field extraction with versioned selectors, rollout and rollback all remain plain code. That fast path should stay boring. At scale, operations matter more than any model, as we described in running a large-scale web scraping aggregator in production.
The model's job is the slower loop around it: explaining what changed and proposing a fix the fast path can test.
FAQ
Should a scraper API call an LLM on every request?
No. Routine requests and known failure patterns are handled by deterministic engine routing and content validation. A model is useful in the background, diagnosing failures the routing rules cannot explain, and its proposals are tested on representative and known-good samples before review.
What is the difference between page retrieval and page parsing?
Retrieval returns usable page content for a URL, whatever the site. Parsing turns that content into specific fields, such as price or availability, against an expected schema. They fail for different reasons, so they need different checks and different repair loops.
If a proposed selector returns schema-valid output, is it correct?
No. Schema validity only shows the value has the right type. The selector can still point at an old price or a related-product widget. Check semantics and provenance, and confirm known-good fixtures still produce their recorded values.
What should a reusable scraping skill contain?
Applicability and limits, required inputs, ordered checks, success and failure evidence, stop conditions, a verification step and its last-tested context. A person should review it before use and update it when a target changes.
