Rebuilding Let The Bots In: what changed underneath the simpler report
Inside the Let The Bots In upgrade: bounded requests, evidence-based scoring, local report comparison, and a simpler report that helps you choose what to fix.
The first thing I wanted to fix was the clutter.
Let The Bots In had accumulated quite a few useful things. Crawler checks, schema analysis, generated fixes, comparison tools, exports. The problem was that they were all trying to explain themselves at once. Someone arriving to check a website had to work through the interface before getting to the work.
So I simplified the homepage and reorganized the report. But once I started deciding what deserved to be visible, I had to look harder at what the scanner was actually measuring.
What counts as a failed check? Does a successful HTTP response mean a file exists? Should an organization homepage lose points because it has no article byline? And if two reports have different scores, did the website change, or did my scoring rules change?
The visual cleanup turned into a review of the request pipeline, scoring rules, and what someone could do with the results.
The fetch pipeline now has explicit limits
The scanner runs on Astro and Cloudflare Pages Functions. A scan fetches the submitted URL using a browser-like user-agent and an OAI-SearchBot user-agent, alongside resources such as robots.txt, /sitemap.xml, and /llms.txt.
I kept that architecture. I made the amount of work it can do more explicit.
| Limit | Current value |
|---|---|
| Shared scan deadline | 12 seconds |
| Timeout for each page request | 8 seconds |
| Timeout for a standard resource request | 6 seconds |
| Wikidata lookup timeout | 3 seconds |
| Maximum HTML body per response | 1 MiB |
Maximum robots.txt body | 512 KiB |
| Default maximum resource body | 256 KiB |
| Additional robots-declared sitemap candidates | 2 |
The initial requests run concurrently. An AbortController shares the remaining scan budget with fetches, and body reads have byte limits. Requests are canceled when that budget is exhausted or the calling request is aborted.
This matters because stopping a request after a timeout is only half the job. A server can send its headers promptly and then keep streaming a body. Reading the whole thing with an unbounded response.text() would still let one page consume far more work than intended.
Sitemap discovery is bounded too. The scanner checks the default location and can inspect up to two additional same-site locations declared in robots.txt. If the declared list exceeds what it can inspect, that limit becomes part of the coverage result. It does not quietly translate to “there is no sitemap.”
When the budget runs out, the report explains the incomplete measurements. Twelve seconds is an analysis budget, not a guarantee of complete coverage.
HTTP 200 stopped being enough
A surprisingly easy way to fool a resource check is to return the website for every path.
Ask for /robots.txt, receive 200 OK, and the body turns out to be <!doctype html>.... The same thing can happen with /sitemap.xml or /llms.txt when an application has a catch-all route.
The resource checks now inspect the response body before accepting the file. An HTML fallback is not treated as a usable text policy or XML sitemap. The report calls out these soft-200 responses so someone can fix the route rather than wonder why a file that “exists” is being ignored.
I also separated resource observations into explicit states:
detected
not-detected
could-not-check
not-applicableA timeout belongs in could-not-check. A completed check that finds no usable resource belongs in not-detected. Those states lead to different advice, which is why reducing them to one boolean caused trouble.
Missing robots.txt, for example, does not by itself mean crawlers are disallowed. An unreadable policy also does not give me permission to declare that every crawler passed.
Coverage is now part of the result
The API returns the measurement coverage alongside the findings and scores. Each check has a state and a description of what happened.
This is an illustrative fragment of an incomplete response:
{
"scoringVersion": "2026-10-18.2",
"score": null,
"coverage": {
"complete": false,
"checks": [
{
"id": "robots",
"label": "robots.txt policy",
"state": "could-not-check",
"detail": "Policy could not be read reliably; no access verdict inferred."
}
]
}
}The full response includes the other checks and metadata. Unavailable evidence does not become a reassuring number.
Core coverage includes the two page responses, robots policy, sitemap discovery, content analysis, and schema inspection. If those required measurements are incomplete, the scanner withholds the complete score. A truncated HTML response also prevents a complete score.
Optional checks stay separate. An unavailable llms.txt check does not invalidate an otherwise complete core scan. Actual access from vendor crawler infrastructure is marked outside the measurement rather than counted as something the scanner verified.
If the crawler-header request is refused but the browser-header request returns useful HTML, the scanner can still analyze that HTML. The report names the source. Otherwise the findings would look like evidence about a response the crawler probe never received.
Coverage now stays visible across the report views. I did not want a warning at the top to disappear from the reader’s decision just because they had scrolled down to a copy button.
One shared content analysis feeds several checks
I moved the document analysis behind a shared function:
analyzePage(html, url, pageTypeOverride?)It returns schema facts, content structure, readability signals, identity evidence, inferred page type, extracted text, a heading outline, and date analysis.
Inside that function, parse5 creates the document tree. The original tree remains available for metadata and structured-data inspection. A cloned tree is cleaned for content analysis.
That cleaned copy removes scripts, styles, templates, navigation, footers, sidebars, and explicitly hidden elements. Header handling is more selective. A global site header should not inflate the content count, but an article header can contain the actual title and publication details.
That distinction took more care than deleting every <header> tag.
The text count uses Intl.Segmenter to count word-like segments. Heading extraction works from the cleaned content and records H1 through H3. Schema parsing is cached within the analysis context so later checks can reuse the facts already collected.
I made gate detection less eager too. A mention of subscriptions is not sufficient evidence of a paywall. The analyzer looks for corroborating signals, such as gate markup and restricted-access schema, before treating it as a stronger restriction. Weaker evidence stays a possible restriction.
Scoring now distinguishes useful markup from a declared type
The six category ceilings still add up to 100:
| Category | Maximum points |
|---|---|
| Access | 28 |
| Readability | 14 |
| Structured data | 16 |
| Identity and attribution | 18 |
| Extractability | 14 |
| Freshness and hygiene | 10 |
The changes are in what qualifies for those points and how much confidence the scanner has in the inputs.
For structured data, finding "@type": "Article" is no longer enough to treat that node as fully useful. The analyzer looks at required-field completeness, schema context, and date validity. These contribute to a per-type quality value.
The structured-data category has up to four points for the detected format and twelve for the recommended types. Both are adjusted by the quality of the supported markup. The type allowance is weighted across what is relevant to the page, so a homepage does not need to add an irrelevant FAQ just to chase the remaining points.
Audit errors also reduce the structured-data subtotal. The current multiplier is max(0.25, 1 - 0.1 × errorCount). That is an internal scoring choice, not a formula published by a search engine.
The audit groups its checks under six principles:
- Content agreement, including whether declared facts agree with the page.
- Entity connections, such as usable author and publisher references.
- Type specificity, where a more accurate type fits the content.
- Direct answers in FAQ markup.
- Syntax, field, reference, and date validity.
- Duplicate or conflicting declarations of the same entity.
Some advice remains heuristic. The FAQ answer-length check uses a 40–300 character range. That is my extraction heuristic. It is not an eligibility requirement from OpenAI or Google, and the report should not present it as one.
Page type and publisher identity needed different rules
The scanner now uses page context when recommending schema. A product page expects Product and breadcrumbs. Documentation expects TechArticle and breadcrumbs. A profile page expects ProfilePage and Person.
Inference considers the main entity and graph references rather than treating every nested node as the page’s subject. A Person describing an organization’s founder should not automatically turn its homepage into a personal profile.
There is also a manual page-type override. The API sends the relevant variants from the facts already fetched, and the browser changes the schema expectations, applicable freshness rules, findings, and generated JSON-LD locally. It does not refetch the URL for every correction.
For identity, I wanted accurate local publisher information to count. The current eighteen-point rubric checks a visible identity name, a same-site URL, a meaningful description that appears in the page text, and accountability evidence. The allocation is four points each for name, URL and description, and six for accountability or supported profile evidence.
Accountability can come from an About link, an author byline, or contact information. A small company should not have to manufacture an article author or a Wikidata entry to describe who runs its homepage.
External identity checks are more conservative as well. Profile URLs are parsed and matched against recognized hosts and path shapes. A domain that merely contains wikipedia.org is not Wikipedia. A result from the Wikidata name search remains a candidate and earns no confirmed-match credit.
Dates now get calendar validation. JavaScript can normalize some invalid dates instead of rejecting them, so a date-shaped string is not enough. The analyzer also flags future publication dates and modification dates earlier than publication.
Freshness scoring applies date expectations to articles and documentation. An evergreen homepage does not need a pretend publication date.
I removed scoring incentives for a few weak proxies too. llms.txt, sitemap lastmod, and a fast single-request timing sample each earn zero points. They can still be useful diagnostics. Their presence does not deserve to substitute for readable content or a usable canonical URL.
Robots policy now separates three different jobs
The report’s policy builder separates AI search and citation crawlers, user-triggered browsing agents, and model-training crawlers.
That lets someone allow OAI-SearchBot while choosing a different policy for GPTBot. They should not have to make that decision through a generic “block AI” toggle.
The report builder starts from the observed file. Enabled controls modify the relevant class, and unchecked controls preserve that class’s existing policy. In particular, leaving the training-block option unchecked does not silently remove an existing training restriction.
Named groups need care. A more-specific user-agent group does not necessarily inherit wildcard exclusions. Adding a bare Allow: / for a bot can change what happens to paths the wildcard group restricted.
An illustrative policy that preserves a private-path exclusion looks like this:
User-agent: *
Disallow: /private/
User-agent: OAI-SearchBot
Allow: /
Disallow: /private/
User-agent: GPTBot
Disallow: /The patch generator carries forward relevant restrictions and handles agents already declared in named groups. I still expect someone to review the output against their actual policy. robots.txt is public crawler guidance; authentication is what protects private content.
Bot view is inspectable, with the request limits beside it
The report now exposes the H1–H3 outline and approximately the first 1,000 characters of stripped text from the crawler-header response and the browser-header response.
That makes an abstract readability finding much easier to investigate. You can see the text each response supplied instead of accepting a label that says the content might be missing.
But a browser-like user-agent is not a browser render. The default comparison makes HTTP requests. It does not execute JavaScript.
Real browser rendering is a separate, explicit diagnostic and requires the existing Cloudflare Browser Rendering credentials. When a rendered response is available, the current content-gap check requires at least 150 rendered words and flags a gap when the raw crawler response has fewer than half as many words. The missing share can then affect readability scoring.
Without that rendered evidence, the tool does not pretend it compared the fully hydrated page with raw HTML.
The same limit applies to WAF conclusions. Both default probes originate from the scanner’s infrastructure. A difference can reveal a header-sensitive response, but it does not establish what happens to a verified crawler visiting from a vendor network.
The report is organized around the next action
The redesign gives the report three views. Overview shows priorities and category scores. Findings holds the canonical finding list. Technical details contains coverage, crawler policy, extracted content, and the lower-level diagnostics.
Generated code now sits beside the relevant finding. A schema problem can expose the JSON-LD suggestion there, rather than sending someone to another section to find a snippet that might belong to it.
The effort filters separate quick configuration work, schema and markup changes, and architecture or off-site work. Passed checks are collapsed. Export and sharing controls open when requested.
The shared navigation is now Scanner, Tools, and Learn. Light and dark themes use the same flatter layout, and saved theme preferences still apply. The reading pages use native expandable sections rather than another UI library.
The coding-agent prompt also has more context. It includes the exact scanned URL, scoring version, page type, coverage limits, observations, and applicable generated fixes. You can select findings to limit its scope, then copy only that work into your coding agent.
The prompt tells the agent to inspect the existing repository, verify the observations, preserve routes and unrelated schema, and run appropriate checks. Report text and snippets are treated as untrusted input rather than permission to follow arbitrary instructions embedded in a fetched page.
Several findings can describe the same category gap, so their apparent deductions should not be added together. Access conditions can also cap the total below the category sum: a high-confidence refusal of the crawler-header probe caps it at 39, blocking three or more major platform search crawlers caps it at 39, and a smaller major-platform block or noindex caps it at 64. The report exposes the uncapped total and the binding cap so that arithmetic is inspectable.
Saved reports can now be compared without a backend
A report can be downloaded as JSON, Markdown, or a Markdown fix pack with the applicable snippets. The selected-fix checklist stays in the browser session.
The saved-report utility compares a before and after JSON file locally. When they are comparable, it shows category deltas and resolved, new, and unchanged finding IDs.
It checks the comparison conditions first. The exact URL, including path and query, must agree. So must the scoring version, page type, category maxima, and applicable coverage. The before report cannot be newer than the after report, and both need complete usable scores.
If the conditions differ, it explains why and withholds the deltas. A higher number from a different scoring version is not evidence that a deployment fixed anything.
Imports have a 2 MiB limit and validate the report structure, timestamps, ranges, duplicate finding IDs, nesting depth, and unsafe object keys. Imported strings render as text rather than HTML.
The same utility can inspect pasted or uploaded HTML. It runs the shared content analyzer locally using the intended page URL for context. It does not execute the supplied scripts, load their assets, or upload the draft.
It also does not assign a live readiness score. A local HTML file cannot establish deployed robots policy, HTTP access, or response timing. It can tell you about the content and markup you handed it.
The new reference pages share the scanner’s vocabulary
This release adds 100 static reference pages:
| Addition | Pages |
|---|---|
| Crawler references | 17 |
| Finding and remediation guides | 73 |
| Focused diagnostic pages | 4 |
| Troubleshooting guides | 4 |
| Directory hubs | 2 |
The focused tools cover AI robots policy, llms.txt format, WAF/header-response differences, and Wikidata/identity evidence. They reuse the scanner’s existing observations rather than starting separate crawls for each landing page.
The llms.txt validator, for example, inspects the response already fetched. It checks for one Markdown H1, a blockquote summary, H2 sections, and absolute HTTP or HTTPS document links. It does not crawl every linked document or certify that an assistant consumes the file.
Findings link to stable guide routes. Each guide explains the measured condition, when a change is appropriate, a remediation example, and how to verify it. A passed check or unknown observation is not automatically a request to edit code.
The sitemap and the site’s own llms.txt are generated from route metadata and the shared reference data. That means a new guide has a path into the indexes instead of depending on me remembering two more files.
I updated How it works to describe the actual scoring rules and measurement limits. It imports shared weight and limit constants where applicable. The longer sections are expandable, and deep links open the relevant section. The glossary has instant filtering and copyable term anchors.
Monitoring now treats uncertainty differently
The monitoring worker keeps its existing hourly schedule. Each invocation processes a bounded batch of due watches. I did not increase the schedule frequency to make this update sound more capable.
A score-drop alert requires comparable scoring versions and a drop of at least five points. An uncertain reachability failure or crawler-header refusal needs confirmation in a second scheduled observation. That confirmation uses the existing schedule rather than launching extra retry requests.
If a robots check cannot complete, the monitor preserves the previous known policy instead of inventing a change. An explicit newly observed block can still be reported from a valid policy response. The monitor also keeps the last successful observation time.
The cron worker now sends its authentication token in an x-monitor-token header instead of a query parameter. That keeps the credential out of the request URL.
Where the data and cost actually go
Fetched page HTML and full reports are not retained by the scanner. The existing share-image and badge cache stores compact score metadata under a SHA-256 digest of the normalized hostname and path/query input. Entries expire after thirty days.
Hashing avoids a raw scanned URL as the cache key. It is not a claim of cryptographic anonymity for a guessable URL.
Opt-in monitoring is different. It needs the URL you ask it to watch, your subscription details, and the compact history and policy needed to detect a change. That distinction is documented rather than hidden inside a blanket “nothing is stored” claim.
The additions reuse fetched responses or run in the browser, without adding a paid service or an LLM request to each scan. Optional rendering and archive work remain explicit actions. Hosting still has request and execution limits.
What I checked before putting it live
The release passed typechecking, the production build, the existing check corpus and Common Crawl checks, and eighty automated tests. Regression coverage includes HTML fallback resources, incomplete responses, attribution and page-type edge cases, report imports, comparison rules, and monitor confirmation behavior.
The content audit inspected all 100 new pages, 315 JSON-LD blocks, and 2,220 internal links. It checks canonicals, heading order, visible FAQ agreement, and index coverage. Browser checks cover the report controls, complete and incomplete states, and the guide layout at desktop and narrow widths.
There was one production-build correction too. My local Node version loaded TypeScript data modules directly. Cloudflare’s Node 22.16 build runtime needed --experimental-strip-types on the content-index command. I added the explicit flag rather than another loader dependency, checked it against that Node version, and let the Git-integrated build run again.
The completed release also passed a live scan on the public domain. The earlier design remains in a Git checkpoint in case I decide I removed something useful.
If you try the updated scanner, save a JSON report before making a change. Fix one finding, rescan the same URL, and compare the two files.