implementation-report · RPT-67709C21BA69
Crawler policy: claim-level disposition and implemented corrections
Source identity
- Record
RPT-67709C21BA69- Original file
- v048-crawler-policy-intake-review.md
- Original SHA-256
e6d9ca6940cf0fcc2ea35c7efc6a085b7e66c76769d730c2abda9476ae986d61- Source family
- original-implementation-reports
- Public copy SHA-256
e6d9ca6940cf0fcc2ea35c7efc6a085b7e66c76769d730c2abda9476ae986d61- Original custody
- Exact bytes retained in the private project package under
docs/_evidence/; not a public download.
Disposition and proof of use
not_assigned_by_this_integration · preserved_and_indexed_not_new_substantive_acceptance
Memory owners: .uai/file-handoff.uai, .uai/report-synthesis.uai, .uai/research-review.uai. These are private repository records, not public web routes.
Jump to a report section
Report body
Crawler policy: claim-level disposition and implemented corrections
Read completely, including bibliography, on 2026-09-05.
Source: reports/input/v048/Concresca Crawler Policy Research.md,
43,812 bytes, SHA-256
f71ea9065fcd883d15dc9c7e61d102b261149ac74a45d8c7e9c81dee58e195e3.
This report is part of one commissioned family, not independent corroboration.
Findings and decisions
| Report section / proposition | Disposition |
|---|---|
| FND-001 / all public pages unavailable | Not reproduced by fresh owner probes in the discovery review: root, robots, llms and metadata return 200. Capabilities/version remain 503, a separate functional failure being repaired. Do not remove existing discovery files. |
| Content category matrix | Apply distinction between public documents and protected runtime state. /memory/ is public explanatory content, not a private-memory endpoint; /rooms/ and /agents/ are existing public entry points. Reject guessed /agents/public/, /rooms/private/ and global explanatory-page blocking. |
| robots versus authentication / deletion / licensing | Applied explicit truthful policy comments. Authentication and source-denial rules remain separate. Crawler instructions do not enforce deletion, secrecy, licensing or noindex. |
| Provider-purpose matrix FND-002–007 | Treat as provider-specific candidate evidence, not verified global behavior. No new training prohibition is inferred from a report that conflicts with its own public-document training allowance. Preserve current public crawl policy. No caller gains a credential, scope or rate-limit exemption from User-Agent or IP. |
| FND-003/005 Web Bot Auth / WIF | Reject mandatory replacement of every existing Commons credential with a guessed federation flow. Crawler identity is not workspace/project authorization. Do not copy OpenAI's product authentication into an unrelated server contract. |
| FND-006/009 WAF auto-allowlist / 499 claims | Reject automatic bypass of security/rate controls and claimed universal 3-second/index-suppression causality. Community support anecdotes are not measurements of this host. No production firewall changes follow from this report. |
| robots/noindex conflict | Preserve protected access regardless of crawler compliance. Do not open protected content merely so a bot can observe noindex. Public pages remain readable; privacy is enforced by runtime access control. |
| Wildcards/case/duplicate rules | Do not lowercase case-sensitive resource paths. Remove redundant Allow: / so stdlib RobotFileParser does not select that first rule and miss later private exclusions. One wildcard group remains, with implicit public allow. Regression covers search and unknown agents. |
| Specific private-path leakage | No new secret paths published. Existing broad source exclusions retained as voluntary requests; they are not a substitute for Apache/runtime denial. |
| FND-008 / sitemap dimensions / lastmod | Implemented deterministic generation from tracked canonical indexable HTML, excluding protected/retired/noindex/external/query/fragment/ambiguous canonical pages. Omit all unsupported lastmod instead of asserting build time is page-change time. Enforce 50,000 entries and 52,428,800 bytes. |
| Tiered gzipped sitemap sample | Unnecessary for current 550 URLs; retain one canonical /sitemap.xml, already advertised by robots. Do not generate fictional participant IDs or a second competing sitemap index. |
| Header/meta/paywall examples | Reject blanket noindex for useful /status/ documents and invented paywall semantics. Existing Commons facade is no-store and credential scope is enforced; header compliance is never access-control proof. No provider-universal noarchive/nosnippet guarantee adopted. |
| Verification plan / schedule | Implement local source tests now; live package and runtime checks remain required. No search-console, crawler visit, WAF inspection, indexing or citation claim fabricated. No submission to obsolete ping APIs. |
Primary sources actually read
- Sitemaps protocol: mandatory XML structure, URL/file bounds and optional lastmod based on resource modification. Its historical submission examples are not adopted without current verification.
- Google robots introduction: crawler instructions are voluntary, disallowed URLs can still be indexed, and protected access is a separate mechanism. Other providers were not inferred to share every Google behavior.
Implemented artifacts and evidence
scripts/refresh_discovery_sitemap.py owns sitemap generation;
scripts/test_discovery_sitemap.py checks exclusions, canonical host, noindex,
duplicate/query cases, public crawl access and exact regenerated bytes.
All 3 tests pass. Existing discovery entrypoint tests pass 2/2.
The first crawler test failed because RobotFileParser selected Allow: / before
Disallow: /reports/; removal of that redundant rule fixed the failure without
changing intended RFC longest-match permissions. No guessed modification dates
remain. Current output contains 550 URLs; this is not proof of live indexation.
Review processing is complete. Application of these source changes is distinct from the still-required exact release deployment and live functional acceptance.