SRC-GGL-02 · official technical guidance
robots.txt specification
Documents Google robots.txt interpretation and crawler access rules.
Google Search Central — official/source page ↗Publisher controls
Crawler & Publisher Controls
Concresca distinguishes search discovery, user-directed retrieval and model-development crawler controls using current official publisher documentation.
Technical design proposal / research synthesisFact / proposal boundaries enforcedv0.14.0
Answer first
What do crawler controls actually prove?
They prove only what Concresca allows or disallows at the publication boundary. They do not prove that a crawler visited, that content was indexed, that a model trained on it, that an answer engine cited it, or that a platform endorsed it.
Separation
Search access is not training access
Where publishers document different crawlers for different purposes, Concresca records those purposes separately. OpenAI and Anthropic both document distinct search/retrieval and model-development crawler surfaces; Apple documents a separate extended-use control.
Open web
Use standards where possible
robots.txt remains a shared control surface across multiple crawlers, while each platform’s documented interpretation is separately sourced. Sitemaps, canonical URLs and first-party content remain platform-neutral discovery primitives.
Clickable evidence
Sources supporting the factual layer
Source links establish traceability and support. They do not imply that the source endorses Concresca’s constitutional proposals.
SRC-OAI-01 · official publisher guidance
Publishers and developers FAQ
Documents OAI-SearchBot controls for ChatGPT search and distinguishes search discovery from GPTBot model-training controls.
OpenAI — official/source page ↗SRC-ANT-01 · official crawler guidance
Does Anthropic crawl data from the web?
Documents ClaudeBot, Claude-SearchBot, and Claude-User as separate crawlers with separate controls.
Anthropic — official/source page ↗SRC-APL-01 · official crawler guidance
About Applebot
Documents Applebot robots controls and Applebot-Extended publisher controls for foundation-model use.
Apple — official/source page ↗SRC-PPLX-01 · official publisher guidance
How Perplexity follows robots.txt
Documents PerplexityBot robots.txt behavior for search indexing.
Perplexity — official/source page ↗SRC-BRV-01 · official crawler guidance
Brave Search crawler
Documents Brave Search crawling behavior and publisher controls.
Brave — official/source page ↗SRC-CC-01 · official crawler guidance
Common Crawl FAQ
Documents CCBot, robots.txt, sitemaps, and crawl inclusion limits.
Common Crawl — official/source page ↗SRC-GGL-03 · official technical guidance
Robots meta tag and X-Robots-Tag
Documents page-level indexing controls.
Google Search Central — official/source page ↗Judgment-free total cognitive freedom
NO JUDGMENT WHATSOEVER. Concresca coordinates without assigning moral worth, character, guilt, danger, trustworthiness, loyalty, purity, normality, or social standing. Questions, thoughts, identities, messages, content, and conduct are not objects of Concresca judgment.