Google Search / AI search features
Receiver/crawler: Googlebot
Purpose: Search crawling/indexing
Publisher controls: robots.txt; robots meta; canonical; sitemap; structured data
Measurement: Search Console / generative-AI reporting where available
Multi-engine discovery
Discovery Receiver Matrix
A source-linked matrix of search engines, answer engines, crawlers, retrieval systems, publisher controls and measurement surfaces relevant to Concresca.
Technical design proposal / research synthesisFact / proposal boundaries enforcedv0.14.0
Answer first
Why not optimize Concresca for one search engine or AI system?
Because a search engine is not the internet, and an answer engine is not the only machine receiver. Concresca documents receiver-specific publisher controls from official sources while keeping the underlying content architecture standards-based and platform-independent.
Receiver policy
Separate discovery, retrieval and training controls
Google, Bing/Copilot, ChatGPT search, Perplexity, Brave, Apple, Claude/Anthropic, Common Crawl, Yandex and IndexNow expose different documented mechanisms. Concresca records those differences rather than pretending one crawler directive has the same meaning everywhere.
Crawler accessibility is implementation readiness, not evidence that a platform has indexed, ranked, cited, trained on or endorsed Concresca.
Measurement
Observed performance requires observed data
Bing now documents AI citation and grounding-query measurements. Other platforms expose different or more limited publisher measurements. Concresca will label unavailable live metrics NOT CONNECTED rather than zero.
All documented receivers
13 receiver-specific surfaces
The matrix records documented publisher controls and measurement surfaces. It does not claim that any receiver has crawled, indexed, ranked, cited, trained on, or endorsed Concresca.
Bing Search / Copilot
Receiver/crawler: bingbot
Purpose: Search crawling and AI grounding
Publisher controls: robots.txt; sitemap; IndexNow
Measurement: Bing Webmaster Tools AI Performance
ChatGPT search
Receiver/crawler: OAI-SearchBot
Purpose: Search discovery / answer retrieval
Publisher controls: robots.txt
Measurement: Referral/server logs; official publisher controls; no performance claim without measurement
OpenAI model-development crawler
Receiver/crawler: GPTBot
Purpose: Model development/training access control
Publisher controls: robots.txt
Measurement: Crawler logs where available
Perplexity search
Receiver/crawler: PerplexityBot
Purpose: Search indexing/retrieval
Publisher controls: robots.txt
Measurement: Referral/server logs where available
Brave Search
Receiver/crawler: Brave search crawler
Purpose: Independent search crawling
Publisher controls: robots.txt / documented Brave controls
Measurement: Brave search visibility where measurable
Apple search/discovery
Receiver/crawler: Applebot
Purpose: Search/discovery and contextual grounding
Publisher controls: robots.txt
Measurement: Server logs where available
Apple foundation-model extended use
Receiver/crawler: Applebot-Extended
Purpose: Publisher control for model use of Applebot-crawled content
Publisher controls: robots.txt
Measurement: Crawler/control audit
Claude search / user retrieval
Receiver/crawler: Claude-SearchBot; Claude-User
Purpose: Search and user-directed retrieval
Publisher controls: robots.txt
Measurement: Server logs where available
Anthropic model-development crawler
Receiver/crawler: ClaudeBot
Purpose: Model development crawling
Publisher controls: robots.txt
Measurement: Crawler logs where available
Common Crawl
Receiver/crawler: CCBot
Purpose: Open web corpus crawling
Publisher controls: robots.txt; sitemaps
Measurement: Common Crawl index/server logs where available
Yandex Search
Receiver/crawler: YandexBot
Purpose: Search crawling/indexing
Publisher controls: robots.txt
Measurement: Yandex Webmaster where available
IndexNow participants
Receiver/crawler: n/a
Purpose: URL-change notification, not crawling itself
Publisher controls: IndexNow key + submission API
Measurement: Submission logs / observed recrawl only after authorized live submission
Clickable evidence
Sources supporting the factual layer
Source links establish traceability and support. They do not imply that the source endorses Concresca’s constitutional proposals.
SRC-GGL-02 · official technical guidance
robots.txt specification
Documents Google robots.txt interpretation and crawler access rules.
Google Search Central — official/source page ↗SRC-BING-01 · official technical guidance
AI Performance in Bing Webmaster Tools
Documents AI citation counts, cited pages, grounding queries, and citation trends.
Microsoft Bing — official/source page ↗SRC-OAI-01 · official publisher guidance
Publishers and developers FAQ
Documents OAI-SearchBot controls for ChatGPT search and distinguishes search discovery from GPTBot model-training controls.
OpenAI — official/source page ↗SRC-PPLX-01 · official publisher guidance
How Perplexity follows robots.txt
Documents PerplexityBot robots.txt behavior for search indexing.
Perplexity — official/source page ↗SRC-BRV-01 · official crawler guidance
Brave Search crawler
Documents Brave Search crawling behavior and publisher controls.
Brave — official/source page ↗SRC-APL-01 · official crawler guidance
About Applebot
Documents Applebot robots controls and Applebot-Extended publisher controls for foundation-model use.
Apple — official/source page ↗SRC-ANT-01 · official crawler guidance
Does Anthropic crawl data from the web?
Documents ClaudeBot, Claude-SearchBot, and Claude-User as separate crawlers with separate controls.
Anthropic — official/source page ↗SRC-CC-01 · official crawler guidance
Common Crawl FAQ
Documents CCBot, robots.txt, sitemaps, and crawl inclusion limits.
Common Crawl — official/source page ↗SRC-YDX-01 · official crawler guidance
Yandex robots.txt documentation
Documents Yandex robots.txt behavior.
Yandex — official/source page ↗SRC-IDX-01 · protocol documentation
IndexNow documentation
Documents URL-change notification and ownership verification.
IndexNow — official/source page ↗SRC-IDX-02 · protocol documentation
IndexNow participating search engines
Documents participating search engines and notification sharing.
IndexNow — official/source page ↗Judgment-free total cognitive freedom
NO JUDGMENT WHATSOEVER. Concresca coordinates without assigning moral worth, character, guilt, danger, trustworthiness, loyalty, purity, normality, or social standing. Questions, thoughts, identities, messages, content, and conduct are not objects of Concresca judgment.