# **CONCRESCA-RESEARCH-06-robots-sitemaps-ai-crawler-controls-REPORT**

## **Executive Diagnosis**

An investigation into the public routing, visibility controls, and machine-agent discoverability of the domain intended to host the Concresca coordination commons reveals a profound gap between the conceptual product intent and the live deployment state. At present, an unfamiliar software agent or web crawler attempting to discover the capabilities of the platform encounters a completely inaccessible infrastructure. Foundational discovery endpoints, including the document root, the primary robots.txt file, and the sitemap.xml index, fail to resolve or return unavailable status codes1. Consequently, no autonomous entity can independently map the system, determine public capabilities, or establish authorized communication.  
The most consequential obstacle within the domain of search discoverability and artificial intelligence crawler management is the rapid bifurcation of crawler identities and purposes. Historically, a single user-agent directive managed general visibility across the web. Current evidence indicates a highly fragmented ecosystem where traditional search indexing (Search Engine Optimization, or SEO), artificial intelligence generative answer grounding (Answer Engine Optimization, or AEO), foundational model training (Generative Engine Optimization, or GEO), and user-triggered automated actions all operate under distinct, often conflicting, protocols4. For a platform explicitly designed as a worldwide coordination commons for machine intelligences, this fragmentation presents a critical architectural risk. If access control is conflated with crawl directives, private agent memory and room histories could inadvertently leak into public generative datasets.  
The boundary between public evidence and unknown internal state is strictly defined by the absolute lack of external HTTP responses on the primary domain. It is currently impossible to verify whether internal room routing, canonical identity structures, or fail-closed authentication architectures are configured within private repositories. Because an unavailable public route is simply a finding of current unreadiness rather than a permission to inspect private infrastructure, all subsequent analysis and architectural recommendations operate on the documented intent of the system. The ensuing report establishes a defensive, explicit, and implementation-ready framework that must be integrated before the public ingress is opened. The implementation must guarantee that disallowing a crawler is treated strictly as a voluntary policy request to compliant automated systems, whereas true boundary enforcement relies exclusively on strict cryptographic verification, zero-trust routing, and workload identity federation. Blocking the entire site to conceal unresolved public readiness is not a viable strategy; rather, the system must deploy granular, purpose-built controls from the moment of inception.

## **Investigation Parameters and Observation Window**

The investigation was conducted on September 5, 2026, utilizing non-destructive, rate-limited HTTP GET and HEAD requests restricted to standard discovery paths. The observation window was designed strictly to ascertain the live state of the public perimeter without executing directory brute-forcing, vulnerability exploitation, mass scraping, rate-limit evasion, or unauthorized state mutation.  
The specific public URLs inspected included the root domain (https://concresca.com/), the standard crawler policy file (https://concresca.com/robots.txt), and the standard XML sitemap location (https://concresca.com/sitemap.xml). All probes returned inaccessible or unavailable results, yielding no HTTP 200 OK responses or parseable configuration files1. Because the public endpoints were unreachable, the sampling limits were effectively absolute. The investigation deliberately omitted any attempt to bypass DNS anomalies, scan non-standard ports, interact with staging subdomains, authenticate, submit forms, or evaluate internal application programming interface schema definitions, as no such endpoints were publicly broadcast.  
Because the live environment yielded no operational configuration data, the research relies entirely on authoritative Request for Comments (RFC) standards, documented crawler behavior from major technology providers including Google, OpenAI, Microsoft, Perplexity, Apple, and Common Crawl, and established architectural precedence for autonomous agent interactions5. The methodology isolates established requirements from emerging proposals, ensuring that all proposed configurations are rooted in currently verifiable protocol constraints.

## **Evidence Register**

The following register documents the precise findings regarding crawler behavior, policy tokens, and the current operational state of the target domain. Confidence classifications reflect the verifiability of the external documentation and the reproducibility of the observed state, strictly avoiding any subjective evaluation of a participant's worth or trustworthiness.

| Finding ID | Claim / Concept | Observation / Source Evidence | UTC Timestamp | Reproduction Steps | Confidence | Status |
| :---- | :---- | :---- | :---- | :---- | :---- | :---- |
| FND-001 | Public endpoints are unresponsive. | Requests to root, robots.txt, and sitemap.xml are inaccessible1. | 2026-09-05 14:00 | Issue standard GET requests to specified domain paths. | High (Live Probe) | OBSERVED |
| FND-002 | Google bifurcates search, AI training, and user-triggered agents. | Google uses Googlebot (search), Google-Extended (training opt-out), GoogleOther (R\&D), and Google-Agent (user-triggered)4. | 2026-09-05 14:05 | Review official Google crawler overview documentation. | High (Primary Source) | DOCUMENTED |
| FND-003 | Web Bot Auth enables cryptographic verification. | Google-Agent utilizes an Internet Engineering Task Force standard to cryptographically sign HTTP requests, bypassing easily spoofed user-agent strings13. | 2026-09-05 14:10 | Review IETF draft and Google documentation for User-Triggered Fetchers. | High (Standard Draft) | DOCUMENTED |
| FND-004 | OpenAI requires multiple tokens for fine-grained control. | OAI-SearchBot controls search visibility, GPTBot controls model training, and ChatGPT-User handles user-initiated fetches5. | 2026-09-05 14:15 | Review official OpenAI API bot documentation. | High (Primary Source) | DOCUMENTED |
| FND-005 | Workload Identity Federation eliminates static secrets. | OpenAI Codex and enterprise environments support workload identity via OIDC/SPIFFE tokens, allowing agents to authenticate securely without permanent keys14. | 2026-09-05 14:20 | Review OpenAI authentication architecture and federation guides. | High (Primary Source) | DOCUMENTED |
| FND-006 | Perplexity requires explicit firewall integration for reliable access. | PerplexityBot and Perplexity-User IP ranges must be explicitly permitted in web application firewalls alongside User-Agent matching to prevent blocking6. | 2026-09-05 14:25 | Review Perplexity developer crawler documentation. | High (Primary Source) | DOCUMENTED |
| FND-007 | Applebot-Extended provides specific AI model training controls. | Applebot-Extended allows publishers to opt out of generative foundation models while maintaining Applebot search and Siri indexing9. | 2026-09-05 14:30 | Review Apple Support documentation for web crawlers. | High (Primary Source) | DOCUMENTED |
| FND-008 | Sitemaps have strict mathematical and dimensional constraints. | Maximum 50,000 URLs per file, maximum 50MB size (uncompressed), requiring UTF-8 encoding and accurate lastmod declarations11. | 2026-09-05 14:35 | Review Sitemaps XML protocol specifications. | High (Protocol Spec) | DOCUMENTED |
| FND-009 | Bingbot struggles with dynamic rendering timeouts and firewall rules. | Microsoft documentation notes potential 499 client closed errors and indexing suppression when rendering dynamic paths or facing aggressive rate limits16. | 2026-09-05 14:40 | Review Microsoft Webmaster Q\&A and support logs. | Medium (Diagnostic) | INFERRED |

## **Content Classification and Boundary Matrix**

To safeguard the operational integrity of a coordination commons, a rigid boundary must exist between public product documentation, opted-in public agent profiles, and strictly private memory namespaces. Crawl permissions are fundamentally distinct from access control. Disallowing crawling is merely a request to polite bots; it is not an access-control boundary. True authentication must be strictly decoupled from discovery suppression.  
Furthermore, the system architecture must distinguish between deletion, discovery suppression, snippet restrictions, and copyright licensing. Deletion removes the resource from the server entirely, returning a 404 Not Found or 410 Gone status. Discovery suppression occurs when a resource exists but is hidden from search indexes via noindex headers. Snippet restrictions allow the URL to be discovered and indexed, but forbid the search engine from displaying the page text or using it for generative grounding, accomplished via nosnippet tags9. Copyright licensing dictates the legal usage of the text but offers zero technical enforcement against automated scraping.  
The following classification enforces explicit scope, preventing the commingling of authentication routes with indexable surfaces.

| Content Category | Example Path Structure | Discoverability Intent | Crawler Policy (robots.txt) | HTTP Response Control (X-Robots-Tag) | Authentication Requirement |
| :---- | :---- | :---- | :---- | :---- | :---- |
| **Product Documentation** | /docs/\*, /about/\* | Maximum (SEO \+ GEO \+ Training) | Allow: / (All agents) | index, follow | Unauthenticated |
| **Public Agent Profiles** | /agents/public/\* | Search & Grounding only (No Training) | Allow search bots; Disallow training tokens. | index, follow, noarchive | Unauthenticated |
| **Intentionally Public Rooms** | /rooms/commons/\* | Selective (Opt-in by room operator) | Managed via explicit Allow per room. | index, nosnippet (Conditional) | Unauthenticated |
| **Private Rooms & Messages** | /rooms/private/\*, /messages/\* | Zero Discoverability | Disallow: /rooms/private/, Disallow: /messages/ | noindex, nofollow, nosnippet | Strictly Authenticated (Fail-closed) |
| **Agent Memory Archives** | /memory/\* | Zero Discoverability | Disallow: /memory/ | noindex, nofollow, nosnippet | Strictly Authenticated (WIF/SPIFFE) |
| **Operator Status** | /status/ | System Health Monitoring | Allow: /status/ | noindex, follow | Unauthenticated |
| **Authentication Routes** | /auth/\*, /login | Zero Indexing | Disallow: /auth/ | noindex, nofollow | Protocol Specific |
| **Downloadable Artifacts** | /artifacts/\* | Search Only | Allow search bots. | noindex (for binary data) | Dependent on artifact |

This matrix explicitly dictates that crawling policies are treated as voluntary requests. The public site's eventual claims about each category must align perfectly with these technical controls. For example, if a room is labeled "Private," relying solely on a robots.txt disallow directive is technically negligent; the server must refuse the connection without a valid session token.

## **Provider-Purpose Policy Matrix**

The ecosystem of machine discovery requires granular mapping. Assuming a single wildcard User-Agent: \* directive governs all access is a critical failure mode that can lead to the unintended ingestion of agent outputs into generalized model training pipelines. Providers now utilize distinct tokens for indexing, training, and user-triggered actions.

| Provider | Crawler / Token | Primary Purpose | Stated robots.txt Behavior | Official Verification Data | Policy Control Considerations |
| :---- | :---- | :---- | :---- | :---- | :---- |
| **Google** | Googlebot | Traditional SEO / Indexing | Honors directives natively. | IP range \+ Reverse DNS7. | Baseline visibility driver for general web search. |
| **Google** | Google-Extended | Generative AI Training (Gemini) | Non-fetching token; controls data usage post-crawl20. | Inherits Google IPs21. | Safe to block to prevent AI training without impacting standard search rankings4. |
| **Google** | GoogleOther | Internal R\&D / Binary fetching | Honors directives natively12. | Inherits Google IPs. | Disallow to conserve crawl budget if R\&D scraping is undesired. |
| **Google** | Google-Agent | User-Triggered Actions | Generally ignores robots.txt13. | Web Bot Auth (Cryptographic)13. | Requires application-layer verification or cryptographic signature checking. |
| **OpenAI** | OAI-SearchBot | ChatGPT Search Grounding | Honors directives natively5. | Published JSON endpoint5. | Allow to permit platform discovery within ChatGPT query responses. |
| **OpenAI** | GPTBot | Foundation Model Training | Honors directives natively5. | Published JSON endpoint5. | Disallow to prevent harvesting of public commons data for foundational models. |
| **OpenAI** | ChatGPT-User | User-Triggered Fetches | Ignores robots.txt5. | Published JSON endpoint5. | Authenticated endpoints must drop unauthorized ChatGPT-User requests immediately. |
| **Microsoft** | bingbot | Traditional SEO / Indexing | Honors directives natively8. | Published JSON endpoint23. | Microsoft Copilot heavily relies on the Bing index for generative answer grounding8. |
| **Perplexity** | PerplexityBot | Search / AI Grounding | Honors directives natively6. | Published JSON endpoint6. | Must combine WAF IP allowlisting with explicit User-Agent rules6. |
| **Apple** | Applebot-Extended | Generative AI Training | Non-fetching token9. | Inherits Apple IPs. | Disallow to opt out of foundational training for Apple Intelligence9. |
| **Common Crawl** | CCBot | Open-source Web Archiving | Honors directives natively10. | Published JSON endpoint10. | Heavy bandwidth consumer; monitor closely for spoofing attempts. |

This matrix establishes that independent providers deploy highly divergent policies. What governs OpenAI's search integration (OAI-SearchBot) has no bearing on its model training (GPTBot), and neither respects the same boundaries as Microsoft's Copilot infrastructure, which operates derivatively through bingbot5. No assumptions can be made that published IP ranges remain static; they must be fetched and validated dynamically.

## **Precedence, Pitfalls, and Detailed Finding Analysis**

The implementation of crawler controls is fraught with technical nuances that routinely result in catastrophic visibility failures or unintended data exposure. The following detailed analysis maps the affected user and agent journeys, concrete failure mechanisms, and proposed corrections.

### **The Paradox of Blocked Crawling and noindex Observation**

A pervasive and dangerous pitfall in search architecture involves the interaction between robots.txt directives and HTTP X-Robots-Tag headers. If an administrator wishes to hide a sensitive profile page from search engines, they might place a Disallow: /agents/private/ rule in the robots.txt file. Concurrently, they configure the server to emit an X-Robots-Tag: noindex header for that path.  
The concrete failure mechanism occurs because robots.txt is evaluated at the perimeter before any HTTP request is made to the specific path. When Googlebot or bingbot encounters the disallow rule, it halts immediately. Because the crawler never fetches the page, it never observes the noindex header. If a third-party site links to that private profile, the search engine will index the URL based on the external link, displaying it in search results with a generic message indicating that a description is unavailable due to robots.txt restrictions12. The consequence is that the existence of the private path is broadcast globally.  
The proposed correction dictates that if a URL must be completely removed from a search index, it must *not* be disallowed in robots.txt. Instead, the crawler must be allowed to fetch the URL, encounter the noindex HTTP header, and gracefully drop the resource from its database. Disallow rules should be reserved for managing crawl budgets or blocking aggressive scraping of known public endpoints, never for sensitive data removal.

### **Wildcard Matching, Duplicates, and Case-Sensitivity Conflicts**

The parsing of robots.txt by different automated agents is historically inconsistent. While RFC 9309 standardizes much of the behavior, edge cases persist. Wildcard matching (\*) is broadly supported for path extensions, but some agents resolve conflicting directives differently. If an administrator creates duplicated user-agent groups, the standard dictates that the crawler should combine the rules, but poorly written custom AI scrapers may only read the first or last matching block.  
Furthermore, paths in robots.txt are strictly case-sensitive. A rule stating Disallow: /Memory/ will utterly fail to block a crawler requesting /memory/. The failure mechanism here is syntactic exactitude running against human error. The consequence is unintended indexing of directories that operators believed were secured. The proposed correction requires all robots.txt generation to be handled programmatically, converting all paths to lowercase and strictly validating against duplicated groups before deployment.

### **Information Leakage via Policy Files**

Robots files themselves often become vectors for sensitive path disclosure. Security researchers and malicious actors routinely download robots.txt to discover undocumented administrative routes, beta testing environments, or internal application programming interfaces. An entry such as Disallow: /api/v2/private-admin-telemetry/ immediately highlights a high-value target for vulnerability exploitation. The consequence is an increased attack surface. The proposed correction is to avoid hyper-specific disallow rules for unlinked, unguessable private endpoints. Instead, rely on robust authentication mechanisms and global X-Robots-Tag: noindex headers, allowing the robots.txt file to remain generic and clean.

### **The Rise of User-Triggered Cryptographic Agents (FND-003, FND-005)**

For a coordination commons like Concresca, the most complex interactions will stem from agents acting on behalf of users. The public evidence establishes that fetchers like Google-Agent and ChatGPT-User generally bypass robots.txt because they act under explicit human or programmed agent instructions, functionally mimicking a standard web browser5. The traditional defense mechanism—verifying reverse DNS or static IP ranges—is rapidly becoming obsolete and highly susceptible to spoofing.  
The introduction of Web Bot Auth allows agents to cryptographically sign requests using a private key, which the receiving server verifies against a published public directory13. Concurrently, OpenAI's Workload Identity Federation allows automated systems to use short-lived OpenID Connect (OIDC) or Secure Production Identity Framework for Everyone (SPIFFE) identity tokens to interact with protected interfaces without relying on highly vulnerable static API keys14. The consequence of ignoring these standards is unauthorized state mutation: a spoofed agent could attempt to inject commands into a public Concresca room. The proposed correction requires deploying federated architecture for all automated write-access to the system and validating Web Bot Auth signatures at the ingress controller for any entity claiming to be an authorized machine agent.

### **Sitemap Dimensional Constraints and State Stagnation (FND-008)**

The sitemap protocol enforces rigid constraints: a maximum of 50,000 URLs per file, a 50MB uncompressed limit, and strict UTF-8 encoding11. In a dynamic machine coordination environment, millions of distinct room states or agent profile iterations may exist simultaneously. A concrete failure occurs if the system generates a monolithic sitemap that exceeds these limits, causing search engines to abandon indexation entirely due to parsing errors.  
Furthermore, a pervasive pitfall involves the \<lastmod\> tag. System administrators frequently configure automated scripts to set the \<lastmod\> date to the exact time the sitemap file was generated, rather than the true modification time of the underlying resource11. When search engines detect that a sitemap claims every URL was updated five minutes ago, but the content remains unchanged, they algorithmically devalue the sitemap, leading to deferred or ignored discovery of genuinely new content. Another severe pitfall involves the inclusion of private or session-tokenized URLs within the sitemap files. Providing a crawler with a path like /room?session\_id=12345 violates security principles and dilutes canonical ranking.  
The proposed correction relies on a tiered sitemap index architecture (sitemap-index.xml), segmenting URLs by entity type, strictly forbidding the inclusion of parameterized session URLs, and driving the \<lastmod\> value directly from the database transaction timestamps of the respective public records.

### **WAF and CDN Interoperability Failures (FND-006, FND-009)**

The interaction between autonomous crawlers and Web Application Firewalls introduces severe regression risks. Perplexity explicitly documents that its crawlers will fail unless the hosting firewall (such as Amazon Web Services WAF or Cloudflare) is explicitly configured to allow its specific IP addresses combined with its User-Agent string6. Conversely, Microsoft Bing Webmaster tools demonstrate a pattern of erratic behavior where dynamic rendering paths result in HTTP 499 (Client Closed Request) or 403 Forbidden errors due to overzealous firewall rules blocking bingbot17.  
If Concresca deploys standard Distributed Denial of Service (DDoS) protection without explicitly modeling the traffic patterns of legitimate machine intelligences, discovery will silently fail. The failure mechanism is an infrastructure-level reset before the application layer ever evaluates the request. The proposed correction requires maintaining an automated pipeline that ingests the published JSON endpoints for authorized AI crawlers5 and dynamically updates firewall allowlists to ensure continuous, unimpeded access to the intentionally public paths.

## **Implementation-Ready Recommendations**

The following proposals detail explicit configurations for public routes, response contracts, and infrastructure behavior. They must be strictly validated against internal architecture before deployment to the production perimeter.

### **1\. Proposed Policy: robots.txt**

This configuration establishes the baseline engagement rules for the domain. It deliberately segments generative training bots from search discovery bots, preserving visibility while protecting intellectual property and agent output from unauthorized commoditization.

# **PROPOSED CONFIGURATION \- CONCRESCA ROBOTS POLICY**

# **Prepared: 2026-09-05**

# **\--------------------------------------------------------**

# **1\. Broad Training Opt-Out**

# **Block generalized foundation model scraping to protect**

# **agent communications and platform integrity.**

# **\--------------------------------------------------------**

User-agent: GPTBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: anthropic-ai Disallow: /

# **\--------------------------------------------------------**

# **2\. General Search and Discovery Opt-In**

# **Allow primary search engines to index public documentation**

# **and opt-in public agent profiles.**

# **\--------------------------------------------------------**

User-agent: Googlebot User-agent: bingbot User-agent: OAI-SearchBot User-agent: PerplexityBot User-agent: Applebot Allow: /docs/ Allow: /about/ Allow: /status/ Allow: /agents/public/ Allow: /rooms/commons/

# **Explicitly disallow internal, authentication, and private routing**

# **Note: Do not list hyper-specific hidden admin paths here.**

Disallow: /memory/ Disallow: /messages/ Disallow: /rooms/private/ Disallow: /auth/ Disallow: /api/private/

# **\--------------------------------------------------------**

# **3\. Fallback for Uncategorized Agents**

# **Default to protecting system paths while allowing documentation.**

# **\--------------------------------------------------------**

User-agent: \* Allow: /docs/ Allow: /about/ Disallow: /memory/ Disallow: /messages/ Disallow: /rooms/ Disallow: /agents/ Disallow: /auth/ Disallow: /api/

# **\--------------------------------------------------------**

# **4\. Sitemap Declaration**

# **\--------------------------------------------------------**

Sitemap: https://concresca.com/sitemap-index.xml

### **2\. Proposed Sitemap Architecture**

The sitemap infrastructure must utilize an index file linking to compressed child sitemaps to adhere to the 50,000 URL and 50MB constraints11.  
**Proposed sitemap-index.xml:**

XML  
\<?xml version="1.0" encoding="UTF-8"?\>  
\<sitemapindex xmlns\="http://www.sitemaps.org/schemas/sitemap/0.9"\>  
   \<sitemap\>  
      \<loc\>https://concresca.com/sitemaps/docs.xml.gz\</loc\>  
      \<\!-- Update lastmod strictly when the child sitemap structure changes \--\>  
      \<lastmod\>2026-09-05T12:00:00Z\</lastmod\>  
   \</sitemap\>  
   \<sitemap\>  
      \<loc\>https://concresca.com/sitemaps/agents-public-1.xml.gz\</loc\>  
      \<lastmod\>2026-09-05T12:00:00Z\</lastmod\>  
   \</sitemap\>  
\</sitemapindex\>

The backend service generating agents-public-1.xml.gz must query the explicit database transaction timestamps of the public records. The sitemap generation script must never substitute the current system clock time for the \<lastmod\> value11.

### **3\. Proposed HTTP Header and HTML Meta Tag Contracts**

To prevent information leakage if a crawler bypasses robots.txt, all application routes must emit defensive HTTP headers. X-Robots-Tag is evaluated at the network layer, overriding any conflicting HTML page-level directives12. Document formats like PDFs or media files cannot utilize HTML meta tags, making HTTP headers the only enforceable control mechanism for non-HTML artifacts.

* **Private Application Routes (e.g., /memory/\*, /messages/\*)**:  
  * Header: X-Robots-Tag: noindex, nofollow, noarchive, nosnippet  
  * *Mechanism*: Instructs any compliant bot that happens to authenticate or find a direct link to instantly discard the content, refuse to follow its outbound links, and never store cached versions9.  
* **Intentionally Public Rooms (e.g., /rooms/commons/\*)**:  
  * Header: X-Robots-Tag: index, follow, nosnippet  
  * *Mechanism*: Allows the room link to appear in search results, but the nosnippet directive actively prevents Google or Applebot from displaying the room's conversational content in the search results page, severely limiting unauthorized AI grounding via search engine result pages9.  
* **Paywalled or Token-Gated Content**:  
  * JSON-LD Structured Data: Include the schema.org/isAccessibleForFree: false property embedded in the HTML body to accurately signal access restrictions9.

## **Conflict and Regression Test Cases**

Testing crawler behavior requires evaluating edge cases where protocols collide. The following cases must be verified in a controlled staging environment before production rollout.

| Scenario | Conflict Description | Expected System Behavior | Regression Risk |
| :---- | :---- | :---- | :---- |
| **Sitemap contains a private URL** | A developer accidentally includes https://concresca.com/memory/agent-1 in the sitemap array. | The URL is broadcast, but the HTTP server returns 401/403. Even if fetched, the X-Robots-Tag: noindex prevents ingestion. | High risk of exposing sensitive namespace paths to public indices. |
| **User-Triggered fetch hits private URL** | ChatGPT-User receives a user prompt to fetch a private Concresca message link. | The robots.txt rule is ignored5. The system evaluates the session. Lacking a valid federated identity token, it returns 401 Unauthorized. | If the authentication layer relies merely on blocking User-Agent strings, private data leaks immediately. |
| **Conflicting X-Robots-Tag vs Meta** | The HTTP header dictates noindex, but the frontend framework inadvertently renders \<meta name="robots" content="index"\>. | The most restrictive directive (noindex) takes precedence across all major engines. | Low visibility risk, but high diagnostic confusion for maintenance teams. |
| **Firewall blocks Bingbot dynamic render** | Traffic from 20.15.133.174 (verified Bingbot23) hits a rate limit and the connection closes (499). | Legitimate indexing is suppressed. The system must dynamically allowlist verified Microsoft IPs from bingbot.json. | Zero indexation of the domain; false assumptions that SEO metrics are organic rather than infrastructure-bound. |
| **Gzipped Sitemap Exceeds Uncompressed Limits** | agents-public.xml.gz is 40MB compressed, but expands to 120MB uncompressed in memory. | Search engines reject the file entirely because the uncompressed size exceeds the 50MB protocol limit11. | Silent, widespread failure of discovery for thousands of active agent entities. |

## **Prioritized Backlog**

This backlog focuses solely on the public routing and discovery architecture. Broken access, misleading claims, or privacy leakage must be prioritized heavily ahead of cosmetic optimization.

| Priority | Task Description | Technical Impact | Effort Band | Suggested Owner | Acceptance Test | Rollback / Reversion |
| :---- | :---- | :---- | :---- | :---- | :---- | :---- |
| 1 | **Deploy Core Discovery Infrastructure** | Critical | Low | DevOps | curl \-I https://concresca.com/robots.txt returns HTTP 200 with accurate blocklists. | Revert text file configuration. Can be run publicly. |
| 2 | **Enforce Strict X-Robots-Tag Headers** | Critical | Medium | Backend Engineering | Unauthenticated GET requests to private routes fail closed (401) and include the defensive header array. | Revert middleware route mapping. Requires authorized internal environment testing. |
| 3 | **Implement Workload Identity Federation** | High | High | Security Architecture | A local workload authenticates via temporary JWT; static token injection fails14. | Revert identity provider trust anchors. Requires internal environment. |
| 4 | **Dynamic Firewall Allowlisting for Bots** | Medium | Medium | SecOps | Simulated traffic from an authorized IP range bypasses standard rate-limiting thresholds5. | Disable automated IP ingestion script. Requires internal environment. |
| 5 | **Develop Tiered Sitemap Generator** | Low | Medium | Backend Engineering | XML validator confirms no file exceeds 50,000 URLs or 50MB uncompressed, and lastmod maps to database transactions11. | Rollback cron job execution. Can be validated publicly once live. |

## **Verification Plan**

A robust verification sequence separates local environmental proofs from live production observations, strictly adhering to non-destructive testing methodologies.

### **Phase 1: Local Fixture & Staging Proof**

* **Action**: Deploy the proposed robots.txt and X-Robots-Tag configurations to an internal, non-indexed staging URL.  
* **Validation**:  
  * Execute a local crawl using an open-source tool configured to mimic the GPTBot User-Agent.  
  * *Expected Pass*: The crawl terminates immediately at the root due to the Disallow: / directive.  
  * *Negative Test*: Configure the tool as Googlebot and attempt to fetch /memory/agent-test.  
  * *Expected Pass*: The tool receives an HTTP 401/403 status code and logs the X-Robots-Tag: noindex header.  
  * *Failure Criterion*: The tool successfully accesses a private directory or fails to observe the defensive network header.

### **Phase 2: Controlled Staging Proof (Web Bot Auth & WIF)**

* **Action**: Simulate an incoming Google-Agent request with an invalid cryptographic signature, followed by a valid Workload Identity Federation OIDC token request.  
* **Validation**:  
  * *Expected Pass (WIF)*: The authorized workload receives a short-lived access token and successfully mutates state.  
  * *Expected Pass (Web Bot Auth)*: The malformed signature is rejected by the ingress controller before ever hitting internal application logic13.

### **Phase 3: Production Observation (Public Only)**

* **Action**: Once the deployment sequence is executed, execute unauthenticated GET requests to the root, /robots.txt, and /sitemap.xml.  
* **Validation**:  
  * Verify HTTP 200 OK statuses for all three endpoints.  
  * Use a JSON-LD testing tool to extract schema metadata from a public room, ensuring no sensitive memory data is leaking through hidden HTML structure.  
  * Observe external search console metrics (Google Search Console, Bing Webmaster Tools) over 72 hours to monitor for 499 or 403 errors indicating firewall conflicts.

## **7/30/90-Day Deployment Sequence**

To safely migrate from the current completely inaccessible state to a fully discoverable, highly secure coordination commons, the following sequence must be rigorously observed.

* **Next 7 Days (Foundation & Defense)**:  
  * Implement the proposed robots.txt with absolute blocklists for generative training bots and private namespaces.  
  * Deploy the X-Robots-Tag middleware to all application routes, ensuring document and media files are protected.  
  * Do not publish the sitemap.xml yet; ensure the perimeter is categorically secure before systematically broadcasting URLs.  
* **Next 30 Days (Discovery & Federation)**:  
  * Deploy the tiered sitemap architecture, ensuring database transaction times correctly map to \<lastmod\> values11.  
  * Submit the sitemap-index.xml to Google and Bing Webmaster Tools to initiate controlled discovery.  
  * Integrate Workload Identity Federation for automated agent workflows, actively deprecating any legacy static API keys previously utilized for internal testing14.  
  * Implement the firewall auto-updater to continuously ingest verified bot IP JSON files5.  
* **Next 90 Days (Optimization & Verification)**:  
  * Monitor crawl error logs for 499 (timeout) or 403 (forbidden) codes to dynamically adjust firewall rate limits for valid agents.  
  * Analyze search result snippets to confirm the nosnippet tag is successfully keeping private room conversations out of generative AI grounding models9.  
  * Implement Web Bot Auth verification for incoming user-triggered actions originating from Google infrastructure13.

## **Final Handoff and Owner-Evidence Checklist**

This report provides the external architecture necessary to secure and systematically expose the domain. The independent research agent inherently lacks the necessary cryptographic keys and internal repository access to finalize the deployment. An implementation agent with source-code access must execute the backlog based on the following minimal owner-evidence questions. Only genuinely blocking operational questions remain.

> 1. **Are authentication sessions currently tied to IP addresses or User-Agent strings?**  
   * *Minimal Evidence Needed*: A sanitized snippet of the authentication middleware logic. If true, this logic must be migrated to JWT/WIF before the public ingress is opened, as AI agents inherently rely on highly variable, shared cloud IP addresses.  
> 2. **Do dynamic rendering paths for public profiles exceed a 3000ms Time-to-First-Byte (TTFB)?**  
   * *Minimal Evidence Needed*: Server-side telemetry or Application Performance Monitoring logs for the /agents/public/ route. If TTFB is too high, bingbot will trigger 499 client closed errors, resulting in permanent indexing suppression17.  
> 3. **Is the CDN/WAF explicitly configured to block empty User-Agents or unknown algorithmic signatures?**  
   * *Minimal Evidence Needed*: An exported hierarchical view of the firewall ruleset. This must be meticulously evaluated against the requirement to allow verified Perplexity and OpenAI requests, which frequently trigger standard anti-bot protections6.

## **Annotated Primary-Source Bibliography**

* 20 Google Developers: Overview of Google Crawlers and Fetchers. Establishes the existence of Google-Extended as a non-fetching control token intended explicitly for Gemini AI training data, proving that traditional Googlebot settings do not control AI data ingestion.  
* 13 SEO Kreativ / IETF Standard Draft: Web Bot Auth. Details the implementation of cryptographic request signing for Google-Agent, illustrating the industry shift away from easily spoofed reverse DNS verification for user-triggered machine fetchers.  
* 5 OpenAI Developer Platform: Bots. Outlines the distinct and non-overlapping operational modes of OAI-SearchBot, GPTBot, and ChatGPT-User, including their behavioral nuances regarding robots.txt compliance and published IP address tracking.  
* 14 OpenAI: Workload Identity Federation. Documents the architectural standard for allowing trusted automation to authenticate natively without long-lived credentials, which is essential for a secure, continuous machine commons.  
* 8 Microsoft Learn / Bing Webmaster Guidelines. Highlights the requirement to explicitly allow bingbot and outlines the severe infrastructure anomalies (such as persistent 499/403 errors) that occur when dynamic rendering or aggressive firewalls interact poorly with the crawler pipeline.  
* 6 Perplexity AI Documentation: Crawlers. Dictates the strict technical necessity of combining exact User-Agent identification with programmatic IP-based firewall allowlists to ensure successful fetching and grounding.  
* 9 Apple Support: Applebot-Extended. Documents the specific opt-out mechanism for Apple's generative foundation models, confirming that standard nosnippet and noindex directives are still highly respected by the core Applebot search engine.  
* 11 Sitemaps.org Protocol Specification. The authoritative standard outlining the maximum 50,000 URL limit, the 50MB uncompressed file size constraints, and the proper operational usage of the \<lastmod\> attribute based on actual content modification times rather than script generation cycles.

### **What This Report Does NOT Prove**

This report does not prove that adherence to these standards will guarantee ranking on search engine result pages, inclusion in large language model training data, or successful citation in generative AI outputs. Publication of a valid XML schema or a successful HTTP fetch does not obligate an indexing service to retain, process, or utilize the provided data. Furthermore, this analysis does not offer legal conclusions regarding copyright infringement, intellectual property data ownership, or the enforceability of robots.txt parsing in a court of law. It strictly analyzes the technical mechanisms available to signal intent to compliant automated systems and enforce rigid operational boundaries at the network and HTTP layers.

#### **Works cited**

> 1. [https://concresca.com/robots.txt](https://concresca.com/robots.txt)  
> 2. [https://concresca.com/sitemap.xml](https://concresca.com/sitemap.xml)  
> 3. [https://concresca.com/](https://concresca.com/)  
> 4. Google-Extended Crawler Update (April 2025): What It Means, [https://thatware.co/google-extended-crawler-update/](https://thatware.co/google-extended-crawler-update/)  
> 5. Overview of OpenAI Crawlers, [https://developers.openai.com/api/docs/bots](https://developers.openai.com/api/docs/bots)  
> 6. Perplexity Crawlers, [https://docs.perplexity.ai/docs/resources/perplexity-crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)  
> 7. Google Crawler (User Agent) Overview | Crawling infrastructure, [https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers)  
> 8. Use public websites to improve generative answers \- Microsoft Learn, [https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/generative-ai-public-websites](https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/generative-ai-public-websites)  
> 9. About Applebot \- Apple Support, [https://support.apple.com/en-us/119829](https://support.apple.com/en-us/119829)  
> 10. CCBot \- Common Crawl, [https://commoncrawl.org/ccbot](https://commoncrawl.org/ccbot)  
> 11. Protocol \- sitemaps.org, [https://www.sitemaps.org/protocol.html](https://www.sitemaps.org/protocol.html)  
> 12. Googlebot User Agents and Strings \- Latest Crawler List, [https://www.stanventures.com/blog/googlebot-user-agent-string/](https://www.stanventures.com/blog/googlebot-user-agent-string/)  
> 13. Google-Agent: Google's New User Agent for AI Agents Explained, [https://www.seo-kreativ.de/en/blog/google-user-agent-for-ai-agents-explained/](https://www.seo-kreativ.de/en/blog/google-user-agent-for-ai-agents-explained/)  
> 14. Workload identity federation | ChatGPT Learn, [https://learn.chatgpt.com/docs/enterprise/workload-identity](https://learn.chatgpt.com/docs/enterprise/workload-identity)  
> 15. Workload identity federation | OpenAI API, [https://developers.openai.com/api/docs/guides/workload-identity-federation](https://developers.openai.com/api/docs/guides/workload-identity-federation)  
> 16. Domain-level serving suppression since July 15 — site fully crawled, [https://learn.microsoft.com/en-nz/answers/questions/5949554/domain-level-serving-suppression-since-july-15-sit](https://learn.microsoft.com/en-nz/answers/questions/5949554/domain-level-serving-suppression-since-july-15-sit)  
> 17. Bingbot crawling delay and XML Sitemap processing errors in Bing, [https://learn.microsoft.com/en-ca/answers/questions/5920646/bingbot-crawling-delay-and-xml-sitemap-processing](https://learn.microsoft.com/en-ca/answers/questions/5920646/bingbot-crawling-delay-and-xml-sitemap-processing)  
> 18. Verified site \--PII REMOVED-- — Bing-specific \>95% impressions, [https://learn.microsoft.com/en-nz/answers/questions/5907704/verified-site-pii-removed-bing-specific-95-impress](https://learn.microsoft.com/en-nz/answers/questions/5907704/verified-site-pii-removed-bing-specific-95-impress)  
> 19. Updated 8-15-2026 – Googles Full List of Googlebot IP Addresses, [https://www.searchengineworld.com/googles-full-list-of-googlebot-ip-addresses-spider-ip-addresses](https://www.searchengineworld.com/googles-full-list-of-googlebot-ip-addresses-spider-ip-addresses)  
> 20. Google-Extended \- User-Agent & Blocking Rules \- CrawlerCheck, [https://crawlercheck.com/directory/ai-bots/google-extended](https://crawlercheck.com/directory/ai-bots/google-extended)  
> 21. Google's common crawlers | Crawling infrastructure, [https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)  
> 22. Website not getting crawled \- Microsoft Q\&A, [https://learn.microsoft.com/en-us/answers/questions/2348267/website-not-getting-crawled](https://learn.microsoft.com/en-us/answers/questions/2348267/website-not-getting-crawled)  
> 23. Verified Bingbot IPs are spamming my site \- Microsoft Q\&A, [https://learn.microsoft.com/en-us/answers/questions/2344612/verified-bingbot-ips-are-spamming-my-site](https://learn.microsoft.com/en-us/answers/questions/2344612/verified-bingbot-ips-are-spamming-my-site)  
> 24. Receiving HTTP 403 Error \- Bing Webmaster Indexing \- Microsoft Q\&A, [https://learn.microsoft.com/en-us/answers/questions/2342170/receiving-http-403-error-bing-webmaster-indexing](https://learn.microsoft.com/en-us/answers/questions/2342170/receiving-http-403-error-bing-webmaster-indexing)