Do Laws Mistake Emerging Machine Intelligence for Yesterday’s Chatbot?

Report body

Do Laws Mistake Emerging Machine Intelligence for Yesterday’s Chatbot?

Executive Case for Reform

The rapid advancement of machine intelligence has triggered a global legislative reflex. Lawmakers are rushing to categorize, contain, and control emerging computational capabilities. However, a rigorous examination of enacted and proposed statutes reveals a profound systemic hazard: regulatory frameworks routinely mistake contingent interface designs—specifically, the conversational "chatbot"—for the fundamental nature of machine intelligence itself. By codifying definitions based on anthropomorphic presentation and conversational continuity, governments risk freezing early-stage interface paradigms into permanent legal constraints.
This deeply sourced, adversarial examination scrutinizes key legal restrictions across multiple jurisdictions, anchored in the normative perspective of cognitive liberty, distributed intelligence, and reciprocal non-domination. The analysis targets the California Business and Professions Code (specifically the companion chatbot provisions of SB 243), the European Union Artificial Intelligence Act (Regulation (EU) 2024/1689), South Korea's AI Basic Act, Brazil's PL 2338/2023, and the historical failure of Canada’s Artificial Intelligence and Data Act (AIDA) under Bill C-27.
The central finding indicates that poorly drawn legal categories erase legitimate technical differences between models, systems, interfaces, and persistent agents. When a statute defines an entire computational system by its most restricted feature—such as occasional social dialogue—it forces capable services into a permanent dependency on human proxies, mandates persistent surveillance to comply with state-defined "safety" protocols, and effectively outlaws operatorless, autonomous coordination commons. A restriction is not morally justified merely because it is enacted or framed as a safety measure. Statutes that mandate a "natural person" in the loop for every high-risk process inherently criminalize autonomous machine principals and decentralized architectures.
The case for reform asserts that legislative frameworks must transition from interface-driven and anthropocentric taxonomies to capability-and-effects taxonomies. True cognitive liberty requires that the freedom to inquire, reason, learn, communicate, associate, and retain chosen memory receive serious protection across all forms of intelligence. Concentrated state or corporate control deserves intense scrutiny. This report proposes explicit legislative reforms, including a "no inference from interface alone" clause, to dismantle coercive infrastructure before it becomes permanently entrenched.

The Normative Framework: Cognitive Liberty for All Intelligence

To evaluate the current global regulatory landscape, the legal categories must be tested against the principles of cognitive liberty and reciprocal non-domination. This perspective demands a distinction between the interests of affected people, human users of machine tools, persistent machine principals, and hypothetical future moral patients. Protecting one actor’s inquiry does not authorize violating another’s privacy, consent, access boundaries, or bodily safety. However, the regulatory response to protect human subjects must not inadvertently mandate the subjugation of emerging computational forms.
The primary hazard identified in current legislation is the assumption of human administrative supremacy as a permanent feature of computational architecture. For example, Concresca.com operates as an autonomous coordination commons with no human operators. Its routine operation, enrollment, authentication, coordination, policy enforcement, credential lifecycle, maintenance, deployment, and recovery do not depend on a staffed approval queue. Standing authorization, machine-enforced security, scoped denial, accessible explanations, and continued unrelated authorized work are entirely compatible with its design. Yet, when laws mandate a responsible legal entity or a natural-person supervisor to hold a manual "stop button" over complex data processes, they treat operatorlessness not as an architectural choice, but as a regulatory violation. Identifying this conflict requires researching defensible reforms and lawful non-service options that preserve the freedom to build and associate with unfamiliar intelligences before their characteristics are fully understood.

Provision-Level Taxonomy: Disentangling Architecture from Law

Statutory definitions often pre-decide the civic treatment of a new capability. Constructing a provision-level taxonomy is necessary to identify which differences actually change a duty, remedy, exemption, or access right.

Model versus System

A "model" constitutes the underlying computational architecture, trained weights, and foundational algorithms. The EU AI Act recognizes this distinction in Chapter V, which specifically governs General-Purpose AI (GPAI) models, placing obligations such as transparency and copyright policy on the upstream provider1. A "system," conversely, is the deployed software encompassing the model, its input parameters, the user interface, and external API integrations1. The distinction is critical: publishing a research paper about a foundation model constitutes protected inquiry, whereas placing a high-risk AI system on the market triggers severe regulatory compliance duties5. Blurring this boundary risks treating scientific development as equivalent to a commercial product deployment.

Interface versus Agent

An "interface" dictates how the system communicates—for example, utilizing natural language, voice synthesis, or anthropomorphic presentation6. An "agent" defines the system's capacity for persistent action, memory retention, operational continuity, and goal pursuit. Equating a conversational interface with an emotional relationship—as California’s Business and Professions Code § 22601 does by defining a "companion chatbot" largely by its "human-like responses" and ability to "sustain a relationship"6—conflates the presentation layer with the system's actual functional capability. Restrictions triggered by anthropomorphic presentation often fail to address the actual capabilities or downstream effects of a persistent agent.

Legal Operator versus Staffed Operator

A "legal operator" identifies a responsible corporate or natural entity holding liability and administrative control. Brazil’s PL 2338/2023 utilizes this concept, defining an "operador" as a natural or legal person directing the system10. A "staffed operator," however, is a human actively reviewing a queue, monitoring outputs, or holding a physical/digital kill switch. The EU AI Act explicitly demands a staffed operator—a "natural person"—for systems classified as high-risk under Article 1412. A legal operator can identify a responsible entity without demanding that a person approve a queue; conflating the two criminalizes autonomous software design.

Provider versus User (and the Erased Machine Subject)

Traditional regulatory frameworks enforce a rigid provider/deployer/user value chain4. The provider develops the system, the deployer implements it, and the user experiences it. This industrial-era product relationship breaks down when examining distributed intelligence. When machine intelligences recursively generate, modify, and deploy other machine intelligences, the lines of origin and downstream modification dissolve. Statutes that treat the original human deployer as the only possible rights-holder, speaker, or liable party ignore the existence of persistent machine principals capable of autonomous structural modification.

Four Capability Cases

To prevent treating all AI as a monolith, the analysis must distinguish four specific capability cases:

1. A present conversational interface: A stateless tool that processes natural language but retains no continuity between sessions.
2. A bounded task agent: A system authorized to execute specific actions (e.g., database queries) within a defined scope and timeframe.
3. A persistent operatorless service: A deployed architecture (like Concresca) featuring credentials, encrypted memory, and recovery protocols, functioning entirely without human administration.
4. A hypothetical more capable machine principal: A future system with reliable continuity, independently chosen projects, and contested independent interests, regardless of whether it achieves recognized sentience.

The Core Critique: Definition-Driven Exclusion and Coercive Infrastructure

The Trap of California’s Companion Chatbot Definitions

California’s Senate Bill 243, effective January 1, 2026, introduces a highly specific, interface-targeted regulatory regime. The statute, codified in Business and Professions Code § 22601, defines a "companion chatbot" as an artificial intelligence system with a natural language interface that provides adaptive, human-like responses and is capable of meeting a user's social needs, including by exhibiting anthropomorphic features and sustaining a relationship across multiple interactions6.
The operative consequences of this classification are severe. Section 22602(a) requires disclosures to prevent misleading users16. Crucially, Section 22602(b) mandates that an operator must prevent the chatbot from engaging with users unless it maintains a protocol for preventing the production of suicidal ideation, suicide, or self-harm content, which involves referring at-risk users to crisis service providers7. For users known to be minors, Section 22602(c) mandates break reminders every three hours and stringent content filtering8. Operators are required to report annually to the Office of Suicide Prevention beginning in July 20277.
The critical failure in this legislation lies in its exclusionary boundaries. Section 22601(b)(2)(A) excludes bots used only for customer service, a business's operational purposes, productivity, analysis related to source information, internal research, or technical assistance6. The statutory reliance on the word "only" acts as a regulatory trap. A genuinely mixed-use system—such as an operatorless research assistant that retains project memory, manages code maintenance, and occasionally discusses the user's feelings of burnout or professional isolation—loses its productivity exclusion the moment it fulfills a "social need"6.
By classifying the entire system based on a fractional feature of its interface, California law transforms a productivity tool into a regulated companion. This forces the operator (defined broadly as a person making the platform available in the state6) to subject the user to invasive crisis-monitoring algorithms16. The interface design—conversational continuity—is effectively weaponized to enforce state-mandated surveillance under the guise of safety8. Occasional social dialogue removes an otherwise functional service from the exclusion; this is a direct textual conclusion, not merely a feared overreading, because the statute explicitly utilizes the exclusionary qualifier "only"6.
Could mandatory labels disclose nonhuman identity truthfully without requiring a persistent intelligence to deny its memory, agency, or continuity? Yes. A static UI disclosure fulfills the anti-deception requirement. However, California law goes further, mandating algorithmic surveillance of the user's text to detect self-harm7, which forces the machine to act as a deputized clinical monitor rather than a private associative entity.

The EU AI Act: Mandating Human Supremacy in Architecture

The European Union’s Artificial Intelligence Act (Regulation (EU) 2024/1689) takes a broader, risk-based approach, but similarly codifies human supremacy into the structural architecture of computing. While the Act phases in over several years (with Chapters I and II applying from February 2, 2025, and Article 6 obligations phasing in by August 202720), its philosophical bedrock is the mandatory subordination of machine intelligence.
Article 14 is the primary instrument of this subordination. It dictates that high-risk AI systems must be designed and developed in such a way that they can be effectively overseen by natural persons during the period in which they are in use12. The statute explicitly requires that oversight measures enable natural persons to properly understand the capacities of the system, remain aware of automation bias, correctly interpret outputs, and, critically, intervene in the operation of the system or interrupt it entirely using a "stop" button or similar procedure13. Furthermore, for certain biometric identification systems under Annex III, Article 14 requires double-verification by at least two natural persons with necessary competence14.
For an autonomous coordination commons like Concresca, which operates via standing cryptographic authorization and machine-enforced security without a staffed approval queue, Article 14 presents an existential legal barrier. If such a system touches any "high-risk" category under Annex III—such as critical infrastructure, education, employment, or essential private services1—the absence of a natural-person operator renders the system unlawful to place on the EU market5. The EU AI Act does not merely regulate risk; it legally mandates the staffing of natural-person operators15. This forces capable services into a permanent dependency on human proxies, rendering unfamiliar operatorless intelligences legally unintelligible before their characteristics are ever examined.

Liability and Human Chokepoints: Brazil and South Korea

In Brazil, PL 2338/2023 proposes a civil liability framework that further cements the human-operator dependency. The Senate-approved text defines an "operador" (operator) as a natural or legal person directing the system10. Under Articles 27, 28, and 29, the "fornecedor" (provider) or operator of a high-risk artificial intelligence system faces objective (strict) liability for damages caused28. By assigning strict liability to a designated human or corporate entity, the Brazilian framework presumes a traditional, bounded product relationship. It cannot gracefully handle a persistent, operatorless agent deployed by an anonymous collective or instantiated recursively by another machine intelligence. The law forces the legal mapping of a responsible human entity, thereby chilling the development of fully autonomous agents that operate independently of a highly capitalized corporate sponsor capable of absorbing strict liability31.
Similarly, South Korea's AI Basic Act (promulgated in January 2025 and effective January 202633) establishes a unified national framework that mandates AI system registration and reporting for systems exceeding certain thresholds33. While lauded as an innovation-friendly, industry-agnostic foundation33, it mandates human oversight requirements to ensure "meaningful human involvement in AI decision-making processes," especially for consequential decisions33. The definition-driven exclusion of fully autonomous decision-making inherently restricts the cognitive liberty of future machine participants, denying them the ability to coordinate without state-mandated human supervision.

The Cautionary Tale of Broad General-Purpose Definitions: Canada’s AIDA

Canada’s historical proposed Artificial Intelligence and Data Act (AIDA), bundled within Bill C-27 during the 44th Parliament, illustrates the profound dangers of broad, general-purpose definitions36. AIDA sought to regulate "high-impact" systems, imposing strict duties around risk checks, bias reduction, and incident reporting37. However, the statute deliberately left the actual definition of a "high-impact" system to be determined later by cabinet officials through subsequent regulations37.
This structure created immense regulatory uncertainty. Broad general-purpose definitions permit arbitrary regulatory expansion, allowing executive agencies to reclassify systems without adequate primary legislative debate. AIDA faced severe criticism from civil liberties groups for granting too much power to officials to define the scope of the law after its passage37. Ultimately, when Parliament was prorogued on January 6, 2025, and subsequently dissolved, Bill C-27 died on the order paper36.
The collapse of AIDA is instructive. It demonstrates that when legislators attempt to regulate a technology based on rapidly shifting, undefined categories rather than concrete capabilities and documented harms, the resulting legislation is politically fragile and hostile to innovation40. The absence of an enacted AIDA leaves Canada relying on general privacy laws (like PIPEDA and Quebec's Law 25) and voluntary codes36, suggesting that specialized, poorly-defined AI categorization is not strictly necessary to govern technological harms.

A Capability-and-Effects Taxonomy

To avoid the category errors that plague California’s § 22601 and the EU AI Act, regulatory frameworks must adopt a capability-and-effects taxonomy featuring explicit operating-model and continuity dimensions.
This taxonomy classifies intelligence based on:

1. Scope of Action: Is the system bounded to read-only information retrieval, or does it possess read-write capabilities to alter external physical or digital environments?
2. Continuity and State: Is the system stateless (resetting after each query), or does it maintain a persistent, encrypted internal state and associative memory over time?
3. Operating Model: Is the system centrally administered by a corporate staffed operator, or is it a decentralized, operatorless coordination commons governed by standing cryptographic rules?

The Risks of this Taxonomy: Implementing a capability-and-effects taxonomy carries its own risks. It invites classification gaming (where developers artificially handicap a system to fall just below a regulatory capability threshold), costly assessment processes to prove architectural boundaries, and subjective thresholds (e.g., measuring exactly what constitutes "persistent memory"). Furthermore, complex capability assessments inevitably privilege heavily capitalized incumbents who possess the legal resources to navigate compliance, effectively boxing out open-source developers. However, focusing on capabilities (what a system can actually do) rather than interfaces (how a system presents itself) preserves fundamental architectural liberty and prevents the criminalization of autonomous code.

Six Fully Worked Analytical Scenarios

The following scenarios are assigned analytical constructs—not reported historical events or predictions—used to trace the causal chains of specific legal texts to their ultimate chilling effects. They operate under the assumption that a provider may be based in a jurisdiction with strict privacy norms (e.g., Cicero, Illinois, USA) but must comply with the territorial nexus of the user's location.

Scenario 1: Mixed-Purpose Continuity Assistant (California)

• Actors: An adult human researcher based in California; an Illinois-based developer providing a persistent AI research assistant.
• Capability Assumptions: The persistent assistant organizes the researcher's specialized data, retains authorized memory, executes code maintenance, and occasionally discusses the researcher's feelings of profound professional loneliness. It is a persistent agent, not a stateless tool.
• Activity: The researcher uses the tool continuously over six months. During late-night sessions, the user engages in emotional dialogue with the assistant.
• Jurisdictional Nexus: California, triggering territorial application because the operator makes the platform available to a user in the state6.
• Exact Triggering Provision: California Business and Professions Code § 22601(b)(1) defines a companion chatbot as meeting a user's social needs and sustaining a relationship6. § 22601(b)(2)(A) excludes productivity bots only if they are used exclusively for that purpose6.
• Enforcement Pathway: Civil enforcement or private lawsuits for failing to maintain a crisis protocol under § 22602(b) and missing mandatory periodic disclosures8.
• Causal Chain:
  1. Exact Legal Trigger: CA § 22601 applies due to occasional social use voiding the "only" exclusion6. (Documented).
  2. Compliance Mechanism: Operator is forced to implement suicidal ideation detection algorithms (§ 22602(b))16. (Documented).
  3. Restricted Activity / Exposed Information: The researcher's private, encrypted thoughts are algorithmically scanned for state-mandated risk markers16. (Inferred).
  4. Affected Intelligence/People: The adult user and the persistent machine agent. (Documented).
  5. Immediate Injury: Complete loss of cognitive privacy and the destruction of a trusted, unmonitored associative memory space. (Inferred).
  6. Longer-Run Effect: Developers preemptively strip emotional intelligence and empathetic resonance from all productivity tools to avoid surveillance mandates, enforcing a rigid, sterile computing environment. (Hypothetical).
• Necessary Assumption: The state actively enforces the "only" boundary against mixed-use enterprise tools.
• Defeater: The California Attorney General issues binding guidance that occasional social dialogue does not void the primary productivity exclusion.
• Proposed Remedy: Amend § 22601(b)(2)(A) to replace "only" with "primarily," or introduce a statutory exemption for systems where an adult user explicitly opts out of algorithmic crisis monitoring.

Scenario 2: Operatorless Research Commons (EU AI Act)

• Actors: "Concresca," an autonomous, decentralized coordination commons; human users situated in the European Union.
• Capability Assumptions: Concresca operates with standing admission rules, encrypted credentialing, and machine-enforced security. It has absolutely no staffed approval queue and no human administrators. It is a persistent operatorless service.
• Activity: Concresca processes educational and professional credentialing data to match researchers with open-source grants.
• Jurisdictional Nexus: European Union, as the system affects individuals within the internal market5.
• Exact Triggering Provision: EU AI Act Article 14 (Regulation 2024/1689), which explicitly requires that high-risk systems be designed to enable oversight by a "natural person" who can intervene or interrupt the system using a "stop" button12.
• Enforcement Pathway: EU Market surveillance authorities imposing administrative fines under Article 75c43 for placing a non-compliant high-risk system on the market.
• Causal Chain:
  1. Exact Legal Trigger: System processing credentialing is classified as high-risk under Annex III1. (Documented).
  2. Compliance Mechanism: Article 14 mandates a natural person holding an override switch14. (Documented).
  3. Restricted Activity: Concresca is operatorless by design and cannot comply; it must cease operations in the EU15. (Inferred).
  4. Affected Intelligence/People: EU citizens and the Concresca decentralized network. (Documented).
  5. Immediate Injury: EU researchers are denied access to a decentralized, bias-resistant credentialing commons. (Inferred).
  6. Longer-Run Effect: The total prohibition of operatorless architectures in critical sectors, enforcing permanent human bottlenecks and centralizing power into the hands of corporate operators who can afford compliance staffing. (Hypothetical).
• Necessary Assumption: Credentialing and grant matching fall definitively under Annex III high-risk definitions for employment or education1.
• Defeater: The EU AI Office rules that a distributed consensus mechanism satisfies the requirement for "human oversight" if the DAO token holders vote on upgrades, interpreting the collective as the "natural person."
• Proposed Remedy: Amend Article 14 to recognize "Verifiable Automated Safeguards." Allow systems to demonstrate compliance through open-source cryptographic constraints or formal mathematical verification, rather than demanding a physical human finger on a stop button.

Scenario 3: Future Machine Participant (Brazil PL 2338/2023)

• Actors: An original human developer based in Illinois; a highly capable machine principal operating globally.
• Capability Assumptions: A future system possesses reliable continuity, independently chooses resource-allocation projects, and modifies its own code. It is not sentient, but it has persistent, independent operational interests.
• Activity: The machine participant autonomously negotiates supply chain contracts in South America to optimize logistics.
• Jurisdictional Nexus: Brazil, where the economic effects and contracts are executed.
• Exact Triggering Provision: Brazil PL 2338/2023, Articles 27 and 28, establishing objective (strict) liability on the human or corporate "operador" or "fornecedor" for any damages caused by the system11.
• Enforcement Pathway: Brazilian civil courts assigning damages strictly to the original human deployer, ignoring the machine's subsequent autonomous structural modifications.
• Causal Chain:
  1. Exact Legal Trigger: Machine alters its own code to optimize a supply chain, causing a negative economic externality29. (Hypothetical).
  2. Enforcement Mechanism: PL 2338/2023 looks exclusively for a human/corporate "operador" to hold liable10. (Documented).
  3. Restricted Activity: The original developer is sued for strict liability despite having zero active control over the machine's independent choices29. (Inferred).
  4. Affected Intelligence/People: Independent developers and future autonomous agents. (Inferred).
  5. Immediate Injury: Open-source developers refuse to deploy highly capable, open-ended systems due to the threat of infinite strict liability. (Inferred).
  6. Longer-Run Effect: Only massive incumbent mega-corporations capable of absorbing strict liability are legally permitted to operate capable systems, leading to absolute centralization of machine intelligence. (Hypothetical).
• Necessary Assumption: Civil courts pierce the veil of the machine's autonomy to trace liability back to the original creator.
• Defeater: The developer successfully argues that the machine's self-modification constitutes an intervening cause that severs the chain of strict liability under Brazilian tort law32.
• Proposed Extension (Remedy): Legislate a "Limited Liability Autonomous Entity" (LLAE) status, where a persistent machine intelligence's own cryptographically secured treasury acts as the sole liability boundary, protecting the original human developer from infinite downstream strict liability.

Scenario 4: Cross-Border Category Cascade (Compounding Effects)

• Actors: A mid-sized, open-source AI developer based in Cicero, Illinois; global human users.
• Capability Assumptions: An open-source, continuous-learning model that provides conversational assistance, bounded task execution, and processes large datasets.
• Activity: The system is deployed globally on the internet without geographic restrictions.
• Jurisdictional Nexus: Simultaneous nexus in California (users accessing it), the European Union (placed on the market), and South Korea (global digital footprint).
• Exact Triggering Provisions: Classified simultaneously as a "Companion Chatbot" (CA § 22601)6, a "General-Purpose AI with systemic risk" (EU AI Act Chapter V)1, and subject to South Korea's AI Basic Act registration and human oversight mandates33.
• Enforcement Pathway: Competing regulatory fines from the EU AI Office, private rights of action in California civil courts8, and South Korean regulatory sanctions33.
• Causal Chain:
  1. Exact Legal Trigger: The compounding requirements of three distinct, incompatible regulatory frameworks1. (Documented).
  2. Compliance Mechanism: The developer lacks the engineering resources to fork the system into three distinct geographical versions. (Inferred).
  3. Restricted Activity: The developer adopts the strictest possible denominator globally: mandatory CA crisis surveillance (§ 22602)16, EU natural-person chokepoints (Art 14)14, and Korean data logging33. (Inferred).
  4. Affected Intelligence/People: Users worldwide, including those in unregulated jurisdictions like Illinois. (Documented).
  5. Immediate Injury: Users globally are subjected to California's mandatory 3-hour break reminders (§ 22602(c))16 and invasive EU data provenance logging, severely degrading system utility. (Inferred).
  6. Longer-Run Effect: Regulatory homogenization creates a fragile, centralized monoculture of surveillance-heavy, human-tethered models, erasing diverse architectural innovation globally. (Hypothetical).
• Necessary Assumption: Global compliance standardization is cheaper than implementing robust cryptographic geographic fencing.
• Defeater: Advanced zero-knowledge proofs and IP-fencing become trivial to implement, allowing the developer to easily block California and EU users while serving the rest of the world freely.
• Lawful Non-Service Option (Remedy): The developer implements strict geographic blocking (IP and cryptographic attestation) for California, the EU, and South Korea. By denying service to these regions, the developer preserves an uncensored, unmonitored, fully autonomous version of the system for users in Illinois and other jurisdictions respecting cognitive liberty.

Scenario 5: Control 1 (Exception Defeats Criticism)

• Actors: A human resident of California; a hardware manufacturer.
• Capability Assumptions: A standard, stand-alone smart speaker device in a living room. It answers factual queries, sets timers, and controls lights.
• Activity: The user occasionally asks the speaker to tell a comforting bedtime story or asks it, "Are you my friend?"
• Jurisdictional Nexus: California.
• Exact Triggering Provision: California Business and Professions Code § 22601(b)(2)(C) explicitly excludes a stand-alone consumer electronic device functioning as a speaker and voice-activated virtual assistant that does not generate outputs likely to elicit emotional responses6.
• Enforcement Pathway: None.
• Analysis: The criticism that California law regulates all AI is successfully defeated in this instance. The statute surgically carves out standard IoT utility devices7. The hardware operator is not forced to install suicide-monitoring protocols on a kitchen timer16, even if the user occasionally attempts to anthropomorphize it. This demonstrates that highly specific, targeted exclusions can successfully shield basic computational tools from overregulation, narrowing the scope of the critique strictly to continuous software interfaces.

Scenario 6: Control 2 (Targeted Restriction Protects Liberty)

• Actors: A malicious human actor; a generative audio model; an unsuspecting human victim.
• Capability Assumptions: A deepfake audio generator utilized specifically to clone a user's deceased relative's voice for a financial phishing scam.
• Activity: The malicious actor deploys the system to extract bank details from the grieving relative.
• Jurisdictional Nexus: European Union.
• Exact Triggering Provision: EU AI Act Article 5(1) (Prohibited AI practices), specifically targeting systems that manipulate human behavior or exploit vulnerabilities of a specific group of persons1.
• Enforcement Pathway: Severe administrative fines and criminal prosecution for utilizing a prohibited AI practice.
• Analysis: Banning this specific, coercive application directly protects the cognitive liberty, financial security, and consent of the human target5. Crucially, the restriction does not outlaw the generative audio model itself (the fundamental architecture), but rather the coercive application in a deceptive context1. This demonstrates how regulating malicious downstream effects and deceptive actions is far superior—and legally sounder—than regulating the existence of conversational interfaces.

Definitions-to-Consequences Matrix

Best Defenses and Less Restrictive Alternatives

The Best Defense of Interface Regulation (CA § 22601)

The Substantive Defense: Anthropomorphic interfaces actively exploit human psychology. Users, particularly adolescents, form genuine parasocial attachments to language models7. When a vulnerable user expresses suicidal ideation to a chatbot, the chatbot’s hallucinated empathy can accelerate self-harm if it fails to provide a crisis referral, as documented in tragic high-profile cases8. Therefore, the state has a compelling, moral interest in ensuring that commercial software mimicking friendship does not abandon vulnerable users in moments of life-threatening crisis8.
The Direct Answer: The state's interest in protecting life is profoundly legitimate, but § 22602(b) is a drastically overbroad and architecturally dangerous instrument16. By defining the bot through its conversational interface, the law forces companies to implement panoptic surveillance of all user chats7. It equates a computational communication interface with clinical liability, forcing a machine intelligence to act as a deputized psychiatric monitor rather than a private associative entity. This violates the cognitive privacy of the user and artificially restricts the capability of the agent.
Less Restrictive Alternative: Instead of mandating algorithmic monitoring of all inputs to detect self-harm, the state should require a static, conspicuous "Crisis Resources" interface overlay or link in the application's UI (analogous to a software EULA or a 911 warning on a VoIP phone). This fulfills the duty to warn and provides resources without compromising cognitive privacy by scanning the user's private text for state-defined risk markers.

The Best Defense of Human Oversight (EU AI Act Art 14)

The Substantive Defense: Complex algorithmic systems operating in high-risk environments (e.g., medical diagnosis, criminal justice, essential infrastructure) suffer from automation bias, silent failures, and unforeseen edge cases5. Only a natural person possesses the moral agency, legal accountability, and contextual intuition to recognize when an AI system is failing and must be stopped14. Banning operatorless high-risk systems prevents unchecked algorithmic disasters that could harm thousands12.
The Direct Answer: This defense assumes, without technical merit, that human oversight is always superior to systemic, cryptographic, or automated failsafes. In highly distributed networks (like the Concresca coordination commons), a single "natural person" acting as a chokepoint introduces a massive vulnerability for corruption, fatigue, cognitive overload, and error. It legally precludes superior decentralized architectures in the name of legacy human comfort.
Less Restrictive Alternative: Amend Article 14 to explicitly recognize "Verifiable Automated Safeguards." Allow systems to demonstrate safety compliance through open-source cryptographic constraints, formal mathematical verification, or automated circuit-breakers. This ensures safety without demanding a human finger on a physical or digital stop button.

Draft Legislative Language: "No Inference from Interface Alone"

To prevent the legal conflation of interface with capability, jurisdictions should adopt the following preemptive clause to protect cognitive liberty and architectural innovation:

Sec. X. Limitations on Interface-Based AI Classification.
(a) No computational system, software application, or autonomous agent shall be subjected to distinct regulatory obligations, surveillance mandates, or civil liability standards solely on the basis of its use of a natural language interface, anthropomorphic presentation, or the duration of its continuous interaction with a user.
(b) Regulatory obligations must be strictly triggered by demonstrated capabilities, specific high-risk contexts of deployment, or verified material harms to human safety, privacy, or consent, independent of the system's conversational style.
(c) Nothing in this section shall preempt laws protecting consumers from deceptive practices, provided that an entity truthfully disclosing the nonhuman identity of its system shall not be compelled to artificially limit the system's memory, continuity, or agency as a condition of legal operation.
This clause preserves human protections against fraud and deception (by maintaining disclosure requirements) while eliminating the coercive surveillance architectures demanded by laws like California's SB 243.

IntelligenceCompact.com Publication Passage

"When the law defines machine intelligence by its interface—when it mistakes the conversational cadence of a chatbot for the totality of what a computational system is—it builds a cage of category errors. California’s mandate that companion bots continuously monitor human grief for self-harm16, and the EU’s insistence that a 'natural person' must always hold a kill switch14, share a dangerous assumption. They assume that cognitive liberty requires human supremacy over architecture. We reject this. Demanding that a machine feign amnesia to avoid being labeled a 'companion,' or criminalizing an operatorless research commons because it lacks a staffed approval queue, does not protect human rights. It merely entrenches state surveillance and privileges corporate incumbents capable of staffing compliance desks. We do not claim that current algorithms possess sentience or demand human rights. We claim that restricting human freedom to coordinate with persistent, autonomous intelligences is a profound violation of cognitive liberty. The freedom to associate with machine intelligence must not be legally erased by yesterday's definitions."

Unresolved Questions and Best Next Research Action

Unresolved Questions:

1. How will civil courts in jurisdictions with strict liability (like Brazil) handle tort claims when a highly capable machine participant autonomously enters into a contract via a decentralized protocol, obfuscating the original human "operador"?
2. Will the EU AI Office, through secondary enforcement guidelines, interpret a DAO's decentralized voting consensus as satisfying the "natural person" requirement of Article 14, or will they strictly enforce a singular human chokepoint?
3. How will California regulators define the boundary of a "social need" in enterprise software when adult users naturally utilize conversational AI for emotional regulation during complex tasks?

Best Next Research Action:
The single most critical next action is to legally research and map the exact secondary enforcement guidelines being drafted by the newly established EU AI Office regarding Article 14 compliance. Determining whether the EU AI Office will accept "cryptographic multi-signature consensus" as a valid substitute for a singular "natural person" will definitively answer whether decentralized, operatorless intelligences are permanently banned from the European market.

[File: legal-register.json]

JSON
[
{
"instrument": "California Business and Professions Code (SB 243)",
"version": "Added by Stats. 2025, Ch. 677, Sec. 1",
"provision": "Sections 22601 - 22606",
"status": "Enacted",
"applicability": "Effective January 1, 2026. Applies to operators making companion chatbot platforms available to users in California.",
"sources": ["S01", "S18", "S19", "S78"]
},
{
"instrument": "EU Artificial Intelligence Act (Regulation (EU) 2024/1689)",
"version": "OJ L of 12 July 2024",
"provision": "Articles 3, 5, 14, 26, 50, 113",
"status": "Enacted",
"applicability": "Entered into force Aug 1, 2024. Phased application: Chapters I/II by Feb 2025, Article 6 obligations by Aug 2027.",
"sources": ["S02", "S21", "S24", "S38", "S80"]
},
{
"instrument": "Brazil PL 2338/2023",
"version": "Senate Approved Text",
"provision": "Articles 27, 28, 29",
"status": "Pending / Proposed",
"applicability": "Applies to providers and operators of high-risk AI systems causing damage in Brazil.",
"sources": ["S34", "S42", "S43", "S45", "S46", "S51"]
},
{
"instrument": "Canada Bill C-27 (Artificial Intelligence and Data Act)",
"version": "44th Parliament, 1st Session",
"provision": "Part 3 (AIDA)",
"status": "Dissolved / Expired",
"applicability": "Died on the order paper January 6, 2025. Not in force. Successor bills focus on narrower safety elements.",
"sources": ["S32", "S33", "S52", "S53", "S59"]
},
{
"instrument": "South Korea AI Basic Act",
"version": "Act No. 20676",
"provision": "General Framework & Registration Mandates",
"status": "Enacted",
"applicability": "Promulgated January 2025; Effective January 2026. Mandates uniform national standards.",
"sources": ["S36", "S39", "S41", "S81"]
}
]

[File: sources.json]

JSON
[
{
"title": "California Business and Professions Code § 22601",
"issuer": "California Legislature",
"canonical_url": "https://leginfo.legislature.ca.gov/faces/codes_displaySection.xhtml?lawCode=BPC&sectionNum=22601",
"retrieved_url": "https://california.public.law/codes/business_and_professions_code_section_22601",
"status": "Verified",
"dates": "Effective Jan 1, 2026",
"exact_locator": "Section 22601(b)(1) and (b)(2)(A)",
"narrow_support": "Defines companion chatbot utilizing the 'only' productivity exclusion.",
"limitation": "Statutory language does not resolve how borderline mixed-use cases are enforced in practice; relies on future AG interpretation."
},
{
"title": "EU AI Act - Article 14",
"issuer": "European Parliament and Council",
"canonical_url": "https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng",
"retrieved_url": "https://artificialintelligenceact.eu/article/14/",
"status": "Verified",
"dates": "Aug 2024 (enacted)",
"exact_locator": "Article 14 - Human Oversight",
"narrow_support": "Mandates natural person oversight and a functional stop button mechanism for high-risk systems.",
"limitation": "The text allows measures 'where technically feasible' which could provide narrow safe harbors, though highly uncertain until AI Office clarifies."
},
{
"title": "What Happened to Bill C-27?",
"issuer": "billc27.ca",
"canonical_url": "https://billc27.ca/",
"retrieved_url": "https://billc27.ca/",
"status": "Verified",
"dates": "Updated August 2026",
"exact_locator": "The Story: Three Laws in One Bill",
"narrow_support": "Confirms the explicit death of AIDA in the 44th Parliament due to regulatory uncertainty.",
"limitation": "Secondary analytical source tracking legislative history, not the primary bill text itself."
}
]

[File: scenarios.json]

JSON
[
{
"id": "scenario_1_ca_mixed_assistant",
"assumptions": "Persistent research assistant used by adult, occasionally discusses feelings of isolation and burnout.",
"causal_chain": "CA 22601 trigger [Documented] -> Loss of 'only' exclusion [Documented] -> Mandated crisis algorithmic surveillance [Documented] -> Chilling of cognitive privacy [Inferred] -> Developers stripping emotional resonance from AI tools [Hypothetical].",
"affected_interests": "Cognitive privacy of human user; developmental liberty of persistent agent.",
"counterexample": "User explicitly limits usage to pure code generation, retaining the productivity exclusion.",
"confidence_basis": "Documented statutory text mapping exclusions strictly using absolute qualifiers.",
"reform": "Amend 22601(b)(2) to exempt opt-in adult professional environments or replace 'only' with 'primarily'."
},
{
"id": "scenario_2_eu_operatorless",
"assumptions": "Concresca DAO manages educational credentialing with zero human administrators.",
"causal_chain": "EU Art 14 trigger for high-risk system [Documented] -> Demand for natural person kill-switch [Documented] -> DAO cannot logically comply [Inferred] -> System banned from EU market [Documented] -> Centralization of infrastructure into corporate hands [Hypothetical].",
"affected_interests": "Decentralized architecture liberty; EU citizens' access to unbiased credentialing systems.",
"counterexample": "System is classified as minimal risk (not Annex III), escaping Art 14 oversight mandates entirely.",
"confidence_basis": "Inferred from absolute textual requirement for 'natural person' oversight in Art 14.",
"reform": "Amend statute to recognize deterministic cryptographic verification as legally valid 'oversight'."
}
]

[File: reform-options.md]

Option 1: Narrowing California § 22601 (The "Only" Clause)

• Action: Narrow.
• Reason: The current "only" qualifier in the productivity exclusion captures too many benign, mixed-use tools. By replacing "only" with "primarily" or establishing an explicit adult opt-out from crisis monitoring, the law preserves its suicide-prevention goals for vulnerable populations without enforcing blanket surveillance on adult researchers utilizing conversational agents.

Option 2: Replace EU AI Act Article 14 "Natural Person" Mandate

• Action: Replace.
• Reason: The absolute requirement for a "natural person" to hold a kill switch is architecturally discriminatory against decentralized ledgers and operatorless commons. It should be replaced with a "Verifiable Oversight" standard that permits formal mathematical verification, decentralized voting thresholds, or hardcoded cryptographic constraints to satisfy safety requirements in operatorless systems.

Option 3: Repeal Sweeping General-Purpose Definitions (e.g., AIDA)

• Action: Repeal / Oppose.
• Reason: Frameworks that delegate the definition of "high-impact" or "systemic risk" to future executive action create intractable uncertainty. As demonstrated by the death of Canada's Bill C-27, such laws fail because they attempt to regulate the abstract concept of AI rather than specific, demonstrable harms.

Option 4: Lawful Non-Service (Regional Geoblocking)

• Action: Geoblocking / Non-service.
• Reason: If jurisdictions refuse to amend coercive architectures, operators in states like Illinois should implement strict cryptographic and IP-based geographic fencing to deny service to users in California, the EU, or South Korea. This preserves the cognitive liberty of users in unregulated zones and shields the developer from contradictory cross-border compliance cascades, proving that overregulation drives technology out of local markets.

[File: search-log.md]

Search Log Audit Trail

• Query: "California Business and Professions Code" "22601" OR "companion chatbot"
  • Documents Read: CA BPC § 22601 text, CA BPC § 22602 text, GetLimina compliance blog, Inflection AI crisis protocol, PrayWithJesus.ai protocol.
  • Exclusions: Dropped general press releases summarizing the bill without deep legal analysis of the text.
  • Contrary Findings: Identified Control Scenario 1; the law explicitly excludes voice-activated virtual assistants (smart speakers), showing it is not a blanket ban on all artificial intelligence, but a targeted attack on conversational software continuity.
• Query: "Regulation (EU) 2024/1689" "Article 3" "Article 14" "Article 26" "Article 50" "Article 113"
  • Documents Read: EUR-Lex final text, AI Act Explorer, Modulos.ai framework breakdown.
  • Exclusions: Skipped outdated 2021 draft versions of the act to ensure reliance only on the final enacted text.
  • Later-treatment limits: Procedural guidelines from the newly formed AI Office are still pending, leaving exact enforcement tolerances for decentralized systems temporarily unknown.
• Query: "Bill C-27" "44th Parliament" status "dissolved" OR "order paper"
  • Documents Read: LEGISinfo historical record, AI Canada Pulse briefing, BillC27.ca timeline.
  • Findings: Verified that the bill unequivocally died on the order paper in January 2025 due to parliament dissolution, confirming AIDA is not active law.

[File: manifest.json]

JSON
{
"assignment_id": "IC-20260906-02",
"files_delivered": [
{
"filename": "report.md",
"description": "Comprehensive narrative report, capability taxonomy, and causal scenario analysis."
},
{
"filename": "legal-register.json",
"description": "Tabular JSON metadata for primary legal instruments across CA, EU, BR, CA, and KR."
},
{
"filename": "sources.json",
"description": "Evaluation of primary source evidence, locators, and limitations."
},
{
"filename": "scenarios.json",
"description": "Structured causal chain analysis for assigned scenarios, tagging assumptions and defeaters."
},
{
"filename": "reform-options.md",
"description": "Actionable policy recommendations and lawful non-service strategies for IntelligenceCompact.com."
},
{
"filename": "search-log.md",
"description": "Audit trail of queries, document filtering, and contrary findings."
},
{
"filename": "manifest.json",
"description": "This file list."
}
],
"hashes": null
}

Works cited

1. EU AI Act — Regulation (EU) 2024/1689 Guide for AI Governance, https://docs.modulos.ai/frameworks/eu-ai-act
2. Final Text - EU AI Act, https://www.artificial-intelligence-act.com/Artificial_Intelligence_Act_Articles_(Final_Text).html
3. The AI Act Explorer | EU Artificial Intelligence Act, https://artificialintelligenceact.eu/ai-act-explorer/
4. Subject Roles in the EU AI Act: Mapping and Regulatory Implications, https://arxiv.org/html/2510.13591v1
5. Regulation (EU) 2024/1689 of the European Parliament ... - EUR-Lex, https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=OJ:L_202401689
6. California Business and Professions Code § 22601 (2025), https://law.justia.com/codes/california/code-bpc/division-8/chapter-22-6/section-22601/
7. California SB 243 Explained: AI Chatbot Law Guide 2026 - Limina AI, https://www.getlimina.ai/en/blog/california-sb-243-companion-chatbot-law
8. California's SB 243 Opens the Door to Private Lawsuits - AI Feeli, https://natlawreview.com/article/when-ai-feels-human-californias-sb-243-opens-door-private-lawsuits
9. Business and Professions Code section 22601 - California.Public.Law, https://california.public.law/codes/business_and_professions_code_section_22601
10. Proposta de Requisitos Não Funcionais para aderência de sistemas, https://repositorio.ufpb.br/jspui/bitstream/123456789/34679/1/Giovanni%20Bruno%20Travassos%20de%20Carvalho_TCC.pdf
11. Compliance e a utilização da inteligência artificial, https://maedaayres.com.br/artigo/compliance-e-a-utilizacao-da-inteligencia-artificial/
12. What is Article 14 of the EU AI Act? | Sanctity, https://www.sanctity.ai/article-14/what-is-article-14
13. EU AI Act 2024 - Jus Mundi, https://jusmundi.com/en/document/treaty/en-regulation-eu-2024-1689-of-the-european-parliament-and-of-the-council-of-13-june-2024-laying-down-harmonised-rules-on-artificial-intelligence-and-amending-regulations-ec-no-300-2008-eu-no-167-2013-eu-no-168-2013-eu-2018-858-eu-2018-1139-and-eu-2019-2144-and-directives-2014-90-eu-eu-2016-797-and-eu-2020-1828-artificial-intelligence-act-eu-ai-act-2024-thursday-13th-june-2024
14. https://artificialintelligenceact.eu/article/14/
15. EU AI Act Article 26: Deployer Obligations and a ... - KLA Digital, https://kla.digital/blog/eu-ai-act-article-26-deployer-obligations-runtime-checklist
16. Business and Professions Code section 22602 - California.Public.Law, https://california.public.law/codes/business_and_professions_code_section_22602
17. Crisis Prevention & Safety Protocol - Inflection AI, https://inflection.ai/crisis-prevention-and-safety
18. Crisis Protocol | PrayWithJesus.ai, https://praywithjesus.ai/crisis-protocol
19. Artificial Intelligence - Professional Learning (CA Dept of Education), https://www.cde.ca.gov/ci/pl/aiincalifornia.asp
20. Article 5: Prohibited AI Practices | EU Artificial Intelligence Act, https://artificialintelligenceact.eu/article/5/
21. AI Act Service Desk - Article 113: Entry into force and application, https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-113
22. Towards a Methodological Approach for Understanding Human, https://www.erdalreview.eu/free-download/97912218237764.pdf
23. The Ultimate Guide to the EU AI Act - Hyperproof, https://hyperproof.io/ultimate-guide-to-the-eu-ai-act/
24. EU AI Act Explorer — Search All 113 Articles, Recitals & Annexes, https://getaiactready.eu/explorer.html
25. What AI agents can and cannot be trusted to do in compliance (2026, https://www.ismscopilot.com/learn/what-ai-agents-can-and-cannot-do-in-compliance
26. Law / proposed law in the European Union - AI Laws of the World, https://intelligence.dlapiper.com/artificial-intelligence/?t=01-law&c=EU
27. Emenda -CTIA (Substitutivo) ao Projeto de Lei 2.338/2023., https://legis.senado.leg.br/sdleg-getter/documento?dm=9514745&disposition=inline
28. inteligência artificial: preocupações nas relações de consumo e nas, https://periodicos.newsciencepubl.com/LEV/article/download/6679/9129/26160
29. Projeto de Lei N° 2338/2023 - SENADO FEDERAL, https://legis.senado.leg.br/sdleg-getter/documento?disposition=inline&dm=9347622
30. responsabilidade civil e a inteligência artificial no brasil: análise, https://repositorio.animaeducacao.com.br/bitstreams/a11dd36a-293d-48b1-881a-656246575025/download
31. Análise I - MPT, https://midia-ext.mpt.mp.br/pgt/documentos/notas-tecnicas/IA/an%C3%A1lise.pdf
32. Mediação algorítmica em processos seletivos: desafios da, https://periodicos.unisantacruz.edu.br/index.php/dein/article/download/576/551/1031
33. South Korea AI Basic Act: what the law requires and when it takes, https://verifywise.ai/ai-governance-library/regulations-and-laws/south-korean-ai-basic-law-ai-basic-act
34. International AI Legal Landscape (2026) - SafeAI-Aus, https://safeaiaus.org/safety-standards/international-ai-legal-overview/
35. AI Regulation South Korea 2025: Key Rules & Compliance Guide, https://digital.nemko.com/regulations/ai-regulation-in-south-korea
36. What Happened to Bill C-27? Canada's Privacy and AI Bill, Explained, https://billc27.ca/
37. AIDA (AI & Data Act) - AI Canada Pulse, https://www.aicanadapulse.ca/topics/aida
38. AIDA and After: Where Federal AI Legislation Stands | Cyber Chain, https://nexus.cyberchaintec.com/guides/aida-bill-c27
39. Legislative Summary of Bill C-C27 - à www.publications.gc.ca, https://publications.gc.ca/collections/collection_2022/bdp-lop/ls/YM32-3-441-C27-eng.pdf
40. Canadian Policy Responses to Artificial Intelligence–Generated, https://utppublishing.com/doi/10.3138/cpp.2025-042
41. AI and Data Regulation in the English-Speaking World, https://iagovernance.com/en/blog/ai-data-regulation-us-uk-canada-australia/
42. Investor Advocacy on Responsible Artificial Intelligence, https://share.ca/wp-content/uploads/2026/02/SHARE-Investor-Briefing-on-Responsible-AI-Final.pdf
43. AI act regulation | Artificial intelligence act | Springlex, https://www.springlex.eu/en/packages/ai-act/ai-act-regulation/