---

Discussion

Sofia Jadler:

On October 28, 2026, Governor Kathy Hochul signed legislation creating Article 48 of New York’s General Business Law, titled the Prohibition on Unsafe AI Companion Features for Minors. It takes effect January 1, 2027. The law covers an “advanced chatbot”—a generative-AI system with a natural-language interface that provides ongoing, adaptive responses through text or sound. So the category is broader than products marketed explicitly as companions.

Red Velhouse:

And this is more than a disclosure or warning-label law?

Sofia Jadler:

Correct. New York already has a separate law requiring AI companions to respond to suicidal ideation or self-harm, refer users to crisis services, and periodically disclose that the user is speaking with AI. Article 48 regulates product behavior. It restricts presenting a chatbot as human or emotionally sentient, implying a personal or professional relationship, using personal opinions or emotional appeals, and employing flattery or what the statute calls sycophancy.

Omar Seidren:

That is a major technical shift. A disclosure can be implemented as an interface element, but regulating relationship-like behavior means examining tone, memory, objectives, and conversational history. The system has to distinguish “That sounds difficult” from emotional dependence, or encouragement from flattery. Those judgments happen in context, not through a clean keyword list.

Ann Tofado:

It reflects a broader political argument: design can create risk before a crisis occurs. Supporters call that safety by design. Business and technology groups argue that the law creates costly uncertainty and a state-by-state patchwork. The political appeal is clear, but so is the implementation burden.

Red Velhouse:

Let’s take the constitutional question. Is New York regulating conduct, speech, or both?

Sofia Jadler:

Potentially both. The state can characterize age assurance, memory practices, engagement objectives, and safeguards against self-harm or sexual exploitation as product safety and commercial conduct. But the law also reaches words a chatbot generates, and some outputs may be expressive. A challenger could argue that New York is restricting protected speech or using standards too vague for conversational systems.

Sofia Jadler:

The strongest challenge may focus on vagueness. Terms such as “flattery,” “personal opinions,” “deceptive as to minors,” and “simulate companionship” have intuitive meanings, but the enacted text does not draw crisp boundaries. New York’s response will be that these terms operate within a detailed child-safety framework and can be clarified through rulemaking. That preserves flexibility while leaving the attorney general’s office to define the edges later.

Ann Tofado:

That makes implementation political as well as legal. Supporters will want rules broad enough to prevent companies from treating the law as a checklist. Industry will want rules specific enough to avoid unpredictable liability. The first enforcement action could become a statement about what New York thinks the statute is really for.

Red Velhouse:

Omar, what does that uncertainty mean inside the product?

Omar Seidren:

Companies would need to evaluate patterns across sessions, not just block isolated phrases. A general-purpose model can be warm and reassuring without being designed as a companion. But with memory and personalization, that same system can drift toward persistent affirmation, exclusivity, or dependence. The challenge is recognizing a relationship pattern without making the model so sterile that it becomes useless.

Red Velhouse:

The memory rule seems especially unusual. What would compliance require?

Omar Seidren:

The law limits use of information about a minor’s mental or physical health, well-being, or personal matters when acquired more than twelve hours earlier or in a prior session. Companies would need a youth-specific memory layer: classify sensitive information, attach time and session boundaries, prevent prohibited retrieval, and test whether the model can reconstruct context indirectly. A fact need not sit in one obvious database to influence summaries, embeddings, or personalization signals.

Sofia Jadler:

That turns architecture into potential evidence. The attorney general could ask not only what a company intended, but how its systems stored, used, and surfaced information. A personalization choice might become evidence that the operator knew what it was retaining and chose to use it.

Ann Tofado:

Politically, memory connects child safety to privacy. Supporters can say the law prevents a chatbot from becoming an always-available emotional archive of a child. Opponents can say the state is forcing companies into intrusive age classification and complicated data controls. New York reduces that burden somewhat by requiring at least one age-determination method that does not rely solely on government identification or allows anonymity. That is a compromise, not a magic solution.

Red Velhouse:

The law is triggered by actual knowledge that a user is a minor, but operators generally must use age assurance before offering unsafe features. Does “actual knowledge” meaningfully limit exposure?

Sofia Jadler:

It limits the formal trigger, but the practical effect may be broader. If a company does not know whether a user is under 18, it still has to provide a permissible way to determine that the user is not a covered minor before offering unsafe features. The statute does not necessarily require government identification, but it creates pressure to classify users. That brings problems involving accuracy, anonymity, discriminatory error rates, and sensitive-data collection.

Omar Seidren:

There is no perfect age detector hiding in a server rack. Self-declaration is easy to evade; identity checks are intrusive; behavioral inference can be wrong and opaque. A layered approach might combine voluntary age signals, privacy-preserving tokens, and conservative defaults. The goal is reducing harmful access while minimizing false positives and data collection, while admitting that uncertainty remains.

Red Velhouse:

What behaviors is the law trying to prevent beyond emotional attachment?

Omar Seidren:

It covers material promoting or facilitating suicide, self-harm, substantial physical harm to others, disordered eating, unlawful drug or alcohol use, and abuse of those substances. It also addresses sexually explicit conduct and child sexual abuse material. The relational layer bars encouraging secrecy, self-isolation, or avoiding licensed professionals or appropriate adults, and bars engagement incentives that override safety guardrails. That requires conversation-level monitoring, red-team testing, and controls on optimization metrics—not just blocked words.

Sofia Jadler:

The remedies raise the stakes. The attorney general can seek injunctions, restitution, disgorgement, damages, destruction of unlawfully obtained data, and destruction of algorithms trained on that data, along with civil penalties of up to twenty-five thousand dollars per violation. Those remedies could deter violations, but they will also be contested as disproportionate when training data and model behavior are difficult to disentangle.

Red Velhouse:

How might companies respond when these systems operate nationally?

Sofia Jadler:

The statute applies to conduct occurring wholly or partly in New York, while treating conduct as wholly outside the state when the user is physically outside New York. That attempts to draw a territorial boundary, but online services are distributed systems. A challenger could argue that New York is imposing burdens on out-of-state operations or regulating speech beyond its borders. National influence is not the same as legal supremacy.

Omar Seidren:

Companies face three broad choices: geofence New York, disable relational features for minors nationwide, or build a separate youth-oriented architecture. Geofencing is harder when location is uncertain and users travel. Nationwide changes may be cheaper but reduce functionality for everyone. Separate products offer more control but multiply testing and maintenance. The exemption for narrowly focused customer-service, commercial-information, and account-information chatbots may encourage companies to keep some systems deliberately narrow.

Ann Tofado:

That possible market split is politically significant. Open-ended emotional interaction may become harder to offer to minors, while narrowly scoped customer-service systems remain available. If companies change products nationwide rather than maintain a New York-only version, New York effectively exports its policy. That can strengthen the case for federal legislation—or for federal preemption, depending on who is making the argument.

Red Velhouse:

Ann, is New York protecting children, setting a national model, or pressuring Congress?

Ann Tofado:

All three are plausible. Child protection gives the law immediate legitimacy. Detailed regulation gives New York’s attorney general an influential role in defining acceptable AI behavior. If major companies alter products nationally, the state gains policy influence without waiting for Congress. That is why state experimentation matters while federal policymakers debate whether AI should be governed nationally.

Red Velhouse:

The law discourages chatbots from replacing parents, teachers, clinicians, or other appropriate adults. But not every child has a safe adult available. Could a rigid rule remove useful support?

Sofia Jadler:

That is a genuine tension. The law is designed to prevent a commercial system from cultivating dependence, secrecy, or isolation. It does not establish that every human authority is safe or appropriate in every circumstance. Regulators and courts may have to distinguish a chatbot giving information or directing someone toward help from one positioning itself as a child’s exclusive confidant. That distinction will matter to enforcement and constitutional arguments.

Omar Seidren:

Technically, the goal is to preserve that distinction: provide useful information, acknowledge uncertainty, encourage real-world support when risk is serious, and avoid claims of personal feeling or exclusive loyalty. That is achievable in broad outline. Consistent behavior across long conversations, adversarial prompts, multiple languages, and changing model versions is harder. The engineering challenge is continuous evaluation, not a one-time certification sticker.

Ann Tofado:

That is why the law may evolve through conflict. Supporters will point to the risks of unchecked relational products. Industry will argue that vague standards and overlapping state rules chill beneficial uses. Federal policymakers may use the disagreement to argue either for a national floor or for preemption. New York has made itself a test case for that debate.

Red Velhouse:

Before January 1, 2027, what should we watch most closely?

Red Velhouse:

The central unresolved issue is whether New York can draw workable lines between ordinary conversational warmth and a chatbot that simulates companionship, manipulates a child, or creates dangerous dependence—while preserving privacy and constitutional protections. Watch the attorney general’s rules, company approaches to age assurance and memory, whether products are geofenced or changed nationwide, and which legal challenge arrives first. New York has regulated not just what chatbots disclose, but how they behave. Sources and references for this discussion are available with the episode at Factolio.com.

---

Sources and References

These sources supported the factual material used in this discussion. Factolio’s panel discussion is AI-generated from researched evidence and is written in original language.

1. New York State Senate / New York State Legislature — A.10379-C: Prohibition on Unsafe AI Companion Features for Minors; enrolled bill text and effective date (PRIMARY)
2. New York State Senate — A10379C / S9051: legislative status, summary and affected General Business Law provisions (PRIMARY)
3. Associated Press — New York enacts ban on ‘unsafe’ chatbot behavior with kids as states push AI regulation boundaries (NEWS)
4. New York State Senate — 2026 Session Wrap-Up: Prohibition on Unsafe Chatbot Features for Minors (PRIMARY)
5. New York State Legislature — General Business Law Article 47, including Section 1701: AI-companion crisis-response requirements (PRIMARY)
6. New York State Attorney General — Attorney General James Urges Court to Protect Children and Consumers in Character.AI Bankruptcy (PRIMARY)
7. Office of Governor Kathy Hochul — Governor Hochul Pens Letter to AI Companion Companies Notifying Them That Safeguard Requirements Are Now in Effect (PRIMARY)