When Does An AI Chatbot Assistant Become A Regulated Chatbot?

Key takeaways

• Bill C‑34 enacts the Digital Safety Act, imposing safety obligations on AI chatbot services, with broader implications for online safety.
• The definition of “chatbot service” includes elements such as communication over the Internet and the ability to simulate human-like relationships.
• The Bill’s focus on capability rather than intent means that various AI systems may fall under the “chatbot service” category, influencing regulatory scope.

As discussed in our earlier overview, Bill C‑34 would enact the Digital Safety Act (DSA) and the Digital Safety Commission of Canada Act, a broad online safety regime covering social media services, chatbot services and other online services identified in future regulations. The inclusion of chatbot services is particularly significant, as Bill C-34 would establish a statutory framework imposing direct safety obligations on operators of regulated AI chatbot services, including obligations relating to harmful content, chatbot behaviour and transparency.

The new framework is not limited to products marketed as “chatbots” or “AI companions.” Rather, it is anticipated that the DSA will extend to general-purpose AI assistants and conversational interfaces embedded in many other digital services. The purpose of this Osler Update is to explore which chatbot services will be regulated. A future Update will describe the compliance obligations applicable to regulated services. 

What is a ‘chatbot service’?

 Bill C‑34 defines a “chatbot service” as “an artificial intelligence system that

1. communicates over the Internet;
2. is made available on a website or through an application that is publicly accessible in Canada;
3. uses a natural language interface to provide, in a conversational format, adaptive, human-like responses to user inputs;
4. is capable of being used, by means of multiple interactions or sessions, to simulate a sustained human-like relationship with a user, including one that may resemble friendship, an intimate relationship or therapeutic support; and
5. generates content or responses that are not fully predetermined by the developer of the system or the person that operates the system”

The five elements of the definition are cumulative, with elements (a), (b), (c) and (e) being satisfied by widely used AI systems, such as OpenAI’s ChatGPT and Anthropic’s Claude. These cloud-based conversational AI systems communicate over the Internet and are made available through publicly accessible websites or applications. In addition, they are powered by large language models, which use natural-language interfaces and generate adaptive, human-like responses that are not fully predetermined. Even a paid service, or one requiring registration, will likely satisfy these elements of the definition if it is generally offered to the public. By contrast, an internal enterprise assistant restricted to an organization’s personnel likely would not.

The definition appears to draw on concepts reflected in recent U.S. state companion chatbot and conversational AI laws. For example, although different in scope and formulation, California’s definition of “companion chatbot” and Washington’s definition of “AI companion chatbot” both refer to a natural-language interface, adaptive, human-like responses and the ability to sustain a relationship across multiple interactions.¹

Capability rather than intent or outcome

The least certain element of the definition of chatbot service is subsection (d), which narrows chatbot services to an AI system that is capable of being used to simulate a sustained human-like relationship with a user.

The phrase “capable of” gives the definition considerable breadth. Bill C-34 does not require that a system be designed, marketed or primarily used for human-like relationships. Similarly, it does not require that users have actually formed human-like relationships. Rather, the test is capability: Is the service capable of being used, by means of multiple interactions or sessions, to simulate a sustained human-like relationship with a user?

Some services are more likely to fall within the definition of chatbot services than others. AI companions explicitly designed for friendship, intimacy or therapeutic support would be strong candidates, as these are the very use cases element (d) appears intended to address.

Other services occupy a greyer zone. Consider virtual personal fitness trainers that offer ongoing motivation and track a user’s progress, language or academic tutors that adapt to learning styles and encourage students through setbacks, or professional career mentors that provide personalised guidance over time. None of these are marketed as companions, yet each may be capable of simulating a sustained, human-like relationship through repeated interactions.

The position is similarly unclear for a general-purpose assistant marketed for research, writing, coding or productivity that can remember a user’s preferences, adopt a requested personality, respond empathetically to personal disclosures and maintain conversations across sessions. Is it sufficient that a user can prompt such a system to behave like a therapist? How reasonably foreseeable must this capability be during ordinary use, and can an AI system remain outside the definition of chatbot service if its terms of service prohibit companion uses, or if technical safeguards make such uses difficult?

U.S. chatbot laws appear to vary in the criteria they use to determine which systems are covered, including design intent, predominant function and capability. For example, New York and Oregon’s laws limit their application to systems “designed” to simulate sustained human-like relationships through memory of prior interactions, unsolicited emotion-related questions and ongoing personal dialogue.² Idaho covers systems that “primarily” simulate human conversation and interaction.³ The laws in California and Washington focus on capability, although California limits its definition to systems capable of “meeting a user’s social needs.”⁴

Although not directly analogous, Bill C-34’s approach also contrasts with the European Union’s Artificial Intelligence Act, where “intended purpose” plays a central role in classification while “reasonably foreseeable misuse” informs the scope of risk management obligations.

What is a ‘sustained human-like relationship’?

The definition of chatbot service in Bill C-34 also turns on what constitutes a “sustained human-like relationship.” The word “sustained” implies continuity beyond a single interaction, though Bill C-34 does not prescribe a minimum duration or frequency. “Human-like relationship” suggests that sounding human is not enough. Features such as memory of prior interactions, personalization, and use of emotional language could all contribute to a human-like relationship. Notably, the Bill does not clarify whether continuity must be created by the AI system itself — for example, through memory of prior interactions — or may instead be supplied by the user across interactions.

How this element of the definition will be interpreted is unclear. Consider a customer support chatbot that remembers a user’s past complaints, adapts its tone to frustration, and follows up over multiple sessions. Such a service may be more difficult to classify than a transactional banking bot, even if neither is marketed as a companion nor designed for relational purposes.

By comparison, New York and Oregon’s laws apply to AI systems designed to simulate a sustained human-like relationship through specific means, namely retaining information regarding prior interactions or sessions and preferences for personalization and ongoing engagement, asking unsolicited emotion-based questions going beyond direct responses to prompts and sustaining an ongoing dialogue concerning matters personal to the user.⁵

‘Exclusively’ serving a specified purpose

Bill C-34 would allow future regulations to exclude AI systems that “exclusively” serve a specified purpose. Because the exclusion turns on exclusivity, an AI system must serve one purpose and nothing else to be eligible. General-purpose or multi-purpose systems appear unlikely to qualify.

By contrast, several U.S. chatbot laws calibrate their exclusions along several axis rather than exclusivity alone. Other factors include whether a system is “primarily designed and marketed” for a purpose or user group, whether the service is a feature within another application, the functional constraints of the service, and the product class of the service.

From chatbot service to regulated chatbot service

Assessing the definition of “chatbot service” is only the first step in evaluating whether Bill C-34 regulates a particular service. The Bill stipulates that a chatbot service becomes a “regulated chatbot service” if it

1. meets a user threshold to be prescribed by regulation for that type of service; or
2. falls below the applicable threshold but is designated as regulated because the Governor in Council is satisfied that it poses a significant risk of harm to individuals in Canada

Different thresholds may apply to different service types. The drafting also suggests that scale and risk are distinct considerations. A smaller service designed around intimate encounters or therapeutic support may attract regulation even if it falls below general user thresholds.

Looking ahead

Bill C-34 reflects an emerging view that conversational AI poses distinctive risks when it becomes persistent, personalized and relational. Organizations offering conversational AI in Canada should begin assessing not only their systems’ intended functions, but also the relationships those systems are capable of simulating. A follow-on article will examine who qualifies as an “operator” of a regulated chatbot service and what operators will be required to do.

Footnotes

1 Cal. Bus. & Prof. Code div. 8, ch. 22.6 (Companion Chatbots); Wash. Rev. Code ch. 19.440 (Artificial Intelligence—Companion Chatbots) (effective Jan. 1, 2027).

2 N.Y. Gen. Bus. Law art. 47 (Artificial Intelligence Companion Models); 2026 Or. Laws ch. 85 (An Act relating to artificial intelligence companions) [PDF] (effective Jan. 1, 2027)

3 Idaho Code tit. 48, ch. 22 (Conversational AI Safety Act) (effective July 1, 2027).

4 Cal. Bus. & Prof. Code ch. 22.6; Wash. Rev. Code ch. 19.440.

5 N.Y. Gen. Bus. Law art. 47; 2026 Or. Laws ch. 85.