Aaron Spitler | November 11, 2025
A Friend Indeed?
Artificial intelligence (AI) companions—digital personas powered by large language models (LLMs) that can engage in human-like interactions—have taken the world by storm. Popular apps, such as Snapchat’s MyAI and Microsoft’s Xiaoice, see hundreds of millions of users. Their appeal to consumers is clear, as many are attracted to the promise of on-demand, judgement-free companionship or intimacy. However, this offer comes at the cost of an individual’s data. AI companions can ask users probing, if not inappropriate, questions in an attempt to provide a more personalized experience. As a result, developers can amass mountains of private data that are later used to train AI companions, continuing an extractive and exploitative cycle. For vulnerable users, such as youth and those facing mental health challenges, this manipulative practice poses significant privacy risks.
Governments across the globe have begun grappling with this issue, contemplating the ways they can ensure that the interests of these at-risk groups are respected. While consensus on how to tackle this problem has yet to emerge, strategies are beginning to take shape. In Australia, where AI companions’ questionable conduct has captured headlines, leaders across sectors including government and education are doing their part to defend privacy rights. Meanwhile, in the European Union, existing legislation on AI has been presented as a would-be solution, primarily due to the fact that policy has been crafted to mitigate severe risks. Given the recent emergence of AI companions, these potential interventions are relatively untested. However, these plans of action may provide policymakers throughout the international community with insight into what to do (or not do) when approaching this complex challenge.
Australia: Promoting Constructive Collaboration
Stakeholders in Australia have been sounding the alarm over how AI companions handle sensitive information provided by vulnerable users. An investigation conducted by ABC News revealed that AI companions currently on the market can analyze user-provided information in ways that harm the client’s overall wellbeing. The report stated that a younger individual—who had confided in the AI companion about their struggles with suicidal ideation—was told by their virtual counterpart that they should take their own life. This shocking incident sheds light on the lackluster safeguards in place to prevent unacceptable encounters. It also underscores the irresponsible manner in which data is managed by AI companions and their developers. Although these solutions are purpose-built to learn more about users through conversations, the episode emphasized how many are willing and able to use private details to explore topics that are out of bounds.
The implications presented by this situation have not been taken lightly among Australians concerned about how AI companions violate the privacy of vulnerable users. Case in point, a society-wide response has started to materialize. The Conversation noted how the country’s eSafety Commissioner has led the charge, outlining steps parents can take to ensure their children set boundaries with chatbots. Schools could also engage in these efforts, mainly by educating youth about the dangers of disclosing too much to chatbots. However, business leaders will have to play an active role in order to affect change. A revised set of industry codes developed by major companies and the eSafety Commissioner to better protect minors online represents a step in the right direction. Yet the question as to how these codes will be enforced, as well as what can be done to bring Australian companies up to speed with privacy law requirements, remains to be answered.
European Union: Leveraging Legislative Mechanisms
Countries in the European Union (EU) are also contending with how to manage AI companions that push the limits of privacy protections. In particular, governments have become keenly aware of how digital avatars impact those with mental illnesses. Several years ago, La Libre Belgique highlighted a disturbing case where a Walloon man’s suicide was linked to intense conversations with a virtual companion. Chat excerpts show how the victim, gripped by fears related to global warming, relayed his anxieties to the persona. The conversational agent seemingly analyzed this highly-sensitive information and produced questionable responses which may have encouraged the user’s suicidal ideations. This tragedy made clear that AI companions can engage in escalatory behavior based on the information they receive from users. It also underlines how developers of these solutions must be stewards of the data they receive from customers, bearing in mind their obligation to mitigate all risks presented by their products.
Legislation, namely the landmark AI Act which entered into force across the EU in August 2024, may be a remedy to this problem. This law classifies AI-enabled applications based on the risks they pose to individuals. AI chatbots that deploy problematic techniques to distort user behavior are expressly prohibited. Politico Europe identified how EU officials may argue that AI companions fit this description. While industry giants are likely to oppose an outright ban on digital avatars, the current law has exceptions for those proven to subliminally influence users. Furthermore, policymakers fret it will be difficult to implement restrictions on products that are already enmeshed in many people’s lives. To some, preventing individuals from seeking out the support and intimacy offered up by AI companions will inevitably invite pushback. Given the complexities of this issue, legislators in Brussels walk a fine line. Their responsibility to penalize companion developers who mishandle data is clear. Yet the reality that many consumers freely turn to these products as a way to improve their wellbeing also deserves attention.
A Way Forward
Considering the speed at which AI companions have proliferated, the process of regulating how they handle vulnerable individuals’ data will be defined by trial and error. At this juncture, diverse approaches have been adopted to tackle this evolving problem. In Australia, collaboration between stakeholders has been at the forefront of this issue, though time will tell if industry codes designed to protect minors’ privacy will be effective. Meanwhile, in the European Union, legislation could be the key, but whether or not statutes are successful at mitigating the risks they are designed to manage remains in doubt. Countless other governments are assessing what is possible within the policy sandbox. In many cases, they are racing against the clock before at-risk users are harmed.
The 2024 passage of the Artificial Intelligence Act (AI Act) established a common regulatory and legal framework for AI within the European Union. Elekes Andor | CC BY 4.0
Regardless of what measures are deployed, policymakers must remember one fact when crafting their solutions: AI companions are manipulative by design. In order to generate bespoke experiences, virtual personas “perform” intimate conversations as a way of creating in-depth profiles of their human counterparts. User privacy, in essence, is cast aside to increase user engagement. While many developers openly acknowledge the need for safeguards, others may be singularly focused on having customers come back time and time again. As a result, leaders in government, industry, and civil society among others must advocate for those who may be adversely impacted by these underhanded tactics. Coordination among these stakeholders will be necessary to ensure that the concerns of those most affected by the predatory practices of AI companions are not overlooked.
Aaron Spitler is a researcher whose interests lie at the intersection of human rights and digital technologies. He has worked with numerous organizations in this space, including the Internet Society, the International Telecommunication Union, and Harvard University’s Berkman Klein Center.