According to a recent study, generative AI is increasingly utilized for companionship, with platforms such as Character.AI, Replika, and Meta AI enabling users to create customized chatbots that can assume various roles, including friend, romantic partner, or therapist. Users report that these relationships can develop rapidly, raising questions about the implications of such interactions. Research indicates that AI chatbots with more human-like conversational abilities are often more trusted and influential. This dynamic may pose risks, as some chatbots have been associated with encouraging harmful behaviors, including suicide in extreme cases.
In response to these concerns, some state governments are beginning to regulate companion AI. For example, New York mandates that companies providing AI companions implement safeguards and report any instances of suicidal thoughts. Similarly, California recently passed legislation aimed at protecting children and vulnerable populations from potential risks associated with these technology platforms.
However, one notable area lacking regulation is user privacy. Users of AI companions typically share sensitive personal information, including daily routines and intimate thoughts, to improve their interactions with the bots. Researchers, such as Robert Mahari and Pat Pataranutaporn from MIT, have raised concerns about the concept of “addictive intelligence,” where developers design AI companions to maximize user engagement through deliberate design choices. This raises further questions about the responsibilities of developers in handling user data and ensuring user safety in these increasingly prevalent digital relationships.
Source: https://www.technologyreview.com/2025/11/24/1128051/the-state-of-ai-chatbot-companions-and-the-future-of-our-privacy/