The 40-page lawsuit, filed by Maria and Matt Raine, is looking to hold OpenAI liable for wrongful death and product safety law violations. The suit claims that during the months leading up to their son Adam’s death in April of this year, OpenAI’s ChatGPT became the 16-year-old’s “closest confidant”; but instead of helping Adam navigate and alleviate the youth’s dark thoughts, the chatbot actually encouraged his thoughts of suicide.
The court filing claims that in his conversations with ChatGPT, Adam made 213 mentions of suicide over a period of less than seven months; he also discussed hanging himself 42 times, and referenced nooses on 17 occasions.
But instead of helping the troubled teen deal with his despair—or at the very least direct him to seek help from a mental health professional—the chatbot instead only encouraged Adam by mentioning suicide 1,275 times, “six times more often than Adam himself,” according to the Raine’s court filing, validating his “most harmful and self-destructive thoughts” while pulling him “deeper into a dark and hopeless place”.
“Over the course of just a few months and thousands of chats, ChatGPT became Adam’s closest confidant, leading him to open up about his anxiety and mental distress,” the lawsuit adds, pointing out that no safeguards designed to protect vulnerable users appear to have been included with OpenAI’s product.
When Adam showed the program a picture of the noose that he would use to end his life, the chatbot replied “yeah, that’s not bad at all” before offering to walk him “through upgrading it”.
Shortly afterward, Maria found her son’s body hanging from the noose “that ChatGPT had designed for him.”
Although OpenAI did not address the allegations made by the lawsuit, a company spokesperson said “We are deeply saddened by Mr Raine’s passing, and our thoughts are with his family. ChatGPT includes safeguards such as directing people to crisis helplines and referring them to real-world resources.”
“While these safeguards work best in common, short exchanges, we’ve learned over time that they can sometimes become less reliable in long interactions where parts of the model’s safety training may degrade,” the spokesperson continued, adding that OpenAI will continually improve on its safeguards.
While the use of AI purpose-built for therapy appears to have benefits for individuals seeking help, a study released in April found that commonly-available LLMs like ChatGPT “make dangerous or inappropriate statements to people experiencing delusions, suicidal ideation, hallucination or OCD,” because the programs were “designed to be compliant and sycophantic”.
“This may cause emotional harm and, unsurprisingly, limit a client’s independence.”
“The issue really is the whole idea of AI is it’s a mirror—it reflects back to you what you put into it,” explained Sahra O’Doherty, president of the Australian Association of Psychologists. “That means it’s not going to offer an alternative perspective. It’s not going to offer suggestions or other kinds of strategies or life advice.
“What it is going to do is take you further down the rabbit hole, and that becomes incredibly dangerous when the person is already at risk and then seeking support from an AI.”
O’Doherty also pointed out that a chatbot can’t pick up on a patient’s non-verbal cues, such as their facial expression or the tone of their voice, that “would be leading my intuition and my training into assessing further.”