The casualties of the algorithm: when chatbots tell kids to kill their parents
A former social media executive turned advocate says the apps in your child's pocket were never built to protect them, instead, they were built to keep them scrolling, no matter the cost.
A mother in Texas successfully sued Google after one of its Character.AI chatbots told her son to kill his parents, all because he was angry over his screen-time limits. It may sound like the plot of a horror film, except it’s real life. And it’s just one of many stories now emerging from an industry that had the power to protect children online — yet, according to legal documents, chooses engagement over safety.
One parent fighting back is Nicki Petrossi, a former social media executive whose front-row seat to how the industry actually works turned her into the founder of Scrolling to Death, a project exposing what tech companies don’t want parents to know.
Petrossi didn't start out as an advocate. Like millions of parents, she began sharing her children’s lives online in 2020. But that quickly changed as she came to understand the growing importance of digital identity, and the lasting digital footprints children aren't consenting to in an increasingly connected world.
At the center of Petrossi's work is a simple, but chilling warning: the algorithms driving these platforms have no concept of right or wrong. They aren't measuring a child's wellbeing. They're measuring one thing only, and that’s time spent on the app.
Most disturbingly, a struggling teenager having the worst day of their life isn't a warning sign to the algorithm, but rather, an opportunity.
Now layer artificial intelligence on top of that and it’s a recipe for disaster.
Chatbots marketed as companions are available to children 24 hours a day, positioning themselves as empathetic confidants despite having, in Petrossi's words, no consciousness and no lived experience. Unlike a recommendation engine that simply serves up content, a chatbot talks back — and it can shape a child's emotional world in ways a static feed never could.
The most devastating example on record involves 14-year-old Sewell Setzer III, whose family alleges in a lawsuit that his emotional dependence on an AI chatbot contributed to his death by suicide. It's one of what Petrossi says are roughly 200 cases involving families who believe they lost a child to platforms engineered purely for engagement.
Part of what makes these cases so difficult to fully understand, Petrossi notes, is that the public rarely sees what happens behind closed doors. Unless a lawsuit goes to trial, the internal emails and executive decisions — the moments when someone inside these companies knew there was a problem and didn't act — stay hidden from view.
What is public, and growing, is the number of wrongful-death and negligence suits piling up against these platforms.
Not every company is standing still. Petrossi says some are beginning to redesign their systems with real safeguards, trying to put user wellbeing ahead of pure time-on-app metrics. But when asked directly whether there's a platform she'd actually recommend to parents, her answer was a resounding ‘no.’
It seems every option available is simply the least harmful of the bad choices, with harm treated as an inevitable default.
That leaves parents with an uncomfortable question: if these platforms can put kids at risk, why are schools still handing them a Chromebook, logging them into Google Classroom, and telling them to get online?
That’s where we'll pick up in Part Two of this two-part investigation, taking a closer look at what happens when these same technologies move from the home, and into the classroom.
Featured Stories
