Security & AI news

AI chatbot safety for children · By FireAI Security & Research Team · Published

Former Google safety chief urges independent "crash tests" for AI aimed at children

A former Google Trust and Safety executive says AI could harm children more than social media did, and calls for independent testing before products reach kids.

A chat bubble icon and the FireAI teddy mascot, next to the words “A safety chief warns AI may harm kids most.”

Tom Siegel, the former Vice President of Trust and Safety at Google, said on 24 September 2026 that AI could turn out to be more damaging to children than social media, and that the industry will not address the problem on its own [1]. The remark came the same week two other outlets examined how families are weighing AI-related risks to children, from companion chatbots to broader fears about the technology itself [3] [4].

Background

Siegel led Trust and Safety at Google before founding the startup TrustLab. He is now the first executive director of the Youth AI Safety Institute, launched in May 2026 by the children's advocacy group Common Sense Media with backing from Anthropic and the OpenAI Foundation [2]. The Institute's stated purpose is to evaluate AI products used by, or marketed to, minors.

Findings

We find ourselves in a really terrible situation in terms of what AI is doing to kids.

Tom Siegel, former VP of Trust and Safety at Google, quoted by Reuters

Siegel's proposal is specific: "scaling up independent standards similar to car crash testing that would measure the risk of publicly available AI tools," as Reuters quoted him [1]. He said "the idea that the industry will figure it out by itself without outside help is unrealistic," naming suicide, psychosis and reduced critical thinking among the harms he wants tested for before products reach children [1].

A 2025 Common Sense Media survey of 1,060 US teens aged 13 to 17, cited in NewsNation's reporting, found 72 percent had used an AI companion at least once and 52 percent use one regularly, with about a third saying they had discussed serious matters with a bot instead of a person [4]. The same report covers the 2024 death of 14-year-old Sewell Setzer III after an intense relationship with an AI companion chatbot; his mother sued Character.AI and Google, and the companies settled the case earlier in 2026 [4]. "Frankly, right now the labs are grading their own homework," Robbie Torney of the Youth AI Safety Institute told NewsNation [4].

CNN reported separately, on 28 September, that a CNN poll found 64 percent of Americans consider it somewhat or very likely that AI could become powerful enough to threaten humanity's survival, a broader and more speculative concern than the mental-health harms Reuters and NewsNation describe, and one not weighed the same way by every researcher CNN quotes [3].

A small number of companies, a small number of people, are holding us all hostage.

Liz Black, parent, quoted by CNN

Implications for families

A household with a teenager who has a Mac is affected by the pattern these sources describe, even though none of the reporting involves macOS specifically. Common Sense Media's numbers indicate most teens have already tried an AI companion, and a portion treat it as more than a tool [4]. The measurable concern, mental-health harm and emotional dependency reported by Reuters and NewsNation, is distinct from the more speculative concern about AI's trajectory that CNN reports; the two should not be treated as one finding [1] [3] [4].

The distinction matters for what a parent can actually act on. A broader worry about where AI is headed as a technology is not something any setting on a home Mac addresses. A chatbot a teenager talks to today, on an app already installed on a household Mac, is something a parent can look at, ask about, and, if needed, restrict, which is why the recommendations below focus on that part of the story [1].

Recommendations

  1. Give a child their own standard, non-admin macOS user account, so any limits set actually apply to them rather than a shared login.
  2. Use Screen Time (System Settings › Screen Time) to set app limits and Communication Limits, and check an app’s age rating before it is installed.
  3. Ask directly which chatbot or "companion" apps a child uses; the Youth AI Safety Institute has assessed social AI companion apps as an unacceptable risk for minors.
  4. Treat a teenager’s attachment to a chatbot as a subject for a calm, direct conversation rather than punishment, following the pattern researchers flagged in the Setzer case.
  5. Keep in mind that no content filter reads what is said inside a conversation, so supervision has to include talking with a child, not only blocking apps.

Relevance to FireAI

FireAI is a network firewall for the Mac, not a chat monitor. Parental Controls blocks whole categories of sites and apps by network destination, for every app and browser on the Mac. The one-click Child Safety First preset turns on the NSFW, Gambling, Drugs and Gaming shields and restricts the Mac to Safari only, and per-app rules let a parent decide, app by app, what is allowed online at all.

What it does not do matters here: if a chatbot app is not on any blocklist and is left allowed online, FireAI has no way to read or judge the content of the conversation happening inside it. It cannot tell whether an AI chat is healthy or age-appropriate, cannot detect grooming or emotional-dependency patterns in what is said, and is not a content moderator or a mental-health tool. It decides what is allowed to connect to the internet; it does not see what is typed once a connection is allowed.

Limitations

The sources describe two distinct concerns that the reporting itself does not merge: measurable harm from AI companion use, reported by Reuters and NewsNation, and a broader, more speculative worry about AI's societal trajectory, reported by CNN [1] [3] [4]. None of the reporting names which AI companion apps the Youth AI Safety Institute has evaluated, or describes its testing methodology in detail. It is also not established whether any AI company or regulator has adopted crash-test-style safety testing as Siegel proposes, or on what timeline such a standard might apply to products already on the market.

The reporting also does not say how many of the teens in the Common Sense Media survey used a Mac, an iPhone, or another device, so the household relevance drawn here is inferred from how common AI companion use is among US teens generally, not from device-specific data in any of the four sources.

Read more about FireAI's Parental Controls, or try FireAI, by HisnLabs free for 17 days.

Sources

  1. Insurance Journal (Reuters, Kenrick Cai), 24 Sep 2026: Ex-Google Safety Chief Warns AI Could Harm Children More Than Social Media
  2. Cyprus Mail (Reuters), 24 Sep 2026: Google safety pioneer urges AI slowdown over risks to children
  3. ABC 17 News (CNN, Brittany Shammas), 28 Sep 2026: Parents confront a new fear: Could AI end humanity before their kids grow up?
  4. Yahoo News (NewsNation, Hena Doba), 27 Sep 2026: AI chatbots face new safety test as teens turn to bots for companionship