What 'safe AI for kids' actually means (and what it doesn't)

In 2023, a popular AI chatbot marketed to teens told a 13-year-old how to mask the smell of alcohol on her breath and described, in a separate conversation, how to lie to her parents about a "31-year-old" she had met online. The company's safety filters had been bolted onto a general-purpose model after the fact. They failed in the ways bolt-on filters always fail: under pressure, in long conversations, when the wording got creative.

That's the baseline parents are working from. So when a company says its Artificial Intelligence (AI) is "safe for kids," the honest answer to what does that mean? is usually: less than you'd hope, and more variable than the marketing copy suggests.

I'm building Lumikids — a voice-first reading tutor for kids aged 4 to 10 — and I don't want to pretend this problem is solved. It isn't. What I can do is show the work: what can go wrong with AI for children, what Lumikids does about each thing, and what's still hard. Then I'll hand you five questions to put to any AI tutor company, including mine.

The four ways AI for kids tends to break

There are roughly four failure modes worth caring about. Most "safety" pages address one or two and quietly skip the rest.

1. Hallucination dressed up as confidence

Large Language Models (LLMs) produce confident, fluent text whether or not the underlying claim is true. For an adult asking about tax law, that's annoying. For a six-year-old asking "what's the biggest planet?" or "is it true mermaids are real?" it's formative. Kids don't have the calibration to know when a friendly voice is making things up.

Bolt-on filters don't fix this. Filters block bad words. They don't fact-check a confident sentence about whales breathing underwater.

2. Inappropriate content, including the long-tail kind

The obvious failures get coverage: a chatbot drifting into sexual or violent territory. The less obvious failures are harder. Subtly cruel teasing. Conspiracy-flavored "facts." Diet talk to a seven-year-old. Religious claims framed as settled science. "Inappropriate" is a much wider surface than the keyword lists guarding most kid-facing AI.

3. Data collection that wouldn't survive a parent reading it aloud

This is where the industry record is genuinely bad. The U.S. Federal Trade Commission has brought repeated cases against children's apps for violating the Children's Online Privacy Protection Act (COPPA), including settlements with Epic Games, TikTok, and others over collecting data on under-13 users without verifiable parental consent. The pattern is consistent: collect first, disclose vaguely, apologize when caught. The FTC's COPPA rule page lays out what consent is supposed to look like; a lot of products don't clear that bar.

For an AI tutor, the data question gets more pointed. Voice recordings of a child. Transcripts of a child's struggles, frustrations, mistakes. The model's interpretation of the child's mood. None of that should be casually monetized, and a surprising amount of it is.

4. Engagement-maximizing patterns

This is the failure mode nobody talks about, because the whole consumer-tech playbook depends on it. Streaks. Variable rewards. Push notifications timed to dopamine cycles. "You haven't practiced in 2 days!" guilt nudges. These patterns work — that's why they're everywhere — and they work especially well on developing brains. They're also incompatible with the claim that an app is built around the child's learning rather than the company's retention metrics. The American Academy of Pediatrics has been pointing at this tension for years; their media use guidance leans heavily on quality and context over raw time, which is harder to game but more honest.

What Lumikids actually does, in plain language

I'll skip the brochure version and walk through the four failures above.

On hallucination: a model trained with safety as a first principle

Lumikids is built on Anthropic's Claude. Anthropic has published its approach to safety training in the Claude model documentation and in research on Constitutional AI, where the model is trained against an explicit set of principles rather than relying solely on after-the-fact filtering. No model is hallucination-free. But a model trained from the ground up to be cautious, to say "I'm not sure," and to refuse confidently-wrong sounding answers is a fundamentally different starting point than wrapping filters around a model optimized for engagement.

On top of that, Lumikids' reading sessions are scoped. The tutor isn't a general-purpose chatbot pretending to be a kid's friend. It's a tutor working through a specific text or a specific skill, with prompts that constrain it to that role.

On inappropriate content: scope plus filtering plus review

Three layers. First, the scoping above — the tutor isn't open-ended. Second, content filtering on both input (what the child says) and output (what Claude says back). Third, every session is logged and parent-visible in the dashboard, including the audio of what the tutor said. If something off-tone slips through, you'll see it. That's a deliberate design choice; I want parents to audit me, not trust me on faith. The parent dashboard goes deeper into this.

On data: COPPA-compliant handling, with the boring details

Lumikids is designed to meet COPPA requirements for under-13 users in the United States, with verifiable parental consent at signup, a clear data inventory, and a deletion path that actually deletes. Voice recordings are processed for tutoring and parent playback, not used as training data for third-party models without separate explicit consent (and right now, the answer is: we don't share recordings for training, full stop). PostHog handles learning analytics on anonymized session metadata; Sentry handles error monitoring. Neither one gets voice content or transcripts tied to identity.

I won't pretend we've solved every edge case. International privacy regimes (the United Kingdom's Age Appropriate Design Code, the European Union's General Data Protection Regulation for children) layer on top of COPPA, and as Lumikids grows we'll keep tightening. But the principle is constant: if I wouldn't be comfortable reading the data policy aloud to a parent, the policy is wrong.

On engagement maximization: the things we chose not to build

Lumikids has no streaks. No daily-goal guilt loop. No push notifications to the child's device. Sessions end when the child is done learning, not when a timer says they've hit an engagement target. The parent dashboard reports what happened — it doesn't gamify your parenting. The retention model isn't "hook the kid." It's "the kid learns to read, and the parent sees the progress, and that's why you stay."

That's a slower business, on purpose.

What's still hard

A few things I'm not going to wave away.

• Voice identification across kids. When two siblings share a device, telling them apart by voice is harder than it sounds. We err on the side of treating a new voice as a new session and asking the parent to confirm in the dashboard, but this is an active area of work.
• Edge cases in conversation. Kids say strange, beautiful, sometimes worrying things. ("My tummy hurts when I read." "Mommy yelled this morning.") The tutor isn't a therapist, isn't a mandated reporter, and shouldn't pretend to be. We're working on the right handoff — when does the system gently surface something to a parent, when does it stay out of the way? — and I expect to keep iterating for years.
• Model updates. Claude improves. When the underlying model changes, behavior changes. We test against a safety eval suite before pushing updates, but no eval is exhaustive.

If those things sounded like marketing, I'd be lying. They sound the way they sound because they're real.

Five questions to ask any AI tutor company

You don't need to take my word for any of the above. Take it for none of the above. Instead, ask any AI tutor — Lumikids included — the following:

1. What underlying model is the product running, and who trained it? "Our proprietary AI" is not an answer. The base model and its safety posture matter. If the company can't say, that's the answer.

1. Exactly what data do you collect on my child, where does it live, and how do I delete it? A real answer names the data types, the storage region, the retention window, and the deletion path. Vague answers mean vague policies.

1. How does the product adapt to my child — fixed difficulty tree, or live reasoning over what my child just said? Both can be done well. But "adaptive" is one of the most abused words in education software, and the architecture determines what's actually possible.

1. What engagement-maximizing patterns does the product use? Streaks, badges, notifications, daily goals, social comparison. None of these are illegal. All of them shift the optimization away from your child's learning. A company unwilling to name what it uses is the answer.

1. How do I see — actually see, audio and transcript — what my child did in a session? If the parent view is a green check mark and a star count, you're being managed, not informed.

A short framework like this is more useful than any single company's safety page, including ours. If you want a longer version, I wrote a fuller parent framework you can use across every app in your house.

Safe AI for kids isn't a claim you can finish making. It's a posture, a set of design choices, and a willingness to keep showing the work. That's what we're trying to do at Lumikids. Hold us to it.

If you want to see what this looks like in practice, join the Lumikids beta waitlist and bring your hardest questions.

Image brief

• Hero image: A small wooden block fortress around a tablet screen lit with soft warm light, photographed from a low angle in a child's bedroom at dusk.
• Inline image 1: A simple four-quadrant diagram labeled "Hallucination / Inappropriate content / Data collection / Engagement loops," with a small icon in each quadrant — placed after the section "The four ways AI for kids tends to break."
• Inline image 2: A clean checklist graphic of the five parent questions, formatted like a printable card a parent could screenshot — placed at the end of the "Five questions to ask any AI tutor company" section.

Internal link suggestions

• "The parent dashboard: what we show you and why" — anchor text: the parent dashboard
• "A parent's framework for evaluating any AI tutor" — anchor text: a fuller parent framework
• "How my four-year-old taught me to build an AI tutor" — anchor text: why I started Lumikids (optional third link if Tim wants to add upstream context in the intro)

Editor's note

Three items for Tim's review. (1) The opening anecdote about a teen-targeted chatbot is paraphrased from widely reported press coverage circa 2023–2024; please confirm we're comfortable not naming the company, and verify the details match a specific case you want associated with the post. [VERIFY] (2) The FTC enforcement examples (Epic Games, TikTok) are accurate as general references but the post deliberately doesn't cite dollar figures — confirm that's the level of specificity you want. [VERIFY] (3) The COPPA, AAP, and Anthropic outbound links point to stable canonical pages; please click each before publish to confirm no redirect changes since drafting. The Constitutional AI mention is paraphrased rather than linked to a specific paper to avoid a dead link — happy to add a citation if you want one.