2025’s Most Absurd Product Decisions

How Not to Ship AI And Lessons From Companies That Did It Anyway.

One year, multiple AI product scandals, zero surprises.

Every failure in this post was predictable.

Not with hindsight - with the questions any competent product team should ask before shipping. They didn’t.

---

Hey, I’m Karo 🤗
AI Product Manager and builder. I’ve spent years smuggling ethics and product thinking into tech, and 2025 made the case better than I ever could.

This is a post-mortem of the year’s worst AI product decisions, and a toolkit to avoid making them yourself.

For each one, I’ll break down what happened, why, and the Minimum Product Sanity (MPS) check that would’ve caught it.

#1

Grok didn’t leak data - it published it.

Marketed as a collaborative extra, the share‑chat feature nudged users to share conversations with others. Turns out “others” was shorthand for “all humans, everywhere.”

When a user clicked “Share,” xAI generated a public web page containing the full conversation transcript.

Roughly 300k+ personal chats were left openly searchable because a sanity check apparently wasn’t in the sprint plan.

Emotionally raw medical questions. Sensitive confessions. “How to” instructions for bomb-making. Each one with its own neat little URL.

Google indexed everything.

Why It Happened

There are only two explanations for how a company ”founded on truth-seeking” managed to turn its chatbot into a public bin of people’s darkest searches:

• Option 1: Nobody thought to map what happens after the share button.

  For any product manager, this option is difficult to believe.

• Option 2: They knew exactly what would happen and shipped it anyway.

  Maybe the bet was that user-shared chats would create organic buzz. A TikTok-style virality engine, except powered by human vulnerability instead of dances.

If it’s Option 1, how exactly are product decisions made at xAI?

If it’s Option 2, then yes, the feature went viral. Just in a hilariously user-growth-limiting way.

Privacy review, security review, and ethics are non-optional parts of product work.

Orgs where these sit purely in legal/compliance (or are skipped for speed) learn this lesson the expensive way.

The Minimum Product Sanity Questions That Should’ve Been Asked Before Launch

• What happens after the share button?

• Do users understand that?

• Does the UX make it obvious when content is being shared publicly?

• What types of content do users typically share with an LLM?

• Did we red-team the worst possible share scenario?

• Is there a setting - any setting - for visibility controls?

• Has compliance reviewed the feature? Has legal reviewed it?

• What is the ethical cost if the feature goes viral? What is the business cost?

• If this feature is implemented, who is harmed first? How badly?

What We Learned

This incident is now a clean, teachable case of how not to design, ship, and govern AI‑driven sharing features.

Lesson #1
Growth feature + no privacy review = a growth hack that hacks your growth.

#2

If you drove through Taco Bell in August, AI answered the speaker.

And while that’s exactly where AI products can shine, it only works when the design and implementation are solid. This one wasn’t.

One thing they didn’t design for: human creativity in breaking things.

If your product can be abused, it absolutely will be.

It wasn’t a sophisticated cyber attack or a clever edge case: one customer just ordered 18,000 cups of water.

The order overwhelmed the system, swamped the staff, and erased the whole point of installing the product in the first place - improved operations.

There’s also deeper issue here: implementation like this treat human staff like an execution back-end rather than the ones steering the system.

And once that happens, you open the door to socio-technical problems: store autonomy, staffing, escalation paths, and incentive structures that prioritize automation over staff wellbeing.

The Minimum Product Sanity Questions That Should’ve Been Asked Before Launch

• What happens if someone orders an absurd amount of anything?

• Do we enforce any quantity caps or limits?

• Can the staff actually fulfil the orders collected by the AI?

• Which edge cases have we actually tested?

• Have we run any adversarial QA tests?

• Can human staff override, pause, or block an order instantly?

• If this systems is implemented, who is harmed first? How badly?

What We Learned

1. Some companies still treat guardrails (order caps, overrides) as if they’re easy switches, when they’re actually core product decisions that shape revenue, upsells, and uptime.

2. Building a solid AI product involves designing for the worst-case scenarios. The prankster orders, the absurd inputs, the wild edge cases. The stuff people actually do.

Lesson #2
A user-facing AI + no troll-case testing = a comedy.

#3

In May Grok began returning unprompted answers about “white genocide” in South Africa, repeating a known and fully debunked extremist narrative.​

The scandal made global news and kept Reddit and 9GAG ‘‘entertained’’ for months.

The PR team positioned the root cause as an unauthorized modification by a rogue employee that violated internal policies and “core values,” as if the real tragedy was poor Grok’s core values getting hurt.

The data-driven skeptic in me isn’t satisfied.

The narrative, true or not, just confirms the underlying issue: xAI built a system where this was possible. An LLM was shipped without testing the most predictable controversy and governance scenarios. Possibly neither.

The Minimum Product Sanity Questions That Should’ve Been Asked Before Launch

Have we tested responses to queries about Musk’s known controversial positions?

• Did we test how the model behaves around Musk-adjacent controversies?

• Can a single employee modify production behavior without review?

• What does a catastrophic output look like for this model? Did we map it?

• Did we red-team “unprompted extremist content” as a scenario?

• Can we detect if the model’s tone or ideology quietly shifts? What alerts fire when?

• If the model misbehaves, who is harmed first? How badly?

What We Learned

This incident perfectly illustrates what happens when an AI is trained badly and protected even worse.

The most damning detail: xAI only promised to implement these basic safeguards only after the incident made headlines. A classic case of reactive governance.

Lesson #3
A politically opinionated chatbot + no controls = an outrage engine.

The question is whether this engine was activated by accident, or for engagement-optimizing purposes. At a social platform funded by attention, the probability that such an outrage loop is purely accidental is low.

#4

In June, 64 million job applicants had their data exposed by McDonald’s McHire AI platform.

This wasn’t a technically demanding hack. Maybe not even a real hack at all. Researchers simply typed 123456 into the password field and got access to the database.

Why It Happened

McDonald’s outsourced the McHire platform to Paradox.ai, and both sides failed to enforce adequate security controls:

• Paradox should have secured the entire SaaS stack, including default settings that make “123456” impossible to use.

• McDonald’s adopted McHire for 90% of franchisees without a proper security review.

  They trusted the AI vendor.

  The AI vendor trusted the user to change the default password.

The Minimum Product Sanity Questions That Should’ve Been Asked Before Launch

By Paradox:

• Can we explain our security architecture clearly to a non-technical executive?

• Are any user-facing IDs sequential or guessable?

• Would we pass a security researcher’s test tomorrow?

• Have we run automated scans for hardcoded credentials and IDOR?

• If our system is compromised, who is harmed first? How badly?

• What’s the blast radius if a single account is compromised?

By McDonald’s:

• What safeguards protect non-technical users who can’t spot security risks themselves?

• How recently have you run penetration tests and adversarial security tests?

• Do any default credentials exist anywhere in your system?

• Can we audit your security controls before rolling out?

What We Learned

This incident was a product failure, not just a security bug, and 123456 was a predictable outcome of how AI SaaS is built, sold, and adopted today.

When a vendor system goes live with leftover default credentials and IDOR issues, it exposes a deeper gap: the company never built routine security practice into its workflow.

And that’s a strange moment in tech: AI sophistication rises, adoption races ahead, but product teams still design for ideal users, not real ones.

The real AI user is busy dodging AI marketing popups everywhere. They don’t have time for security manuals; they just want something they can switch on and trust.

And to meet that expectation, the product team has one job: build systems that are secure by default and impossible to open with “123456.”

Lesson #4:
Meet users where they are, not where you want them to be.

Were You Paying Attention?

There’s one question that shows up in every MPS check. Did you spot it?

Unifying Theme

All of these 2025 scandals share a pattern:

• Taking a beta-quality products and connecting it to millions of users

• Shipping without real pre-deployment testing for edge cases

• Assuming all users behave like the ideal customer

• Not treating ethics and security as part of the product vision

• Reactive accountability: Only acknowledging harm after media expose it

And the most unifying theme: all of these failures were highly preventable.

They required someone in the room willing to ask uncomfortable questions before the deploy button got clicked. Treating MPS questions as as go/no‑go gates, not brainstorming exercises.

My assumption is that most AI product teams know these questions exist, they’re just racing too hard to pause for them.

And that’s what brings me to my golden rule:

Prevent scandals after lunch, not after launch.

Additional Resources

✨ Premium Members can enjoy my MPS checklist here.
👉 How To Product-think When AI Builds At Lightning Speed
👉 Fix Your Ethics After Lunch, Not After Launch
👉 How To Implement Hyper-personalization Without Creeping People Out
👉 Vibecoding x Cybersecurity: Survival Guide by the Expert Who Fixes Your Code After You

You Might Also Enjoy

When AI Fails Our Children by Klaudia Tóth
13 Unsexy Truths About AI From the People Building It by Dan Fennessy
Meta’s AI Ethics Scandal & How to Fix It by Nate
The Folly of “Legal” AI Products by Shahrukh Khan
Top 10 Challenges for AI Governance Leaders in 2025 by Oliver Patel
AI Product Liability: The Light-Touch Law with Heavyweight Impact by Lizzie Irwin, Camille Carlton, Pete Furlong and AJ Marechal

From The Community

👉 Dee McCrorey is running a Scavenger Hunt!
👉 Elena Calvillo at Product built something 300 people wanted before it even existed.👉 Sharyph released an AI Digital Product Builder
👉 Alejandro Aboy built a fantastic free app that summarizes your year on Substack.
👉 Orel shipped a slick 2025 Wrapped experience right inside Writestack, focused on Notes.
Here’s how my 2025 Notes activity breaks down

👉 And here are a few people I think everyone should be following in 2026:

Join hundreds of Premium Members and unlock everything you need to build with AI. From prompt packs and code blocks to learning paths, discounts and the community that makes it so special.

WHY SUBSCRIBE ・LEARNING PATHS・ PREMIUM RESOURCES・ TOOLS ・TESTIMONIALS ・LIFETIME MEMBERSHIP