Healthcare’s Fake AI Epidemic

The fastest-growing category in healthcare IT isn’t ambient scribing, prior auth automation, or triage optimization. It’s something much harder to track and infinitely more dangerous: fake AI.

Not “fake” in the sense of deepfakes or fabricated patient records, although those are coming. This epidemic is quieter. It takes the form of pitch decks bloated with buzzwords, demo environments that quietly swap out model outputs for hard-coded templates, and sales teams that don’t know the difference between regression and reinforcement learning.

The reality in mid-2025 is this: a large number of companies pitching AI in healthcare aren’t actually using artificial intelligence at all. They’re using manual labor hidden behind workflows, rules-based automations, or generative language templates that aren’t clinically tuned. Some know they’re faking it. Others just don’t know what AI really requires. Either way, the system is swallowing it—and paying the price.

Vaporware in a Lab Coat

At a national healthcare technology roundtable held this spring, a payer-side CTO summed it up in one sentence: “We’ve stopped asking whether the vendor uses AI. We ask who trained the model, what data they used, and what error bounds they’ve validated. If they can’t answer all three, we walk.”

That caution didn’t arrive by accident. In multiple cases now documented by STAT, health systems have ended pilots after discovering that vendors misrepresented their AI stack by swapping in human-in-the-loop workflows where automation had been promised, or delivering outputs with no visibility into risk scoring or reasoning logic.

A digital transformation officer at a large urban academic medical center shared: “One vendor showed up with an ‘AI-based prior auth generator.’ Their LLM was just a prompt template. When we asked to see the model documentation, they sent a whitepaper with no methods section.”

It’s not isolated. In a Q1 2025 Rock Health report, 74 percent of surveyed digital health buyers said they were “moderately to extremely concerned” about AI misrepresentation in vendor claims. The number is up 30 percent from 2023.

The LLM Shell Game

The explosion of large language models especially open-source variants and cloud-hosted copilots—has created a grey zone where product teams can claim AI “presence” without any AI ownership or oversight.

According to a recent CB Insights AI 100 deep dive, nearly half of all healthcare startups that claim AI capabilities are actually using third-party foundation models with minimal internal training, fine-tuning, or domain adaptation. Many of them don’t control their own prompts. Some don’t even host their own API endpoints.

A former product lead at a digital health startup that recently pivoted out of the AI space said, “We had two engineers. We called it AI. In reality, we piped ChatGPT into a note summarizer with some filters. It looked impressive. But it wasn’t healthcare-ready. And it definitely wasn’t safe.”

This creates a false equivalence in the market. Buyers see a spectrum of “AI vendors” that range from companies with real pre-trained clinical models to teams duct-taping public LLMs into UI demos. The price tags don’t reflect the difference. And in many cases, neither do the outcomes until something fails.

The Risk Gets Real in the Clinic

One of the most alarming effects of this fake AI wave is how often it goes unchallenged until it touches a real patient workflow. A clinical informaticist at a large nonprofit system recalled vetting an ambient documentation vendor whose transcripts looked polished. But the system had no fallback for low-confidence outputs, no structured logic tagging, and no transparency around errors.

“When we tested it in a noisy ED,” she said, “the hallucinations were subtle. Wrong dates, missed allergies, medication names slightly off. If we hadn’t done a manual check, we never would’ve caught it.”

This is exactly the scenario the FDA’s 2024 AI/ML guidance was intended to prevent. But the enforcement reality is far looser. Most vendors stay just inside the “non-clinical support” designation. They claim not to diagnose or prescribe—only to assist.

But these assistants are increasingly making decisions. And if the AI behind them is fake, so is the confidence that surrounds it.

Why the Incentive Structure Still Favors Hype

Part of the problem is commercial. Investors want AI. Enterprise buyers want scale. Product teams are under pressure to claim AI maturity before it exists. The result is a market full of AI theater, where the appearance of intelligence matters more than its provenance.

A health IT advisor to two regional payer-provider networks said, “We’ve seen vendors pull AI from their pitch deck because it scared compliance. We’ve seen others add it just to raise. There’s no consistent threshold. It’s narrative arbitrage.”

Even legitimate vendors get caught in the fog. Companies with real models and validation pipelines are now spending cycles explaining how they differ from the pretenders and fighting upstream against suspicion that should be directed elsewhere.

What Real AI Looks Like

The real AI firms, few though they may be, are investing in explainability, model transparency, audit trails, and enterprise observability. Companies like Hippocratic AI, Abridge, and Navina have begun publishing safety frameworks, citing clinical validators, and demonstrating accuracy boundaries in real clinical environments.

These vendors may still hallucinate. But they don’t hide it. They publish metrics. They show model lineage. They name their sources. And increasingly, they walk away from buyers who expect magic.

Healthcare doesn’t need perfect AI. But it needs real AI. And in a marketplace full of exaggerated claims and unvalidated tools, the presence of even a few honest actors is the difference between safety and fraud.

What Comes Next

This series will continue tracking vendors, investment signals, and governance models that are shaping the next stage of AI’s role in healthcare. If you’ve encountered fake AI whether it was inside a demo, a contract, or a workflow, we want to hear about it. Tips can be shared anonymously through our secure editorial channel. No names or companies will be cited without consent.

Because in healthcare, “AI” without infrastructure is misleading and dangerous.