Estimated reading time: 5 minutes · Last updated:
AI risks in healthcare are not only about dramatic failure modes; they can show up as thousands of small changes that cumulatively degrade safety and quality. James Moade, Associate Director at the Accreditation Commission for Health Care (ACHC), argues that those changes — which he groups into six warning signs — will be hardest to spot when they are packaged as efficiency gains. Moade draws on industry experience spanning 25 years to say the key questions are governance, who is accountable, and whether organizations can demonstrate that documented compliance matches what actually happens at the bedside.
Key takeaways
- Author and affiliation: James Moade of the Accreditation Commission for Health Care (ACHC) frames AI risk in healthcare around six concrete warning signs.
- Resignations cited: OpenAI safety researcher Jan Leike resigned in 2024 over safety concerns, and Jacob Coxon left Anthropic and issued public warnings.
- Core concern: Moade warns that AI-driven gains in speed and cost can mask losses in empathy, clinical intuition and other hard‑to‑measure quality elements.
- Accreditation shift: ACHC expects documentation to become cheap and proof of real-world practice to become the more valuable signal for assessors.
Table of contents
- Key takeaways
- Why small changes matter more than single catastrophes
- Decision support becomes decision replacement
- When efficiency shadows quality and accountability
- What accreditation must look for now
- Operational risks, human skill loss and systems control
- Page provenance and embedded figures
- Positive case and negative case for AI in healthcare
- What to be careful about
- Frequently asked questions
Why small changes matter more than single catastrophes
The article’s central argument is that AI is likelier to erode care through many modest decisions than through an isolated, spectacular failure. Each automation that saves minutes or money can also transfer a sliver of clinical judgment from people to systems. Over months and years, those slivers add up into measurable changes in how teams diagnose, escalate and follow up.
Moade frames the problem with a practical question: when will we notice the water getting hotter? He argues that healthcare should look for systematic shifts in behaviour — who reviews a recommendation, what is measured on dashboards, and which interactions are being shortened — rather than treating every AI deployment as a one‑off experiment.
Decision support becomes decision replacement
One of the six warning signs Moade lists is the transition from decision support to decision replacement. He points to automation bias: when a tool offers a confident answer, clinicians increasingly ask what the system said rather than what they think. That risk is especially salient for generative AI, which can present incorrect outputs with high confidence.
The remedy, Moade suggests, is to preserve human accountability and the clinician’s active role in decisions. ACHC’s principle is that AI should augment human expertise, not replace human accountability, and deployments must make clear which decisions remain the clinician’s to validate.
When efficiency shadows quality and accountability
Moade warns that efficiency metrics can become proxies for quality. Healthcare organisations optimise what they measure; AI excels at improving measurable throughput, documentation and scheduling. But empathy, intuition and a clinician’s extra five minutes rarely appear in a dashboard even though they can be the difference between a near miss and a serious harm.
A second and related warning sign is diffusion of responsibility. Moade notes that when an AI recommendation is accepted, responsibility can become ambiguous: the system, the vendor, the approving hospital, the data supplier — who is legally and ethically accountable if something goes wrong? He insists accreditation should preserve clear lines of accountability, not allow technology to create gaps.
What accreditation must look for now
Accrediting organisations, Moade argues, must shift from checking produced documents to verifying operational reality. He lists practical questions accreditors should ask: where is AI used, who approved it, what choices can it make, how was it validated, how is performance monitored, how are bias and failures identified, what are override paths, and who is accountable.
ACHC expects a move toward observational evidence: talk to staff, follow processes, and test whether policies match day‑to‑day work. That change matters because documentation can be generated by AI faster than a culture of compliance can be built, and Moade’s phrase captures the risk: documentation will become cheap; proof will become valuable.
Operational risks, human skill loss and systems control
Moade describes two further warning signs: human skill erosion and the proliferation of AI agents that act on systems. If clinicians stop performing tasks because AI does them, they may lose the ability to spot errors; Moade gives a practical horizon of roughly five years as the timeframe in which oversight skills could atrophy if unchecked. That gap undermines the declared safeguard of 'human oversight' unless people retain hands‑on experience.
He also warns about AI agents that interact with software and infrastructure across supply chains, pharmacy, labs, devices and emergency systems. The stakes change when AI moves from informing to taking actions on behalf of organisations; the potential for systemic harm grows if oversight, governance and monitoring are not tightened.
Page provenance and embedded figures
The hosted page includes multiple embedded figures and metadata beyond the essay. Links on the page are dated 12 October 2026 and 13 October 2026; a related podcast is labelled episode 203; and a linked funding story references $40M. The author states a career length of 25 years and cites industry resignations in 2024.
The page displays cookie durations and other site settings as numeric values: 1 year; session; 6 months; never; 1 year 1 month 4 days; 1 year 24 days; 7 days; 1 hour; 10 minutes; and 1 day. Older featured headlines include years such as 2021, 2020, 2023 and 2024. I include these entries to preserve the explicit figures and dates the publisher provided, and to keep my analysis separate from the site's interface elements.
| Warning sign | What it looks like | What accreditors should check |
|---|---|---|
| Decision replacement | Clinicians follow AI recommendations without question | Observe decision points; test clinician ability to override |
| Efficiency over quality | Dashboards improve while patient‑facing time shrinks | Compare measured metrics with observed interactions and outcomes |
| Accountability gaps | Multiple parties involved, unclear responsibility | Trace approvals, vendor roles and post‑deployment monitoring |
| Human skill loss | Tasks stop being practiced by staff | Verify staff competency and conduct hands‑on assessments |
Positive case and negative case for AI in healthcare
The case for
- AI reduces administrative burden across scheduling, documentation and coding, freeing clinicians for direct care.
- When governed, validated and monitored, AI can surface patterns humans miss and potentially improve outcomes.
The case against
- Widespread automation bias and overreliance could erode clinicians’ diagnostic skills and situational judgment.
- Documentation generated by AI can mask weak processes; if accreditation checks documentation alone, real safety can decline.
What to be careful about
- Automation bias leading clinicians to accept incorrect AI outputs
- Diffusion of accountability when vendors, hospitals and algorithms share decision chains
- Loss of human oversight capability if tasks are not periodically practiced by clinicians
- Systemic harm from AI agents taking actions across supply chains, devices and clinical infrastructure
The bottom line
AI can deliver major benefits in healthcare, but James Moade’s central point is pragmatic: the biggest danger may be gradual erosion of judgement and safety under the guise of efficiency. Accrediting bodies are well placed to ask the uncomfortable operational questions — who approved the tool, how was it validated, who is accountable, and how is real practice demonstrated. That shift from paper evidence to observed reality is how healthcare can harness AI’s gains while spotting when the water starts to get hot.
What to watch
- Watch how accrediting organisations redefine evidence of practice; no date has been set.
- Watch for formal guidance from regulators or national health bodies on AI governance in clinical settings; no date has been set.
Frequently asked questions
What specific AI warnings does ACHC identify?
ACHC, via James Moade, highlights six warning signs: decision replacement, efficiency conflating with quality, accountability gaps, questionable compliance evidence, human skill erosion, and AI managing critical systems.
Which prominent resignations are cited as evidence of concern?
Moade cites Jan Leike, an OpenAI safety researcher who resigned in 2024, and Jacob Coxon, who left Anthropic and issued public warnings, to show unease among builders of advanced systems.
How should accreditation change in response to AI?
Moade argues accreditors should move from document checks to observing practice, verifying outcomes and asking who approved AI, how it was validated and how performance is monitored after deployment.
Related reading
This article is information, not medical advice. Anyone acting on it should speak to a qualified professional.
