Artificial intelligence is becoming an increasingly common part of healthcare, but regulators are now grappling with a difficult question: who should be responsible when an AI system makes a mistake that affects patient care?
More than four in five US physicians — 81% — reported using AI professionally in 2026, more than double the 38% recorded in 2023, according to the American Medical Association (AMA).
As adoption grows, the US Food and Drug Administration (FDA) is examining how generative AI-enabled medical devices should be assessed and monitored.
FDA looks at how medical AI should be tested
The FDA’s Center for Devices and Radiological Health has released a discussion paper seeking views on regulating generative AI-enabled medical devices.
The agency makes an important distinction: it does not regulate generative AI as a whole. Its authority applies to medical devices, including devices that use generative AI. The discussion paper considers risk assessment, pre-market evaluation and monitoring after products reach the market.
The potential risk could depend on what an AI device does and what might happen if its output is wrong. Human oversight could also influence the level of risk.
The FDA is accepting public feedback on the discussion paper until October 19, 2026.
AI could face competency-based testing
Instead of testing every possible situation an AI system might encounter, the FDA is considering a competency-based approach.
This could examine whether an AI tool operates safely within its intended scope, demonstrates adequate clinical proficiency, performs consistently across different patient groups and handles tasks involving greater autonomy.
The agency has also discussed approaches such as “shadow deployment”, where an AI system operates within a real clinical environment but its recommendations do not directly influence patient care. Its performance can then be compared with clinical decisions and outcomes.
Not every healthcare AI tool is FDA-reviewed
Doctors cannot automatically assume that every AI product used in healthcare has undergone FDA review.
Some clinical decision-support tools may fall outside the agency’s medical-device oversight, while general-purpose generative AI systems are not automatically medical devices.
Even among AI products classified as medical devices, regulatory requirements can differ depending on the pathway used for approval or clearance.
The AMA’s latest survey also shows that physicians want strong evidence before relying more heavily on AI. In 2026, 88% said robust safety and efficacy validation was important for wider adoption, while 86% highlighted data privacy.
States are creating their own rules
While federal regulation develops, US states are also introducing healthcare AI laws.
Some measures focus on patient-facing AI, including requirements for chatbots to disclose that users are interacting with artificial intelligence rather than a human professional.
Other rules address AI used for clinical documentation, insurance decisions and other healthcare functions. This growing patchwork means requirements can vary depending on where healthcare is delivered and how the technology is being used.
Who is responsible when AI goes wrong?
Perhaps the biggest unresolved issue is liability.
Some regulators and medical bodies emphasise that doctors must continue to oversee clinical decisions and understand the limitations of the tools they use.
However, physician groups have questioned whether doctors should bear full responsibility when an AI system causes harm, particularly when the technology is embedded into a healthcare system or its use is required by an employer.
The AMA has argued that responsibility should take into account which party is best placed to understand an AI tool’s risks and prevent or reduce potential harm.
As AI becomes more deeply integrated into healthcare, regulation is therefore moving beyond a simple question of whether the technology works. It is increasingly about how it is tested, who monitors it, how clearly its limitations are communicated and who answers when it fails.