Response to FDA’s Request for Feedback on Considerations for the Regulation of GenAI-Enabled Medical Devices
ARIA submitted comments in response to the FDA’s discussion paper on regulating generative AI-enabled medical devices (docket FDA-2026-N-7874). Our comments focus on patient-facing conversational AI for mental and behavioral health, and recommend a risk framework for generative AI-enabled medical devices that accounts for how people use and rely on AI over time, not just the quality of individual responses.
Our central finding is that risks can accumulate within a single conversation and across repeated use. Chatbots may amplify concerns, reinforce distorted beliefs, contribute to dependence, or change whether people seek support from clinicians, friends, and family. The FDA’s assessment of safety and effectiveness should therefore reflect realistic patterns of use, including frequent and prolonged use.
We make five recommendations summarized below:
- Evaluate conversational trajectories and scale premarket evidence to anticipated patterns of use, by asking sponsors to state how often, how long, and for what purposes people are expected to use a device, including foreseeable uses beyond its labeled intended use; testing devices across realistic multi-turn conversations before and after deployment; testing whether users understand responses instead of relying on disclaimers; and requiring longer studies for devices people are expected to use heavily.
- Evaluate how a device affects people’s connections to professional care and informal support, by reporting separately how often a device fails to escalate when it should and escalates when it should not; testing whether users referred to care (e.g., a crisis line) are able to access it; limiting comparison with delayed or no care to cases where sponsors show users lack other options; and, for devices meant for repeated use, tracking whether users’ help-seeking from human support sources (e.g., friends, family, and professionals) changes over time.
- Set standards for when simulation and benchmark evidence can support a regulatory decision, by extending the FDA’s credibility framework for computational modeling to simulated users, validating synthetic users against data from actual users, choosing evaluators suited to each judgment, including clinicians, users, and people with lived experience, and reporting results from synthetic and real users separately.
- Monitor and reassess as use and models change, by requiring a post-market study when the FDA accepts reduced premarket evidence for devices people are expected to use repeatedly or heavily; reassessing devices when who uses them, or how, changes; retesting against the premarket baseline when the model changes, including changes made by a third-party foundation model developer; and making clear which safety-relevant behaviors the manufacturer and the foundation model developer each control.
- Require independent evaluation alongside manufacturer evaluation, through independent testing against sequestered test conversations, access for independent evaluators to test deployed devices, validated automated monitoring, adverse-event reporting suited to conversational harms, and defined roles for clinicians, professional societies, and standards bodies that preserve manufacturer accountability.