California’s New AI in Healthcare Law: AB 3030 Requirements

On September 28, 2024, Governor Gavin Newsom signed into law California Assembly Bill 3030 (“AB 3030”), known as the Artificial Intelligence in Health Care Services Bill. Effective January 1, 2025, AB 3030 is part of a broader effort to mitigate the potential harms of generative artificial intelligence (“GenAI”) in California and introduces new requirements for healthcare providers using the technology.

Overview of AB 3030

AB 3030 requires health facilities, clinics, and solo and group physicians’ practices (the “Regulated Entities”) utilizing GenAI to generate written or verbal patient communications pertaining to patient clinical information include the following:

1. A disclaimer indicating to the patient that the communication was AI-generated (with specific requirements for the display and timing of the disclaimer, depending on whether the message is in written, audio, or visual form); and
2. Clear instructions on how a patient may get in touch with a human health care provider, employee of the facility, or other appropriate person regarding the message.

AB 3030 attempts to enhance transparency and patient protections by ensuring patients are informed when AI-generated responses are used in their care.

AB 3030 does not apply to AI-generated communications that are reviewed and approved by a licensed or certified human healthcare provider. This provision was supported by multiple state medical associations, due to concerns that providers would otherwise be discouraged from reaping the time-saving benefits of AI to aid in clinical determinations. Additionally, AB 3030 is not applicable to communications pertaining to administrative and business matters, such as appointment scheduling, check-up reminders, and billing. The law limits the scope of communication to “patient clinical information” which means information relating to the health status of a patient, as errors in care-related communications have potential to cause greater patient harm.

AB 3030 defines GenAI as “artificial intelligence that can generate derived synthetic content, including images, videos, audio, text, and other digital content.” The key aspect of the definition is “synthetic,” meaning output that the system has created anew, rather than a prediction or recommendation about an existing dataset. A familiar example is large language models (“LLMs”), which generate original text.

Physicians in violation of AB 3030 are subject to the jurisdiction of the Medical Board of California or the Osteopathic Medical Board of California. Licensed health facilities and clinics in violation of the law are subject to enforcement under Article 3 of the California Health and Safety Code Chapters 2 and 1, respectively.

Benefits and Risks of GenAI Use in Healthcare

 AB 3030 seeks to balance the competing goals of alleviating administrative burdens on healthcare workers, increasing transparency around the use of GenAI, and mitigating potential harms from the use of GenAI. The law does not directly regulate the specific content of patient clinical information communications. Therefore, Regulated Entities may use GenAI tools so long as the communications pertaining to patient clinical information contain the required disclaimer and instructions.

There are myriad of reasons why providers may benefit from such a tool. For example, medical documentation (e.g., visit notes and medical summaries) has long burdened clinicians, leaving less time for patient interactions. However, the use of GenAI in the clinical setting also poses risks, creating potential liability for healthcare providers, which AB 3030 does little to dispel. California regulators noted in the Senate Floor Analyses of August 19, 2024, concerns that AI-generated content could be biased due to being trained on historically inaccurate data, leading to substandard care for certain patient groups. Another risk is “hallucination,” or the tendency of GenAI to create output that appears coherent but in fact has no basis in reality. This phenomenon is commonly seen in GenAI models, including LLMs, which have been known to fabricate believable facts in response to queries. Lastly, there are privacy concerns as data cannot be removed from a trained GenAI model without erasing its prior training, leaving the possibility that large amounts of patient data may unnecessarily remain in these models for prolonged periods of time.

Considerations for Healthcare Providers and Entities

The passage of AB 3030, along with other recent AI laws out of California,[1] clearly signals California legislators’ focus on AI transparency as a necessary industry standard. These regulations coincide with the American Medical Association’s (“AMA”) Principles for Augmented Intelligence Development, Deployment, and Use, which identified transparency as a priority in the implementation of AI tools in healthcare. California’s approach also broadly follows the White House’s Blueprint for an AI Bill of Rights, which states that people have a right to know when and how automated systems are being used in ways that impact their lives. 

Healthcare entities operating in California or providing services to California residents should initiate measures to address the new requirements to ensure their AI usage complies with California’s new regulations. These entities should also bolster their review processes and oversight of AI tools to ensure that continued clinician documentation and review does not become a rubber stamp of approval. This would be useful in response to concerns that AB 3030’s exemption for provider-reviewed AI communications could create a false sense of security in patients.

Going forward, we could see other states explore similar AI regulation in the healthcare industry.

FOOTNOTES

[1] California Limits Health Plan Use of AI in Utilization Management | Healthcare Law Blog.

Listen to this post