AI Agents in Healthcare: What HIPAA Actually Allows

Healthcare organizations building AI agents face a compliance question that has no clean answer in most existing HIPAA guidance documentation: does an AI model that receives a prompt containing patient information count as a business associate? Does the inference call itself constitute a use or disclosure of PHI? Does automated clinical decision support require a separate consent framework from the one already in place for the EHR system?

These are not theoretical questions. They are the questions that determine whether an AI agent built for a healthcare client can be deployed in six weeks or six months, whether it requires a BAA with every LLM API provider in the chain, and whether the audit logging architecture needs to capture individual inference calls or just aggregate system events.

The U.S. AI in healthcare market was valued at USD 18.1 billion in 2025 and is projected to reach USD 222.9 billion by 2033, per Grand View Research. The AI Agents in Healthcare segment alone reached USD 1.11 billion in 2025 and is growing at 44.1% CAGR to USD 6.92 billion by 2030, per MarketsandMarkets. The compliance question is not going to get less important as that market grows. This post maps what HIPAA actually requires for AI agent deployments, where the regulatory language is clear, and where it requires legal interpretation.

What HIPAA covers: the three rules that apply to AI agents

HIPAA is not a single rule. It is a framework comprising three rules that operate simultaneously on covered entities and their business associates. All three are relevant to AI agent deployments that handle patient information.

The Privacy Rule

The Privacy Rule governs the use and disclosure of PHI. It defines who may access patient information, for what purposes, and with what safeguards. For AI agents, the Privacy Rule establishes the boundary of what the agent is permitted to do with the information it receives.

• An AI agent that uses PHI to support treatment, payment, or healthcare operations does not require individual patient authorization for those uses, provided appropriate safeguards are in place
• An AI agent that uses PHI for purposes outside treatment, payment, and operations, such as training a commercial AI model on patient data, requires explicit patient authorization
• The minimum necessary standard requires that PHI be limited to the amount needed to accomplish the intended purpose. An AI agent that receives a patient’s full EHR when it only needs medication history to answer a scheduling question may violate this standard
• HHS guidance on business associates is explicit: a business associate may not use PHI in a manner that would violate the Privacy Rule. This applies directly to AI vendors receiving PHI through API calls

The Security Rule

The Security Rule governs the safeguarding of electronic PHI (ePHI). It requires administrative, physical, and technical safeguards. For AI agent deployments, the Security Rule specifies the controls that must be in place before the agent can handle any ePHI.

• Technical safeguards: access controls limiting ePHI access to authorised users and systems, audit controls recording activity on systems containing ePHI, integrity controls to ensure ePHI is not altered improperly, and transmission security (encryption) for ePHI in transit
• Administrative safeguards: workforce training on handling ePHI, security management processes including risk analysis, and contingency planning for system failures affecting ePHI availability
• An AI agent’s API endpoint that receives or transmits ePHI must be covered by transmission security. The agent’s logs that capture ePHI must implement appropriate access controls and audit trails

The Business Associate Agreement: what it means for AI vendors

HHS’s HIPAA cloud computing guidance addresses AI infrastructure directly. When a covered entity or business associate engages a cloud service provider to create, receive, maintain, or transmit ePHI on its behalf, the CSP is a business associate under HIPAA. This is true even if the CSP processes or stores only encrypted ePHI and lacks an encryption key for the data.

This clause has direct implications for AI agent architecture. Every service in the chain that touches ePHI, including the LLM inference endpoint, the vector database storing patient data, the logging service, and any intermediary API gateway, is a potential business associate requiring a BAA.

What AI agents in healthcare are actually being built

IBM documents that AI agents in healthcare are autonomous software systems that use AI to perceive information and reason across data, running tasks that support clinical care, operations, and patient engagement with minimal human prompting. IBM identifies four primary application categories that are seeing production deployment in 2025.

How Microsoft architects HIPAA-compliant healthcare agents

Microsoft’s December 2025 healthcare AI framework provides a detailed governance model for deploying agentic AI in healthcare that is compliant with HIPAA. Microsoft’s approach reflects the controls that enterprise healthcare deployments actually require and provides a useful reference architecture for teams building on any platform.

• Azure Health Data Services: Microsoft’s HIPAA-eligible managed service for ePHI in the cloud, providing FHIR API access with BAA coverage. AI agents that query patient records through Azure Health Data Services operate within Microsoft’s BAA scope
• Entra Agent ID: every AI agent is assigned a unique identity in Azure Active Directory, enabling Conditional Access, role-based access control, and risk-based policies to apply individually per agent. This is the identity layer that allows an audit trail to distinguish which agent accessed which data
• eDiscovery and audit logging: Microsoft documents that eDiscovery logs every agent interaction, giving legal, compliance, and IT teams the transparency required for HIPAA, GDPR, and FDA 21 CFR Part 11
• Communication Compliance: monitors AI-driven interactions for regulatory or ethical violations and unauthorized disclosures. For healthcare agents, this is the automated content scanning layer that flags potentially non-compliant outputs before they reach patients or clinicians
• Conditional Access for agents: Microsoft documents an example policy allowing a prior authorization agent to access claims data only from approved devices and locations during business hours, demonstrating how granular access control applies to agents specifically, not just to human users

Microsoft’s May 2025 agent governance blog also documents the commercial case: a Forrester Total Economic Impact study of Microsoft 365 Copilot found 132% three-year ROI with payback in under one year for a composite healthcare organization, with nine hours saved per Copilot user per month.

HHS’s position on AI in healthcare

In December 2025, HHS published a Request for Information on accelerating the adoption and use of artificial intelligence as part of clinical care. The announcement confirms the federal government’s directional posture: AI adoption in healthcare is actively encouraged, with HIPAA framed as a compliance framework that AI must operate within rather than a barrier to adoption.

• HHS stated directly: ‘As AI-driven technologies continue to advance, HHS is focused on interoperability and ensuring that patient data is safe, secure, and only used as allowed under HIPAA’
• HHS’s published guidance frames data liquidity and patient-provider trust in how data moves as essential prerequisites for AI adoption. The National Coordinator for Health IT framed the RFI as designing for both access and trust simultaneously
• The RFI reflects a recognition that HIPAA’s existing framework was not designed for AI-specific data flows. HHS is actively gathering input on where guidance needs to be updated or clarified for AI use cases

Building HIPAA-compliant AI agents: the technical checklist

Compliance in healthcare AI is not a post-deployment audit. It is an architectural discipline. The following controls must be present before a healthcare AI agent handles ePHI in production.

Data handling

• Minimum necessary principle: scope every prompt to include only the PHI needed for the specific task. An agent answering a scheduling question should not receive the full clinical summary
• No PHI in training data: AI models must not be fine-tuned on real patient data without explicit authorization and appropriate safeguards. This is a common violation in healthcare AI proof-of-concept projects that use real EHR exports for testing
• ePHI retention: HHS guidance confirms that business associates are not required to maintain ePHI beyond the time they provide services. AI inference logs containing ePHI should be deleted on a defined schedule or de-identified before retention

Access and identity

• Per-agent identity: every AI agent in the system should have a distinct service identity, not a shared API key, so that audit logs can attribute each ePHI access to a specific agent and purpose
• Least privilege: agents should be granted access only to the data sources and EHR fields required for their function. A scheduling agent does not need access to clinical notes
• Human-in-the-loop for clinical decisions: AI agents that surface clinical recommendations must present them for clinician review and approval before any action is taken on the patient record. Fully autonomous clinical decision-making without oversight raises both HIPAA and FDA SaMD concerns

Audit and monitoring

• Full inference logging: every ePHI access by an agent, including the query, the data accessed, and the agent identity, must be logged in a tamper-evident, encrypted audit trail
• Breach detection: automated monitoring for anomalous agent behaviour, such as unusually high ePHI access volumes, queries outside business hours, or outputs that contain PHI not present in the source data, should be in place before go-live
• Regular risk assessment: HIPAA requires covered entities to conduct regular risk analyses. AI agent deployments should be included in scope and reassessed when new agents are deployed or existing agents are given access to new data sources

WebOsmotic’s healthcare AI development practice builds the audit logging, access controls, encryption, and BAA compliance architecture into every engagement. For clients in fintech and healthcare, compliance architecture is scoped and documented before development begins.

Frequently asked questions

Does using OpenAI’s API in a healthcare application require a BAA?

Yes, if the prompts sent to the OpenAI API contain protected health information. HHS’s cloud computing guidance is explicit: any cloud service provider that creates, receives, maintains, or transmits ePHI on behalf of a covered entity is a business associate, even if it processes only encrypted ePHI and lacks an encryption key for the data. OpenAI offers a Business Associate Agreement for covered entities and business associates under its enterprise and ChatGPT Team/Enterprise plans. Before sending any patient data to an LLM API, verify that a valid BAA is in place. Sending PHI to a LLM API without a BAA is an impermissible disclosure under the Privacy Rule.

What is the minimum necessary standard and how does it apply to AI agents?

The minimum necessary standard requires that uses and disclosures of PHI be limited to the minimum amount needed to accomplish the intended purpose. For AI agents, this means that prompts and API calls should include only the specific patient data required for the task being performed. An agent processing an appointment reminder does not need the patient’s diagnosis. An agent answering a billing question does not need the clinical notes. Teams should audit the context window sent to the LLM for each agent use case and strip fields not required for the specific function. Violating the minimum necessary standard is a compliance issue even when the overall system is HIPAA-authorized.

Is EHR integration covered by the existing BAA with the EHR vendor?

It depends on the scope of the existing BAA. The BAA with an EHR vendor like Epic or Cerner covers the vendor’s processing of ePHI on behalf of the covered entity. It does not automatically cover a third-party AI agent that accesses the EHR’s FHIR API. If the AI agent is built by a separate vendor or runs on separate infrastructure, that vendor is also a business associate and requires its own BAA. Review the scope of every existing BAA and execute a new BAA for any vendor not already explicitly covered.

Can AI agents make autonomous clinical decisions under HIPAA?

HIPAA does not prohibit AI agents from surfacing clinical information, but it does not address the question of clinical autonomy. That boundary is governed by the FDA’s regulatory framework for clinical decision support software. Under FDA guidance, clinical decision support software that makes patient-specific recommendations to clinicians using patient-specific data and relies on a clinician to review those recommendations before acting is generally exempt from FDA device regulation. Software that automates the clinical decision without clinician review may be subject to FDA oversight as Software as a Medical Device (SaMD). Under HIPAA, fully autonomous clinical actions taken by an AI agent on a patient record must be authorized, logged, and auditable. Human-in-the-loop is both a compliance safeguard and a clinical safety requirement.

Does HIPAA apply to patient-facing healthcare chatbots?

Yes, if the chatbot collects, processes, or references PHI. A patient-facing chatbot that asks about symptoms, schedules appointments, or provides medication reminders is handling PHI. The HIPAA rules apply to the covered entity operating the chatbot. If the chatbot is provided by a third-party vendor, that vendor is a business associate and requires a BAA. HHS’s guidance notes that an app’s facilitation of access to an individual’s own ePHI at the individual’s request does not automatically create a business associate relationship, but apps that create, receive, maintain, or transmit PHI on behalf of the covered entity for healthcare operations purposes clearly do.

How does WebOsmotic approach HIPAA compliance in healthcare AI projects?

WebOsmotic scopes compliance requirements at the architecture stage, before any development begins. For healthcare AI projects, this includes: identifying every component in the agent stack that touches ePHI, confirming or executing BAAs with all relevant vendors, designing the minimum necessary data flows for each agent use case, implementing per-agent identity with audit-trail logging, configuring encryption for ePHI in transit and at rest, and documenting the compliance architecture for the client’s privacy officer or legal team. We work with healthcare providers, health tech companies, and digital health platforms in the US and India.