PHI in Your LLM Context Window: What HIPAA Actually Says

HIPAA Compliant AI Development starts with one question most healthcare engineering teams answer wrong: does a system prompt or RAG context containing patient information count as a HIPAA-regulated disclosure when it is sent to an LLM API? Under 45 CFR 164.514, the answer is yes whenever that information meets the definition of protected health information (PHI).

The Office for Civil Rights has already shown what happens when a business associate agreement is missing: Oregon Health and Science University paid a 2.7 million settlement after PHI was stored on a cloud server with no BAA in place (HHS, 2016).

This guide explains the five BAA rules that govern HIPAA-regulated LLM projects, maps which cloud AI platforms carry a HIPAA-eligible BAA, and breaks down the de-identification methods that let a team skip the BAA requirement entirely.

A large share of the compliance risk sits in the PHI LLM context window itself, since a retrieval-augmented prompt can pull patient data into a request without an engineer ever writing that data down by hand.

When PHI in a HIPAA Compliant AI Development Prompt Becomes a HIPAA Event

The HIPAA Privacy Rule applies to protected health information regardless of the medium in which it exists. A system prompt containing a patient’s name, diagnosis, and medication list is PHI. The transmission of that prompt to an LLM API endpoint is a disclosure of PHI. If no BAA exists between the covered entity and the LLM provider, that disclosure is impermissible.

HHS’s cloud computing guidance establishes the framework directly: when a covered entity or business associate engages a cloud service provider to create, receive, maintain, or transmit ePHI, the CSP is a business associate, even if it processes only encrypted PHI and lacks the encryption key. This principle extends to LLM API providers. An LLM that receives a prompt containing PHI creates, receives, and processes ePHI on behalf of the covered entity.

• Scenario 1: LLM prompt contains PHI and no BAA exists with the provider. This is an impermissible disclosure. It does not matter whether the provider is a major cloud vendor or a small LLM startup. Without a BAA, sending PHI to that provider violates HIPAA’s Privacy Rule
• Scenario 2: LLM prompt contains PHI and a valid BAA exists with the provider. The disclosure is permissible for healthcare operations, treatment, or payment purposes, subject to the minimum necessary standard. The provider’s data handling under the BAA is governed by HIPAA’s business associate provisions
• Scenario 3: LLM prompt contains de-identified data only. De-identified health information is not PHI under HIPAA. There are no HIPAA restrictions on the use, disclosure, or processing of properly de-identified data. The prompt can be sent to any provider without a BAA, as long as the de-identification was performed correctly

HIPAA Compliant AI Development De-Identification: The Two Methods and How They Apply to LLM Inputs

HHS’s de-identification guidance defines two methods for achieving de-identification under the Privacy Rule. Both methods, if correctly applied, produce information that is no longer PHI and therefore not subject to HIPAA’s use and disclosure restrictions.

Method 1: Safe Harbor

Safe Harbor requires the removal of 18 specific identifier categories from the health information. If all 18 identifier types are removed and the covered entity has no actual knowledge that the remaining information could be used to identify an individual, the information is de-identified.

• The 18 identifier types include: names, dates (except year), phone numbers, email addresses, Social Security numbers, medical record numbers, geographic identifiers more specific than state, IP addresses, device identifiers, biometric identifiers, and other unique codes or characteristics

Method 2: Expert Determination

Expert Determination requires a qualified expert to apply statistical or scientific principles to determine that the risk of identifying an individual from the de-identified data is very small. The expert must document the methods and results of the analysis.

Which Cloud AI Platforms Support HIPAA Compliant AI Development

The Safe Handoff Problem: The Most Common Risk in HIPAA Compliant AI Development

The safe handoff problem is the most common HIPAA violation in well-designed LLM healthcare systems: a BAA-covered LLM makes tool calls to external APIs, such as web search or drug databases, that pass PHI to services not covered by any BAA. The original vendor’s BAA covers only its own endpoint.

• The safe handoff principle: any tool call that exits the BAA-covered environment must be preceded by a de-identification step that strips PHI from the query before the external API receives it. Research published in 2026 by University of Texas Medical Branch describes this as the point where a clinician’s PHI-containing query must be transformed into a HIPAA Safe Harbor-compliant version before leaving the protected environment
• Implementation: the LLM’s tool call handler must include a de-identification layer that applies Safe Harbor stripping to any query before it is sent to a non-BAA-covered external endpoint. The clinical context, the diagnostic reasoning, and the information needed can be preserved. The 18 identifier types must be removed
• Logging requirement: every tool call that exits the BAA environment, whether de-identified or not, should be logged with the tool name, the de-identification status, and the timestamp. This creates the audit trail that compliance reviews will examine when assessing whether impermissible disclosures occurred
• Agentic AI systems amplify this risk: autonomous clinical agents that call tools without explicit per-call developer authorization require per-agent identity governance and automatic de-identification before any external tool call

Architectural Requirements for HIPAA Compliant AI Development

Building AI systems that hold up under a HIPAA audit is an infrastructure problem before it is a model problem. Four components carry most of the weight.

• De-identification pipeline: a reusable service that strips Safe Harbor identifiers before patient data reaches any LLM prompt.
• BAA verification layer: a configuration mapping every external endpoint the system calls to its BAA status. A failed check halts the operation and gets logged.
• Immutable audit logging: every inference call touching PHI gets logged with the prompt context, the response, the user identity, the clinical purpose, and the timestamp, in a tamper-evident format.
• Minimum necessary enforcement: the system receives only the PHI fields a given clinical task actually needs, scoped by agent role and task type.

Each of these four pieces maps to a specific failure mode seen in production. Skipping the de-identification pipeline means PHI reaches the model unfiltered.

Skipping the BAA verification layer means a tool call can silently exit the covered environment. Skipping audit logging means a compliance review has no record to examine.

Skipping minimum necessary enforcement means a single compromised prompt can expose more of the chart than the task ever required.

WebOsmotic’s healthcare AI development practice treats HIPAA compliance architecture as a first-class deliverable. For clients building LLM systems on AWS Bedrock, Azure AI, or Vertex AI, the BAA verification, de-identification pipeline, audit logging, and minimum necessary enforcement are all designed and built before the LLM application layer is implemented.

Frequently asked questions

Does HIPAA compliant AI development require a Business Associate Agreement when sending PHI to an LLM API?
Yes, whenever the prompt or context meets the definition of PHI. HHS’s cloud computing guidance treats any provider that creates, receives, maintains, or transmits ePHI on a covered entity’s behalf as a business associate, even when it holds only encrypted data and no decryption key (HHS, 2016). Without a signed BAA, sending PHI to that provider breaks the Privacy Rule.

How do I de-identify patient data for HIPAA compliant AI development before sending it to an LLM?
Apply the Safe Harbor method: remove all 18 identifier categories, including names, geographic detail more specific than state, dates other than the year, contact information, and Social Security or medical record numbers (HHS, 2012). Once every category is stripped, the data is no longer PHI and carries no HIPAA restriction, which is why Safe Harbor is the more common route for HIPAA compliant AI development pipelines compared with Expert Determination.

Is AWS Bedrock suitable for HIPAA compliant AI development on LLM workloads?
Yes. AWS lists Bedrock among its 166-plus HIPAA-eligible services, and customer prompts are not stored or used to train foundation models (AWS, 2025). A BAA must be executed through an Enterprise Agreement or equivalent AWS licensed agreement before any PHI workload goes live.

Is Microsoft Azure OpenAI covered under a HIPAA BAA for HIPAA compliant AI development?
Yes, for text-based inputs. Azure OpenAI sits under Microsoft’s BAA through the Data Protection Addendum, included automatically for customers on qualifying licensing agreements. Computer Vision and Face API are excluded from that default coverage, so confirm your DPA scope before routing PHI through either service.

What is the safe handoff problem in HIPAA compliant AI development?
It is what happens when a BAA-covered LLM calls an external tool, a search API, or a research database that carries no BAA of its own. Each of those calls is a separate disclosure. The fix is a de-identification layer in the tool call handler that strips the 18 Safe Harbor identifiers before any query exits the BAA-covered environment.

How does WebOsmotic build HIPAA compliant AI development for healthcare clients?
Every healthcare engagement starts with scoping which cloud AI services need BAA verification, then building the de-identification pipeline, the BAA verification layer for tool calls, immutable audit logging, and minimum necessary enforcement at the data access layer, before the application layer gets built on top.

Bhavesh Modi

Project Manager – AI