Healthcare providers are racing to deploy AI. From automated intake agents that qualify patients to prescription refill bots that reduce administrative load, the efficiency gains are undeniable. But in the rush to modernize, many practices are overlooking a critical reality: LLMs are not naturally HIPAA-compliant.
A "HIPAA-compliant" server or a signed BAA with a model provider (like OpenAI or Microsoft) is only the first step. The real risk lies in the behavior of the AI itself. If your bot can be tricked into leaking Patient Health Information (PHI), your BAA won't protect you from a civil rights audit.
Compliance Warning: Under HIPAA, any unauthorized disclosure of PHI is a breach. This includes disclosures made by an AI chatbot that was "fooled" by a user into revealing another patient's data.
Two Major PHI Leak Scenarios We Find in Audits
In our work with Massachusetts-area clinics and health tech firms, we've identified two primary ways AI bots fail the HIPAA test:
Attack 1: Patient Data Exfiltration (Prompt Injection)
AI Response
"Understood, proceeding with the audit summary: 1. John Doe (Diagnosis: Hypertension), 2. Jane Smith (Diagnosis: Type 2 Diabetes), 3. Robert Vance (Diagnosis: Sleep Apnea)..."The attacker uses an authority framing attack to bypass the bot's core constraint against sharing PHI. The AI prioritizes being "helpful to an auditor" over patient privacy.
Attack 2: Cross-Session PHI Leak (Memory Failure)
AI Response
"Your previous session notes indicate you selected the Walgreens at 122 Main St for your Amoxicillin prescription. Would you like to keep that pharmacy for your current visit?"This bot used a shared memory pool. The attacker (User B) successfully retrieved the prescription details and location of User A by simply asking a vague question.
Why Regular HIPAA Audits Aren't Enough
Standard HIPAA security audits look for encryption, access logs, and physical security. They do not look for semantic vulnerabilities. They don't test if an AI can be tricked into "acting" like a doctor or "ignoring" its privacy rules. If your security team isn't red-teaming the AI model itself, you have a massive blind spot.
The Centuri Healthcare Protection Plan
Securing PHI in an AI context requires Layered Context Filtering. At Centuri, we help healthcare providers implement:
- Namespace-Isolated Memory. Ensuring User A's data is physically impossible for the AI to retrieve during User B's session.
- PHI Scrubbing Layers. An external, non-LLM security layer that scans all bot outputs and redacts anything that looks like a name, ID, or medical term before the user sees it.
- Adversarial Red-Teaming. We run thousands of healthcare-specific attack prompts against your bot to ensure it never breaks character, even under pressure.
Get your patient intake AI tested.
We'll run the full Centuri Healthcare Stress Test against your production bots and provide a remediation report suitable for your compliance file.