Open-Source LLMs in Healthcare: What Works in HIPAA-Compliant Deployments

Healthcare organizations want the capability of frontier AI models without sending patient data to a third-party API. Open-source LLMs make that possible. But "open-source" is a category, not a recommendation. Some open-weight models perform measurably better on clinical tasks. Some licensing terms are incompatible with commercial healthcare deployments. And the architecture required to satisfy HIPAA with an on-premise open-source LLM is more specific than most engineering teams realize before they start building.

This post answers the questions that healthcare technology leaders are actually asking: which open-source models are most capable on clinical tasks, what their licenses actually permit, how to structure the deployment to satisfy HIPAA's Security Rule, and what the honest tradeoffs are compared to commercial cloud APIs with BAA coverage. It is written for CIOs, CMIOs, and the AI engineering leads who report to them.

The underlying argument is straightforward. A well-selected open-weight model, deployed on-premise with appropriate architecture, can deliver clinical AI capability that is both HIPAA-compliant and cost-effective at scale. The organizations that have gotten this right share a common pattern: they picked the model second, after they built the data boundary first.

Model Options: What Is Actually Available

The open-weight model ecosystem for healthcare applications has three meaningful tiers: general-purpose foundation models with commercial licenses, domain-adapted models fine-tuned on biomedical or clinical text, and task-specific models trained for narrow clinical NLP tasks. Each tier involves different capability-compliance tradeoffs.

General-Purpose Foundation Models

Meta's Llama family is the most widely deployed foundation model in regulated industries. The Llama 2 architecture and training methodology are documented in Touvron et al. (arXiv:2307.09288, 2023). Llama 2 was trained on approximately 2 trillion tokens from publicly available sources, with deliberate exclusion of common personal information datasets. The model card documents data sources in sufficient detail to support regulatory due diligence. The Llama 2 Community License permits commercial use for most organizations but includes a restriction on use by entities with more than 700 million monthly active users, a threshold that no healthcare organization will approach. Subsequent Llama releases retain similar licensing structures.

Mistral 7B (Jiang et al., arXiv:2310.06825, 2023) introduced grouped-query attention and sliding window attention mechanisms that deliver strong performance on reasoning benchmarks at 7 billion parameters, a size that fits comfortably on a single A100 40GB GPU. Mistral AI releases models under the Apache 2.0 license, which permits commercial use without restriction and without requiring disclosure of derivative works. For healthcare organizations with constrained GPU budgets, Mistral 7B and its instruction-tuned variants offer a practical entry point to on-premise inference.

Domain-Adapted Biomedical Models

BioMistral (Labrak et al., arXiv:2402.10373, 2024) adapts Mistral 7B through continued pretraining on PubMed Central Open Access articles and supplementary medical licensing examination data. The authors evaluate BioMistral on ten biomedical NLP benchmarks and report measurable improvement over the Mistral 7B base on clinical NLP tasks including medical question answering and named entity recognition. BioMistral is released under the Apache 2.0 license and is available at a size that supports single-GPU deployment.

Clinical Camel (Toma et al., arXiv:2305.12200, 2023) is a fine-tuned Llama-based model trained on a curated dataset of clinical case discussions and medical dialogue derived from publicly available sources. Clinical Camel targets the conversational clinical reasoning tasks that are relevant to clinical decision support and patient intake automation. The paper documents evaluation on United States Medical Licensing Examination question formats, providing a standardized benchmark for comparison.

MedAlpaca (Han et al., arXiv:2304.08247, 2023) combines instruction fine-tuning on medical flashcard data, clinical notes, and medical dialogue datasets. MedAlpaca's evaluation methodology and benchmark results are documented in the paper and provide another point of comparison for healthcare-specific capability assessment.

What HIPAA Actually Requires of Your Inference Pipeline

HIPAA's Security Rule under 45 CFR Part 164 applies to electronic Protected Health Information: individually identifiable health information transmitted or maintained in electronic form. An LLM inference pipeline that receives patient data, processes it, and returns a clinical output is a system that handles electronic PHI. Every component of that pipeline is subject to the Security Rule's safeguards.

The Administrative Safeguards (45 CFR §164.308) require, among other things, a Security Risk Analysis covering all ePHI-handling systems, including AI systems. The Security Risk Analysis is not a one-time exercise. It must be reviewed and updated in response to environmental or operational changes, including the deployment of new AI capabilities. Healthcare organizations that have deployed cloud AI tools without updating their Security Risk Analysis to reflect those tools are in a common state of non-compliance that most organizations discover during their first AI-focused audit.

The Technical Safeguards (45 CFR §164.312) require access controls, audit controls, integrity controls, and transmission security. For an on-premise LLM pipeline, these translate to: role-based access to the inference endpoint, audit logging of every query with sufficient metadata to identify the user and the data accessed, encryption of model outputs in transit and at rest, and technical controls that prevent unauthorized modification of ePHI during processing. None of these requirements are novel in themselves; the challenge is that AI inference pipelines add a new layer of data processing that must be explicitly mapped to existing controls.

The Physical Safeguards (45 CFR §164.310) require facility access controls and workstation security. For GPU infrastructure hosting an LLM, this means the server room must have controlled physical access with documented access logs, and the GPU servers themselves must be in a physically secured environment. This is standard data center practice, but organizations that use co-location facilities rather than owned data centers must verify that their co-location contract explicitly covers these HIPAA physical safeguard requirements.

"HIPAA's Security Rule does not care whether your AI is open-source or proprietary. It cares whether your controls over electronic PHI are adequate, documented, and operating effectively."

The Architecture That Satisfies HIPAA

A HIPAA-compliant on-premise LLM architecture for healthcare has four mandatory components that go beyond a standard inference deployment: a PHI detection and classification layer at the application boundary, an encrypted audit log with write-once access controls, a network boundary that prevents model inference traffic from reaching the public internet, and an identity management integration that ties every inference request to an authenticated user identity.

PHI Detection at the Application Boundary

Before any query reaches the LLM inference runtime, it must pass through a PHI detection step. This step identifies the presence of HIPAA's 18 identifiers: names, geographic subdivisions smaller than a state, dates more specific than year, phone numbers, fax numbers, email addresses, Social Security numbers, medical record numbers, health plan beneficiary numbers, account numbers, certificate or license numbers, vehicle identifiers, device identifiers, web URLs, IP addresses, biometric identifiers, full-face photographs, and other unique identifying numbers or codes.

PHI detection can be implemented using a combination of rule-based pattern matching (regex patterns for SSN, MRN, phone, email, date formats) and a lightweight NER model for names and locations. Microsoft Presidio (open-source, Apache 2.0) provides a production-ready PHI detection framework that covers all 18 HIPAA identifiers and can be deployed in the same private network as the inference runtime. The detection step logs that PHI was present in the request without logging the PHI itself. This distinction matters for audit log management: the audit log should record that a PHI-containing query was processed by user X at time T, not the contents of the PHI.

The Inference Audit Log

Every inference request involving PHI must generate an audit log entry. The minimum required fields for a HIPAA-compliant inference audit log are: a unique request identifier, the authenticated user identifier of the requesting user, the timestamp of the request and response, the model version used, the data classification label of the input (PHI present or PHI absent), and the application context (which clinical workflow triggered the request). The audit log must not be deletable by any operational user, including administrators. This requires write-once storage with object lock policies, or an equivalent immutability control. Log entries must be encrypted using a key managed outside the inference system itself.

Network Boundary

The inference runtime must operate in a private network subnet with no outbound internet access. This is a stronger requirement than most enterprise software deployments, which typically permit outbound HTTPS for software updates and telemetry. For a HIPAA-compliant inference deployment, the model weights must be downloaded and verified through a separate, audited provisioning process and stored locally. The inference runtime must be configured to disable any telemetry, usage reporting, or model update mechanisms that would create outbound network connections. The network boundary is verifiable: egress firewall rules should log all outbound connection attempts, and any unauthorized outbound connection should trigger an immediate security alert.

Evaluating Models Before You Deploy

Model evaluation for healthcare deployments must happen before PHI enters the inference pipeline. The evaluation process has two distinct phases: capability evaluation on de-identified or synthetic data, and compliance review of the model's license, data card, and intended use documentation.

For capability evaluation, the relevant benchmarks depend on your target clinical tasks. For medical question answering, the USMLE Step 1-3 question formats used in papers such as Clinical Camel (Toma et al., arXiv:2305.12200) provide a standardized comparison point. For clinical NLP tasks including named entity recognition, relation extraction, and clinical note summarization, the BioNLP shared task datasets and the i2b2 challenge datasets provide established evaluation sets. For ICD-10 coding accuracy, the MIMIC-III discharge summaries (de-identified, available through PhysioNet with data use agreement) provide a clinical text benchmark that reflects real deployment conditions.

The MEDFIT-LLM evaluation framework (Rao, Jaggi, Naidu, IEEE RMKMATE 2025, DOI: 10.1109/RMKMATE64574.2025.11042816) addresses evaluation of open-source LLMs in healthcare contexts, including criteria relevant to HIPAA's requirement that covered entities assess reasonably anticipated threats to ePHI. A structured pre-deployment evaluation using this or a comparable framework creates defensible documentation that due diligence was performed before PHI was introduced into the inference pipeline.

For compliance review, examine three documents for each model you consider: the model card, the license terms, and the pretraining data documentation. The model card should document the intended use, limitations, and known failure modes. The license must permit your specific commercial use case and must not impose data sharing or disclosure requirements that conflict with your HIPAA obligations. Pretraining data documentation should provide sufficient detail for you to assess whether the pretraining corpus included data that you would not want associated with your deployment.

"Run your capability evaluation on synthetic data. Run your compliance review on the model card. Only after both pass do you introduce real patient data."

Use Cases That Work Today

Not every clinical use case is equally suited to open-source LLM deployment. The use cases that are working in pilot deployments today share three characteristics: the task is well-defined, the output can be verified by a qualified clinician before it affects patient care, and the PHI surface is bounded and auditable.

Clinical documentation assistance is the leading use case. Structured note generation from voice transcription or free-text summaries, ICD-10 and CPT code suggestion from clinical notes, and automated prior authorization letter drafting from structured patient data all fit this profile. The LLM generates a draft; a qualified clinician reviews and approves before the output enters the record or the billing workflow. The quality bar is high enough that the model's output is useful but low enough that human review catches errors before they cause harm.

Medical literature summarization is a second strong use case for on-premise LLMs. Summarizing PubMed abstracts, synthesizing treatment guidelines for a specific condition, and generating differential diagnosis literature summaries all process public scientific text, not patient PHI. These tasks can run on a smaller model, require less GPU infrastructure, and have a substantially simpler compliance posture because no ePHI is involved.

Administrative automation, including scheduling message drafting, referral letter templates, patient intake form processing, and eligibility verification query formulation, represents a third category. Many of these tasks can be restructured to use de-identified or synthetic inputs. Appointment reminder messages do not require PHI in the prompt; a template-based approach with patient identifiers inserted after LLM generation keeps the LLM processing PHI-free while delivering the administrative benefit.

What does not work reliably today: real-time clinical decision support in acute care settings where LLM latency and hallucination risks are not compatible with patient safety requirements, autonomous prescription or order generation without a verified human review step, and diagnostic imaging interpretation without a specialized vision model fine-tuned on annotated clinical imaging data.

The BAA Question: Open-Source vs. Commercial APIs

Healthcare organizations frequently ask whether they should deploy an open-source model on-premise or use a commercial cloud API with a BAA. The honest answer is that it depends on your use case, your data volume, and your existing infrastructure.

Commercial cloud APIs with BAAs are appropriate when your PHI volume is low, your use case is bounded and well-tested, and you do not have the GPU infrastructure to run a capable model on-premise. The BAA shifts liability for PHI handling to the cloud provider for their portion of the processing, which simplifies your compliance posture for low-volume deployments. The limitations are that you are constrained to the models and model versions covered by the BAA, you cannot fine-tune on your own clinical data without additional contractual terms, and your per-query cost at scale can substantially exceed the cost of an on-premise deployment.

On-premise open-source deployment is appropriate when your PHI volume is high, when you need to fine-tune on proprietary clinical data, when your data residency requirements are strict, or when the total cost of ownership over three years favors owned infrastructure over API spend. The MEDFIT-LLM framework (Rao, Jaggi, Naidu, IEEE RMKMATE 2025) is specifically designed for evaluating open-source models in these contexts. The tradeoff is that you own the entire compliance burden: the controls, the audit, the incident response, and the risk analysis are your responsibility rather than the API provider's.

The inference cost optimization literature is also relevant here. FrugalGPT (Chen, Zaharia, Zou, arXiv:2310.11409, 2023) demonstrates that cascading inference across models of different capability levels reduces cost substantially for mixed-complexity query workloads. This cascading pattern applies to on-premise deployments: running a small model for administrative tasks and a larger model for complex clinical reasoning keeps GPU utilization efficient without overprovisioning for the largest possible task.

What to Ask Before You Start

• Have we updated our Security Risk Analysis to include AI systems? HIPAA's Administrative Safeguards require a current risk analysis covering all ePHI-handling systems. An LLM deployment that processed PHI before the risk analysis was updated is a compliance gap that must be remediated.
• Do we have a PHI detection layer in front of every AI inference endpoint? If a user can send a free-text query to an LLM that contains PHI, and there is no control that detects and manages that PHI, you have an uncontrolled ePHI flow. This is true regardless of whether the model is on-premise or cloud-hosted.
• Who reviews and approves model output before it enters the clinical record? This is the clinical governance question, and it has no technical answer. Establishing a clear policy for which AI-generated outputs require physician review and which can be auto-populated into the record must be decided before deployment, not after an incident.
• What is our incident response procedure for an LLM-related PHI breach? HIPAA's Breach Notification Rule requires notification within 60 days of discovery. AI systems create new breach scenarios: a prompt injection attack that causes the model to output another patient's data, a misconfigured audit log that exposes PHI, a model that memorized and reproduces training data. These scenarios should be in your incident response runbook before the model is in use.
• Have we evaluated the model's performance on our specific clinical tasks before deployment? Published benchmarks on USMLE questions do not predict performance on your specific clinical documentation, coding, or summarization tasks. Domain-specific evaluation on representative de-identified samples is required before a model is used in any workflow that affects patient care or billing.