On-Premise LLM Stack for Regulated Industries: HIPAA, FedRAMP, and SOC 2 Deployment Patterns

Regulated industries are not moving slower on AI. They are moving differently. The cloud-first LLM deployment model that worked for software companies collides with data residency mandates, audit requirements, and breach liability in healthcare, financial services, and government. The organizations deploying AI at real scale in these sectors are building on-premise, and the stack they are converging on is now well-defined enough to document.

This post describes the architecture of an on-premise LLM deployment that satisfies HIPAA Security Rule requirements, FedRAMP authorization pathways, and SOC 2 Type II audit requirements simultaneously. It is not a comprehensive compliance guide. It is a technical architecture post written for the engineering leaders and AI strategists who need to understand what they are actually building before they start building it.

The core insight is that compliance in this context is an architecture constraint, not a process overhead. The organizations that fail at regulated AI deployments treat compliance as something you add after the architecture is designed. The organizations that succeed design the data boundary, the audit log, and the access control model first, then build the AI capability on top of that foundation.

Why Cloud LLM APIs Fail the Compliance Test

The compliance problem with cloud LLM APIs is not primarily a security problem. Major cloud providers run infrastructure that is demonstrably secure. The problem is a data processing agreement problem, an audit problem, and a data residency problem, each of which cloud API deployments handle badly by default.

HIPAA's definition of a Business Associate under 45 CFR §160.103 covers any entity that creates, receives, maintains, or transmits Protected Health Information on behalf of a covered entity. When you send patient data to an external LLM API to answer a clinical question, that provider becomes a Business Associate. Business Associates must sign a Business Associate Agreement that specifies how PHI can be used, stored, and retained. Most general-purpose LLM APIs are not willing to sign BAAs that prohibit model training on submitted data, restrict data retention to the minimum necessary period, and provide audit log access at the query level. Those that do offer BAAs typically restrict PHI processing to specific model versions in specific data center regions, eliminating the cost and capability advantages that made the cloud API attractive in the first place.

FedRAMP's challenge is different. The FedRAMP Moderate authorization baseline, drawn from NIST SP 800-53 Rev. 5, requires continuous monitoring, penetration testing, and audit log retention for defined periods. Commercially available LLM APIs do not expose the infrastructure layer required for FedRAMP's continuous monitoring controls. Federal agencies can use cloud services through a FedRAMP Authorized cloud provider, but the AI capabilities available within those boundaries are constrained to models the provider has included in their authorization boundary, which typically lags the commercial frontier by 12 to 24 months.

SOC 2 Type II audits evaluate the operating effectiveness of controls over an observation period, typically 12 months. A SOC 2 audit of an AI system requires evidence that access to the model and its training data was controlled throughout the period, that changes were logged, and that any data processed was handled per the stated retention and deletion policies. Cloud API deployments cannot provide this evidence because the auditable infrastructure is owned and controlled by the API provider, not the enterprise seeking the audit.

"Compliance in regulated AI is not something you add after the architecture is designed. The data boundary, the audit log, and the access model are the architecture."

The Reference Stack

The on-premise LLM stack for regulated industries has five layers. Each layer maps to a set of compliance controls and can be implemented with specific open-source or commercially licensed components. The layers are: the hardware layer, the inference runtime layer, the data boundary layer, the application and API layer, and the audit and observability layer.

Layer 1: Hardware

On-premise LLM inference requires GPU hardware. The practical minimum for running a capable open-weight model in a regulated setting is a server with at least two NVIDIA A100 80GB GPUs, which supports models up to approximately 30 billion parameters at full precision or 70 billion parameters at 4-bit quantization. For healthcare deployments where the model will handle complex clinical language, a 13B to 70B parameter model range is typical. For financial services document processing, 7B to 13B models fine-tuned on financial text often perform well on specific tasks at substantially lower hardware cost.

The compliance-relevant hardware decisions are around physical access control, hardware attestation, and encryption at rest. Physical access to the GPU servers must be controlled and logged, satisfying HIPAA's Physical Safeguards requirements under 45 CFR §164.310. Hardware-level encryption for data at rest on the NVMe drives hosting model weights and inference caches satisfies the encryption requirements in HIPAA's Technical Safeguards and SOC 2's availability and confidentiality criteria. NVIDIA's Confidential Computing features, available on H100 GPUs, provide hardware attestation that inference is running on unmodified hardware in an unmodified environment, a meaningful control for FedRAMP's supply chain risk management requirements.

Layer 2: Inference Runtime

The inference runtime is the software layer that loads model weights, manages GPU memory, handles concurrent requests, and returns completions. The dominant open-source option is vLLM, which introduced PagedAttention to manage the key-value cache efficiently across concurrent requests (Kwon et al., SOSP 2023, arXiv:2309.06180). vLLM's continuous batching approach substantially improves GPU utilization compared to naive inference implementations, which matters for regulated deployments where overprovisioning hardware to compensate for inefficient inference is expensive.

For compliance purposes, the inference runtime must satisfy two requirements: it must not send data outside the controlled environment, and it must expose a logging interface that captures request metadata without logging PHI in cleartext. vLLM's API server can be configured to log request and response metadata (timestamps, token counts, model version, user identifier from the calling application) without logging the full prompt and completion text, which satisfies the audit logging requirement while limiting PHI exposure in log files. Prompt content logging, where required for specific audit purposes, should be encrypted at the application layer before being written to the audit log store.

Layer 3: Data Boundary

The data boundary layer enforces the compliance perimeter around PHI and other regulated data. It comprises three components: the network boundary, the identity and access management layer, and the data classification pipeline.

The network boundary is implemented as a private subnet with no outbound internet access from the inference servers. All model weight downloads must happen through a separate, audited provisioning path. The inference API endpoint is exposed only within the organization's internal network or through a site-to-site VPN. This satisfies HIPAA's network access controls under 45 CFR §164.312(a)(1) and is a prerequisite for FedRAMP's boundary protection controls.

Identity and access management for the inference layer should be integrated with the organization's existing identity provider (Active Directory, Okta, or equivalent) through a service-to-service authentication mechanism. The calling application authenticates to the inference API with a service account credential scoped to the minimum necessary access. User-level access is enforced at the application layer, not the inference layer, because the inference runtime has no user context. Every service account credential must be rotated on a schedule that satisfies your SOC 2 audit requirements, typically 90 days or less.

Layer 4: Application and API Layer

The application layer is where PHI classification happens before data reaches the inference runtime. No raw PHI should enter the inference API without first passing through a classification and de-identification check. The NIST de-identification standard for health data, described in NIST Special Publication 1800-25, provides a practical framework for this classification step. At minimum, the application layer should detect and flag the presence of 18 HIPAA identifiers before routing any query to the inference runtime.

Whether to de-identify before inference or to infer with PHI present is a deployment decision that depends on the specific use case. Clinical decision support where the model needs to reason about specific patient details requires PHI-present inference, which in turn requires the full data boundary architecture described above. Administrative automation tasks, such as generating appointment reminder templates or coding clinical notes for billing, can often be restructured to use de-identified or synthetic inputs without degrading quality. Structuring your use cases to minimize PHI-present inference reduces compliance surface area and simplifies the audit.

The API interface between the application layer and the inference runtime should be an authenticated REST or gRPC endpoint that enforces a rate limit per service account, logs request metadata to the audit store before forwarding to the inference runtime, and returns a request ID that allows correlation between the audit log and the inference log. This correlation chain is what an auditor will follow when investigating a potential PHI access incident. Building it correctly from the start avoids expensive forensic reconstruction later.

Layer 5: Audit and Observability

HIPAA's Audit Controls standard under 45 CFR §164.312(b) requires covered entities to implement hardware, software, and procedural mechanisms that record and examine activity in information systems that contain or use electronic PHI. For an LLM inference system, this translates to three concrete audit capabilities: who asked what (identity and query metadata), when they asked it (timestamps correlated across application and inference logs), and what the model returned (response metadata, not necessarily content).

The audit log store must be write-once: no authorized user, including administrators, should be able to delete or modify audit log entries. Append-only object storage with object lock policies satisfies this requirement. Log entries should be encrypted using a key managed by your organization's key management infrastructure, not the inference runtime. Retention periods for HIPAA audit logs are a minimum of six years from the date of creation or the date it was last in effect, whichever is later, per 45 CFR §164.530(j).

Observability beyond compliance includes model performance monitoring: response latency percentiles, token throughput, error rates, and output quality metrics from your evaluation harness. These operational metrics live in a separate observability stack from the compliance audit log, but they are equally important for maintaining the reliability commitments that your users depend on.

Compliance Control Mapping

The table below maps the five architectural layers to the primary compliance controls they satisfy across the three major frameworks.

Model Selection for Regulated Deployments

The open-weight model ecosystem has matured considerably, and several model families are well-suited to regulated industry deployments. The key selection criteria differ from commercial AI deployments in important ways: licensing terms must permit commercial use and must not require disclosure of training data you use for fine-tuning, the model card must document pretraining data sources to satisfy due diligence requirements, and the model must achieve acceptable performance on domain-specific benchmarks before PHI is introduced into the deployment.

Meta's Llama family (Touvron et al., arXiv:2307.09288 for Llama 2; the architectural lineage continues through subsequent releases) provides a permissive license for commercial use, well-documented pretraining data exclusions (the model card specifies data sources and removal of common personal information), and a range of sizes from 7B to 70B that covers the hardware cost spectrum from a single A100 to a multi-GPU cluster. For healthcare-specific deployments, domain-adapted variants such as BioMistral (Labrak et al., arXiv:2402.10373), which fine-tuned Mistral 7B on PubMed Central and medical licensing exam data, demonstrate measurable improvement on clinical NLP benchmarks compared to the base model without PHI fine-tuning.

The MEDFIT-LLM evaluation framework (Rao, Jaggi, Naidu, IEEE RMKMATE 2025, DOI: 10.1109/RMKMATE64574.2025.11042816) provides a structured methodology for evaluating open-source LLMs in healthcare contexts, including criteria that map to HIPAA's requirement that covered entities implement policies and procedures to protect PHI from "reasonably anticipated" threats. Running a structured model evaluation before pilot deployment, using this or a comparable framework, is defensible evidence that you performed appropriate due diligence.

The Pilot Deployment Path

The path from architecture design to a compliant on-premise LLM pilot has four phases, each with a defined exit criterion.

Phase one is infrastructure provisioning and security baseline: stand up the GPU hardware, configure the private subnet, implement IAM integration, and demonstrate that the audit log captures a complete request trace. Exit criterion: a security baseline assessment confirming all controls in your compliance matrix are implemented and operational.

Phase two is model deployment and evaluation: load the selected base model, run your domain-specific benchmark suite on de-identified data, and establish the quality baseline before any PHI is introduced. Exit criterion: benchmark results reviewed and accepted by clinical leadership (for healthcare) or the relevant domain authority, with documented sign-off.

Phase three is the limited PHI pilot: introduce a constrained set of real use cases with real PHI, with enhanced audit log review and incident response procedures activated. The PHI surface should be the minimum necessary to validate the clinical or business utility of the deployment. Exit criterion: two to four weeks of pilot operation with no audit log anomalies, PHI access incidents, or material quality failures reviewed by compliance and legal.

Phase four is scaled deployment: expand use cases and user populations, maintain continuous monitoring, and prepare the evidence package for SOC 2 audit or FedRAMP assessment. Exit criterion: evidence package complete and external auditor engaged.

"The organizations that get compliant AI into production fastest are the ones that start with the audit log, not the model."

What to Ask Your CISO and Legal Counsel

• Do we have a signed BAA covering our current AI vendor? If you are sending PHI to any external AI API without a signed BAA, you are in violation of HIPAA's Business Associate requirements under 45 CFR §164.308(b). This needs to be resolved before any other AI architecture decision.
• What is our data residency requirement, and does it permit any cloud processing? Some regulated organizations have contractual or regulatory obligations that prohibit any data processing outside specific geographic boundaries. Confirming this before evaluating hybrid architectures saves significant rework.
• What is our current audit log retention policy, and does it meet the six-year HIPAA minimum? Many organizations discover during their first AI audit that their log retention infrastructure was designed for shorter retention periods. Retrofitting immutable long-term log retention is more expensive than building it correctly from the start.
• Who owns the compliance determination for AI-generated clinical or financial decisions? This is the governance question that often has no clear answer. Establishing a documented decision-making process before the model is in use is vastly easier than establishing one after an incident.
• Have we conducted a Security Risk Analysis that covers our AI systems? HIPAA's Risk Analysis requirement under 45 CFR §164.308(a)(1) requires a thorough assessment of potential risks to PHI. An LLM system processing PHI is a new risk surface that must be assessed and documented.