The Local
Model

Foreword: The Perimeter Is the Product

There is a tension at the center of enterprise AI in 2026. On one side: the genuine capabilities of large language models, which have matured to the point where extraction, classification, summarization, and domain-specific reasoning are not research problems but engineering ones. On the other side: the regulatory, contractual, and operational reality that for healthcare systems, financial institutions, defense contractors, and government agencies, the data those models need to process cannot leave a defined perimeter.

For three years, the default answer to this tension was the Business Associate Agreement. Cloud AI vendors offered BAAs; legal teams reviewed them; CISOs signed them; and the data went to the cloud anyway, protected by contract rather than by architecture. That answer is no longer sufficient. Regulators are examining AI API calls with the same scrutiny they apply to other third-party data sharing. The EU AI Act imposes deployer obligations that contractual pass-throughs cannot fully satisfy. And the practical reality is that a model that has seen your data, even briefly, even within the terms of a BAA, is not the same as a model that has never seen it.

The good news is that the engineering situation has changed decisively. Open-weight models have reached quality levels, and quantization tools have reached efficiency levels, that make local inference genuinely viable for the task distributions that regulated industries need to automate. A 7B model running on a single GPU inside your data center can now perform extraction, classification, and structured summarization at quality levels that would have required a frontier API call two years ago. A 13B fine-tuned model can outperform a general-purpose frontier model on a narrow, well-defined domain task.

This book is a complete operational guide to building that infrastructure. It covers the full stack: selecting open-weight models suited to your task distribution, sizing hardware for the models you need to run, applying quantization to fit large models into constrained budgets, fine-tuning for domain specificity, serving at enterprise throughput, evaluating without cloud dependencies, securing the inference endpoint, building the governance and audit trail that regulators require, and deciding how to route between local and cloud in a hybrid architecture.

The frameworks here come from real regulated-industry deployments and from published research in model efficiency, evaluation methodology, and AI governance. Where a number is cited, the source is named and traceable. Where a pattern is offered, the failure mode it prevents is explained.

The perimeter is not an obstacle to AI deployment in regulated industries. It is the product. Building AI infrastructure that operates entirely within it, with the governance and observability that regulators require, is what turns AI from a liability into a competitive advantage for the organizations that the world most needs to get this right.

The Regulatory Perimeter

The question of whether regulated data can be processed by cloud AI models is not primarily a technical question. It is a legal and regulatory one, and the answers vary by jurisdiction, industry, and data type. Understanding the specific regulatory frameworks that apply to your organization is the prerequisite for any local AI infrastructure decision.

HIPAA, the Health Insurance Portability and Accountability Act, governs Protected Health Information in the United States. The HIPAA Security Rule (45 CFR Part 164 Subpart C) requires covered entities and their business associates to implement technical safeguards that limit access to PHI to authorized personnel and protect PHI from unauthorized use or disclosure. Cloud AI processing of PHI requires a Business Associate Agreement with the AI vendor. A BAA establishes liability allocation and requires the vendor to implement appropriate safeguards, but it does not change the technical fact that PHI is transmitted to and processed by a third party. For organizations in high-risk categories, particularly those subject to federal healthcare contracting requirements, the BAA solution may be insufficient.

GLBA, the Gramm-Leach-Bliley Act, governs nonpublic personal financial information at US financial institutions. The Safeguards Rule requires financial institutions to develop, implement, and maintain a comprehensive information security program. Transmission of NPI to third-party AI vendors is a data-sharing event that must be evaluated under the Safeguards Rule and, where relevant, under the Federal Reserve's SR 11-7 model risk management guidance, which requires documentation of model purpose, validation, and ongoing monitoring.

GDPR and its national implementations govern personal data across the European Union. AI processing of personal data is data processing under GDPR and requires a lawful basis. Cross-border transfers of personal data to US cloud AI vendors require either Standard Contractual Clauses or an applicable adequacy decision. The EU AI Act adds a risk-based regulatory layer on top of GDPR: AI systems used in regulated high-risk categories, including medical devices, critical infrastructure, and financial services credit scoring, face additional conformity, logging, transparency, and human oversight requirements under Article 26.

Defense, Intelligence, and Controlled Technical Data

Defense and intelligence environments operate under classification regimes that prohibit transmission of controlled technical data to any third party without explicit authorization. ITAR, the International Traffic in Arms Regulations, controls the export of defense-related articles and services, including technical data. CMMC, the Cybersecurity Maturity Model Certification, establishes tiered cybersecurity requirements for Defense Industrial Base contractors. EAR, the Export Administration Regulations, controls dual-use items and technology.

For organizations operating under these frameworks, cloud AI processing of controlled technical data is not a risk to be managed through contracts: it is prohibited. The only compliant path for AI processing of ITAR-controlled or CUI data is local inference on hardware that never transmits the data to an external network.

This is not an edge case. It encompasses every defense contractor, every cleared facility, every government agency, and every supplier to those organizations that handles controlled technical data in the course of its work. The addressable market for on-premise AI in defense and intelligence alone is substantial and structurally growing, as AI capabilities expand into more functions of defense operations.

The BAA Gap

Business Associate Agreements have become the default governance mechanism for cloud AI processing of PHI. Most major cloud AI vendors offer BAAs, and many healthcare organizations have signed them, treating the signed BAA as sufficient compliance documentation. This is a reasonable starting position but an incomplete one.

A BAA establishes that the vendor is a business associate that will handle PHI in accordance with HIPAA requirements. It does not establish that the vendor's AI systems are themselves compliant with HIPAA requirements for PHI handling. It does not prevent the vendor from using PHI transmitted during API calls for model training or improvement, unless that restriction is explicitly negotiated. And it does not address the question of whether the AI system qualifies as a covered entity's agent under other applicable regulations.

The more fundamental issue is architectural. Even a well-negotiated BAA does not change the fact that PHI travels over the internet to a third-party data center, is processed by hardware and software that the covered entity does not control, and may be logged, cached, or retained in ways that the covered entity cannot directly verify. For organizations where the risk calculus does not tolerate this, local inference is not a preference but a structural requirement.

Building the Classification Gate

The first operational requirement for regulated AI deployment is a data classification gate: a system that assesses the sensitivity of any input before it is processed by an AI system and enforces the routing policy accordingly. Without a classification gate, regulated and unregulated data flow to the same AI endpoints without distinction, creating compliance risk that accumulates invisibly until a review or incident makes it visible.

A classification gate operates at the input layer. It examines incoming data against a set of sensitivity rules, pattern libraries, and metadata labels that correspond to the regulatory frameworks applicable to your organization. It assigns a sensitivity classification, and it enforces a routing policy that either permits local AI processing, blocks AI processing entirely, or routes to a specific approved endpoint based on the classification result.

The gate is not a replacement for data governance. It is a programmatic enforcement point that complements human classification processes. It catches cases where data classification labels were not applied at the source, where data was aggregated from multiple sources with different classification levels, or where AI processing was initiated without a human review of data sensitivity.

The Compliance Architecture

A compliant local AI architecture for a regulated industry organization has four structural properties. First, the inference endpoint is inside the network perimeter and reachable only from authorized internal systems. Second, every query to the inference endpoint is logged with the caller identity, data classification of the input, model identifier, and timestamp. Third, the data classification gate enforces routing policy before every AI invocation. Fourth, the governance framework documents the model's purpose, validation status, and ongoing monitoring, satisfying the requirements of NIST AI RMF and applicable sector-specific guidance.

These four properties are not aspirational: they are the minimum viable compliance posture for regulated AI deployment. Organizations that meet this bar can demonstrate to regulators, auditors, and boards that their AI systems operate within the same governance framework as their other critical systems, rather than as ungoverned shadow IT.

The investment required to build this architecture is real but bounded. The chapters that follow address each component in detail: selecting the right model (Chapter 2), sizing the hardware (Chapter 3), applying quantization (Chapter 4), fine-tuning for the domain (Chapter 5), building the inference serving layer (Chapter 6), running evaluation (Chapter 7), securing the endpoint (Chapter 8), building the audit trail (Chapter 9), and deciding how to route between local and cloud (Chapter 10).

What Open-Weight Means

Open-weight models are models whose trained weight parameters have been released publicly, allowing organizations to download, run, and modify them without API access. The distinction matters for regulated industries: open-weight models can be deployed on infrastructure the organization controls, without any data leaving the perimeter.

Open weights do not imply open training data, open source code, or freedom from all licensing restrictions. The weights are released; the training pipeline, the training data, and the associated systems may or may not be disclosed. For regulated industries, this matters for two reasons. First, licensing terms govern commercial use, redistribution, and derivative works. Second, training data provenance affects whether the model may have been trained on data with privacy, copyright, or sector-specific compliance implications. Both require separate assessment before enterprise deployment.

The major open-weight model families as of 2026 include Llama 2 from Meta AI (Touvron et al., arXiv:2307.09288), Mistral from Mistral AI (Jiang et al., arXiv:2310.06825), and Phi-3 from Microsoft Research (arXiv:2404.14219), among others. Each occupies a distinct position on the quality-efficiency frontier and is suited to different deployment contexts.

Llama 2

Llama 2, introduced in Touvron et al. (arXiv:2307.09288), is available in 7B, 13B, and 70B parameter configurations, with instruction-tuned variants (Llama 2-Chat) optimized for conversational use. The 7B and 13B models are suitable for extraction, classification, and structured summarization on most enterprise task distributions. The 70B model approaches frontier quality on complex reasoning tasks at the cost of significantly higher hardware requirements.

Llama 2 is released under a custom license that permits commercial use with a restriction: organizations with more than 700 million monthly active users must request a license from Meta. For most enterprise deployments, this restriction is not binding. Organizations should verify the current license terms before deployment and consult legal counsel on any applicable restrictions.

Mistral 7B

Mistral 7B (Jiang et al., arXiv:2310.06825) demonstrates that parameter count is not the only determinant of model capability. Mistral 7B employs grouped-query attention and sliding window attention to achieve performance competitive with larger models on standard benchmarks. It is released under the Apache 2.0 license, which permits commercial use, modification, and redistribution without usage restrictions. For enterprise deployment, the Apache 2.0 license is the most permissive available in the open-weight landscape and simplifies legal review significantly.

Phi-3

Phi-3 (Microsoft Research, arXiv:2404.14219) represents a different approach: a compact model trained on carefully curated data rather than raw web scale. The Phi-3-mini (3.8B parameters) achieves performance competitive with models several times its size on reasoning and coding benchmarks. Microsoft Research's technical report describes the model as capable of running locally on a phone, which implies it can run on almost any enterprise GPU hardware.

Phi-3 is released under the MIT license, the most permissive available. For regulated industries where legal review of model licenses is a bottleneck, Phi-3 and Mistral 7B (Apache 2.0) offer the simplest path to deployment approval.

Matching Models to Task Distributions

The single most important model selection decision is matching parameter count and model family to the task distribution your deployment will serve. The right model for extraction from structured financial reports is not the same as the right model for summarizing complex clinical case notes or drafting compliance analysis.

For extraction and classification tasks, where the model is identifying and labeling specific fields, entities, or categories within a document, a 7B model fine-tuned on domain examples typically performs well. The task is bounded, the output is structured, and the model does not need deep reasoning capability to succeed. Phi-3-mini (3.8B) or Mistral 7B are strong starting points for these task types.

For summarization and synthesis tasks, where the model must compress, interpret, and restructure content, a 13B model offers meaningfully better performance than a 7B model on most benchmarks, particularly for content that requires understanding relationships between entities or sections. Llama 2-13B or Mistral-7B with domain fine-tuning are appropriate for this tier.

For complex reasoning tasks, code generation, or tasks that require the model to follow multi-step instructions reliably, a 34B or 70B model is usually required. These models have higher hardware requirements but can be made feasible with quantization, which Chapter 4 addresses in detail.

Instruction Tuning and Chat Variants

Most open-weight model families release both a base model (trained on next-token prediction) and an instruction-tuned variant (trained with RLHF or similar techniques to follow instructions). For enterprise deployment, instruction-tuned variants are almost always the right choice. They follow structured prompts more reliably, refuse unsafe requests more consistently, and produce output in the requested format more often than base models do on the same inputs.

The tradeoff is that instruction-tuned models are harder to fine-tune further without degrading instruction-following behavior. If your deployment requires significant domain adaptation via fine-tuning, consult the fine-tuning literature for your target model family before committing to an instruction-tuned base. LoRA and QLoRA (Chapter 5) mitigate this tradeoff by adapting only low-rank matrices rather than the full model.

The VRAM Constraint

Graphics Processing Units are the infrastructure on which local AI inference runs. The primary constraint is VRAM: the high-bandwidth memory on the GPU die that must hold the model weights during inference. If the model does not fit in VRAM, it either does not run or runs at drastically reduced throughput by paging weights through system memory. Understanding VRAM requirements is the prerequisite for every hardware decision in local AI infrastructure.

The VRAM requirement for a model is approximately the parameter count multiplied by the bytes per parameter at the chosen precision, plus overhead for the KV cache (which grows with batch size and context length) and framework overhead. At FP16 (2 bytes per parameter), a 7B model requires approximately 14 GB for weights alone, plus overhead: roughly 16 GB total at modest batch sizes. At INT4 (0.5 bytes per parameter), the same 7B model requires approximately 3.5 GB for weights, fitting easily on consumer GPU hardware.

These are mathematical relationships derived from parameter count and precision, not estimates. They allow hardware requirements to be calculated before purchasing, given knowledge of the model family and the precision level the deployment will use. The calculation for a 70B model at INT4: 70 billion parameters at 0.5 bytes per parameter yields 35 GB for weights. Add 4-5 GB for KV cache at typical batch sizes and the total is approximately 39-40 GB, fitting on a single A100 80 GB GPU.

GPU Selection

The NVIDIA data center GPU product line is the dominant platform for enterprise local AI inference. The A100 in 80 GB configuration is the current reference GPU for enterprise deployments: it provides 80 GB of HBM2e memory, sufficient for a 70B model at INT4 or a 34B model at FP16, and has broad software support across all major inference frameworks. The H100, NVIDIA's successor to the A100, offers higher throughput on the same 80 GB memory footprint and includes hardware acceleration for FP8 precision, though FP8 inference support in open-source frameworks is still maturing.

For organizations deploying 7B and 13B models at moderate concurrency, the A6000 (48 GB) and RTX 6000 Ada (48 GB) offer lower cost per GPU than the A100 while providing sufficient VRAM for these model sizes. The consumer RTX 4090 (24 GB) is viable for development, testing, and low-concurrency deployment of quantized 13B models, but does not have the reliability guarantees of data center hardware for enterprise pilot deployments.

Multi-GPU Configurations

When a model does not fit on a single GPU or when concurrency requirements exceed what a single GPU can serve, multi-GPU configurations extend the available VRAM and throughput. The two primary parallelism strategies are tensor parallelism and pipeline parallelism, and they address different constraints.

Tensor parallelism splits individual model layers across multiple GPUs. Each GPU holds a portion of each layer's weight matrix and computes a portion of each forward pass. The results are combined via inter-GPU communication at each layer. Tensor parallelism requires high-bandwidth inter-GPU connections, typically NVLink in NVIDIA's architecture. With NVLink, the communication overhead is low enough that tensor parallelism across 2 or 4 GPUs adds modest latency penalty while multiplying available VRAM. Without NVLink, PCIe-connected multi-GPU tensor parallelism incurs significant latency overhead and should be used only for throughput-sensitive, latency-tolerant workloads.

Pipeline parallelism assigns different model layers to different GPUs. GPU 0 processes the first N layers, GPU 1 processes the next N layers, and so on. This is more memory-efficient than tensor parallelism but introduces pipeline bubbles, idle time while one GPU waits for another to complete its stage. Pipeline parallelism is appropriate for high-throughput batch workloads where minimizing latency is secondary to maximizing total throughput.

For most enterprise regulated-industry deployments, a single A100 80 GB with INT4 quantization is sufficient to serve a 70B model and provides a simpler operational posture than multi-GPU configurations. Multi-GPU becomes necessary when serving 70B models at FP16 precision, when concurrency requirements demand more throughput than a single GPU can provide, or when serving multiple models simultaneously on shared infrastructure.

Storage Architecture

Model weights are large files: a 7B model at FP16 is approximately 14 GB, a 70B model at FP16 is approximately 140 GB. Storage architecture must account for both the initial model download and the serving path, where weights are read from disk into GPU memory at model load time.

For inference serving, NVMe SSDs are strongly preferred over SATA SSDs or spinning disks. The difference in sequential read throughput (NVMe: 5-7 GB/s versus SATA SSD: 500-550 MB/s) translates directly to model load time, which matters for services that spin up model instances dynamically. For a 70B model at INT4 (approximately 35 GB), loading from NVMe takes roughly 5 seconds versus 60+ seconds from a SATA SSD.

Model files should be stored on storage that is separate from the operating system and framework installation, enabling model updates without system maintenance windows. For organizations managing multiple model versions simultaneously, a model registry with content-addressed storage (where the model file name is derived from its hash) ensures that updates do not silently overwrite serving models and that rollbacks are atomic.

Precision Levels and Their Tradeoffs

Neural network weights are floating point numbers. Training typically uses FP32 (4 bytes per parameter) or BF16 (2 bytes per parameter, a format with better training stability than FP16). Inference can use any precision level, with lower precision providing smaller memory footprints at the cost of representational fidelity. The relationship between precision and quality is not linear: the first reduction from FP32 to FP16 loses almost nothing; the reduction from FP16 to INT4 loses more but often less than expected for well-calibrated models on narrow task distributions.

FP16 (half-precision, 2 bytes per parameter) is the baseline for on-premise inference. Most pre-trained open-weight models are released in FP16 format. FP16 inference requires a GPU with half-precision support, which includes all NVIDIA GPUs from the Pascal generation onward. Quality loss from FP32 is negligible on standard benchmarks.

INT8 (8-bit integer, 1 byte per parameter) reduces the weight footprint by half compared to FP16. INT8 inference requires dequantization during the forward pass, adding compute overhead, but modern inference frameworks (bitsandbytes, llama.cpp) implement efficient INT8 kernels that mitigate this. Quality impact is minimal for most tasks: perplexity increase from FP16 to INT8 is typically small, and task-specific performance is usually within noise of FP16 for extraction, classification, and summarization.

INT4 (4-bit integer, 0.5 bytes per parameter) provides the maximum memory reduction in common use. A 13B model requires approximately 7-8 GB at INT4, fitting on a single RTX 4090 (24 GB) or A6000 (48 GB). A 70B model at INT4 requires approximately 35-40 GB, fitting on a single A100 80 GB. The quality impact of INT4 is larger than INT8 but typically acceptable for well-defined enterprise tasks, particularly when using more sophisticated quantization schemes like Q4_K_M (which applies different quantization to different parts of the model based on sensitivity analysis).

The GGUF Format and llama.cpp

The GGUF format (a successor to the earlier GGML format) is a file format designed for efficient storage and loading of quantized model weights. It supports multiple quantization levels within a single file format, enabling a consistent toolchain across precision levels. llama.cpp, a C++ inference implementation by Georgi Gerganov, supports GGUF natively and runs on CPU and GPU with no CUDA dependency required for CPU-only deployments.

The practical implication is that GGUF and llama.cpp enable local AI inference on hardware that has no GPU at all, with quality models, using only CPU resources. CPU inference is dramatically slower than GPU inference (10x to 100x depending on the model and hardware), but it is viable for low-concurrency use cases, development environments, and deployments where GPU hardware cannot be procured on the required timeline.

QLoRA: Fine-Tuning in Quantized Precision

QLoRA (Quantized Low-Rank Adaptation), introduced in Dettmers et al. (arXiv:2305.14314), combines two techniques: 4-bit quantization of the base model weights (which remain frozen during training) with full-precision LoRA adapter training. This combination enables fine-tuning of models that would not otherwise fit in the VRAM available for training.

The key insight of QLoRA is that the base model weights do not need to be in high precision during fine-tuning, only during inference. By quantizing the base model to 4-bit and adding high-precision LoRA adapters on top, QLoRA achieves similar fine-tuning quality to full-precision LoRA while requiring a fraction of the VRAM. The Dettmers et al. paper demonstrates fine-tuning a 65B model on a single 48 GB GPU, a result that was previously infeasible without multiple high-end GPUs.

For regulated industries, QLoRA's practical impact is that domain-specific fine-tuning of 7B to 70B models is now feasible on hardware that organizations can own and operate on-premise without cloud GPU rental. A single A100 80 GB can run QLoRA fine-tuning of a 70B model. A single RTX 4090 can run QLoRA fine-tuning of a 7B or 13B model. This puts fine-tuning within reach of any regulated organization with a modest GPU infrastructure investment.

Evaluating Quantization Quality

Perplexity is the standard benchmark metric for quantization quality assessment. It measures how well the model predicts a held-out text corpus; lower perplexity indicates a model that assigns higher probability to the correct tokens. Perplexity increases monotonically as precision decreases, but the magnitude of the increase varies by model, quantization scheme, and dataset.

For enterprise deployments, perplexity is a diagnostic metric, not a deployment criterion. The deployment criterion is task performance: does the quantized model meet the quality threshold for the specific regulated task it will perform? A model with a slightly elevated perplexity compared to its FP16 baseline may still perform identically on a narrow extraction task. Conversely, a model with acceptable perplexity may fail on a domain-specific reasoning task that requires the full precision of its larger weights.

The correct evaluation process is: (1) define the quality threshold for each use case before testing; (2) run the quantized model on a representative sample of actual task data; (3) compare output quality against the threshold; (4) approve the quantization level if it meets the threshold, or select a higher precision if it does not. This process should be documented as part of model governance and repeated whenever the quantization level, model version, or quantization toolkit is updated.

Why General Models Fall Short

Open-weight models trained on general web data are capable generalists. They have seen enough text to understand language structure, follow instructions, and perform many tasks acceptably. But regulated industries operate on specialized vocabularies, document structures, and reasoning patterns that general pre-training data under-represents. A general model asked to extract specific fields from a clinical note, classify a financial instrument type, or identify regulatory risk in a contract will perform below the quality threshold of a model that has seen thousands of examples of those exact tasks.

Fine-tuning bridges this gap. By training the model on labeled examples from your actual task distribution, you adapt its internal representations to the vocabulary, formats, and patterns of your domain. The result is a model that performs significantly better on your specific tasks than the general-purpose base model does, without requiring more capable (and more compute-intensive) model families.

The alternative to fine-tuning is prompt engineering: crafting system prompts and few-shot examples that guide the general model toward acceptable performance. Prompt engineering is faster and requires no training infrastructure, and it is the right starting point for any new use case. But for regulated deployments where consistent, high-quality performance across thousands of daily invocations is required, fine-tuning typically provides a more robust quality floor than prompt engineering alone.

LoRA: Low-Rank Adaptation

LoRA, introduced in Hu et al. (arXiv:2106.09685), is the dominant fine-tuning technique in the open-weight model ecosystem. The core insight is that the weight updates learned during fine-tuning can be approximated by low-rank matrices, even when the base model weights are high-rank. Rather than updating all model parameters during fine-tuning, LoRA freezes the base model weights and trains two small matrices A and B whose product AB approximates the weight update for each adapted layer. The base weight matrix W is augmented with the product: the effective weight during inference is W plus AB.

The rank r of the LoRA matrices is the primary hyperparameter. Low rank (r=4 or r=8) provides a small number of trainable parameters and is appropriate for simple adaptation tasks. Higher rank (r=16 to r=64) provides more expressive adapters at the cost of more parameters. For most enterprise domain adaptation tasks, r=16 with alpha=32 (which scales the LoRA output) is a reasonable starting point.

The parameter efficiency of LoRA is substantial. For a 7B model, full fine-tuning updates approximately 7 billion parameters. A LoRA adapter with rank 16 applied to the attention matrices updates approximately 2-4 million parameters, a reduction of over 1,000x. This translates directly into reduced training time, reduced memory requirement, and reduced data requirement.

Data Requirements and Quality

The quality of fine-tuning data is more important than the quantity. A few hundred high-quality labeled examples from your actual task distribution will produce more useful domain adaptation than thousands of noisy, inconsistent, or misaligned examples. Quality in fine-tuning data means: the input matches the format and content of actual production inputs; the output is the correct, high-quality answer that you want the model to produce; the labeling is consistent across examples; and the examples cover the full range of variation in your task distribution, not just the easy cases.

Constructing fine-tuning data for regulated industries requires involving domain experts. For healthcare fine-tuning, clinical experts must validate that the labeled outputs represent correct clinical reasoning and appropriate clinical language. For financial fine-tuning, subject matter experts and compliance reviewers must validate that the labeled outputs are accurate and compliant. This expert involvement takes time but is not optional: a model fine-tuned on incorrect labels will learn to produce incorrect outputs reliably.

Fine-tuning data is itself regulated data. A training set built from patient notes is PHI. A training set built from loan applications is NPI. The pipeline that constructs, stores, and processes fine-tuning data is subject to the same data governance controls as other regulated data processing in your organization: access controls, encryption at rest and in transit, audit logging, and retention policies. This requirement is frequently overlooked in AI pilot deployments that focus on the model training step without assessing the governance obligations of the training data pipeline.

The Fine-Tuning Pipeline

A governed fine-tuning pipeline for a regulated industry deployment has five stages. First, data preparation: collecting, anonymizing (where required), formatting, and labeling the training examples. Second, baseline evaluation: running the base model on a held-out test set to establish the pre-fine-tuning quality baseline. Third, training: running LoRA or QLoRA fine-tuning on the training set with validation loss monitoring. Fourth, fine-tuned model evaluation: running the fine-tuned model on the same held-out test set and comparing against the baseline. Fifth, governance documentation: recording the dataset version, training parameters, evaluation results, and approver sign-off in the model registry.

Each stage has governance checkpoints. The training dataset must have a version identifier and an access log showing who constructed it and when. The training run must have logged hyperparameters and loss curves. The evaluation results must be reviewed by a qualified domain expert and a compliance reviewer before the fine-tuned model is approved for enterprise pilot deployment. This process is heavier than ad-hoc fine-tuning but produces a model whose quality and compliance posture can be documented and audited.

Inference Frameworks

Three inference frameworks dominate the open-source local AI serving landscape in 2026: llama.cpp, vLLM, and Ollama. Each addresses a different set of requirements, and choosing the right one depends on the deployment context, concurrency requirements, and the operational sophistication of the team managing the infrastructure.

llama.cpp is a C++ inference engine that supports GGUF format models and runs on both CPU and GPU. It requires minimal dependencies, compiles on Linux, macOS, and Windows, and provides a REST server via the llama-server binary. llama.cpp is the right choice for single-GPU deployments, development and testing environments, CPU-only deployments, and any situation where operational simplicity is a priority. Its GPU implementation uses CUDA (NVIDIA) or Metal (Apple Silicon) and can offload some or all layers to GPU while running the remainder on CPU, enabling model sizes that exceed GPU VRAM at the cost of reduced throughput.

vLLM is a Python-based inference server built around PagedAttention, a memory management algorithm for the KV cache that treats KV cache allocation like virtual memory paging. PagedAttention allows vLLM to serve more concurrent requests on a given GPU than naive KV cache implementations, because it avoids over-allocating KV cache memory per request. vLLM exposes an OpenAI-compatible REST API and supports tensor parallelism across multiple GPUs. It is the right choice for high-concurrency production serving on GPU hardware.

Ollama is a user-friendly application that wraps llama.cpp with a REST API, a model library, and a command-line interface for pulling and running models. It is optimized for developer experience: pulling a model and starting inference requires a single command. Ollama is appropriate for developer access, prototyping, and small team deployments. For enterprise-scale serving with concurrency requirements, vLLM or a direct llama.cpp deployment provides more control over serving parameters.

The OpenAI-Compatible API

One of the most important practical decisions in local inference serving is whether to expose an API that is compatible with the OpenAI API surface. Both vLLM and llama-server implement OpenAI-compatible endpoints for chat completions, completions, and embeddings. Applications that use the OpenAI Python SDK or any other client that targets the OpenAI API surface can switch from cloud to local inference by changing the base URL and API key in their configuration, with no code changes required.

This compatibility is not cosmetic. Enterprise AI deployments typically involve tens or hundreds of internal applications, scripts, and workflows that call the AI API. If migrating from cloud to local requires code changes in every caller, the migration cost is high and the organizational friction is high. If migration requires only a configuration change (setting OPENAI_BASE_URL to the local endpoint), the migration is low-friction and reversible.

Batching and Throughput

GPU utilization during inference depends on batching: processing multiple requests simultaneously in a single forward pass. A GPU that serves one request at a time runs its compute units at a small fraction of their capacity. A GPU that batches 8 or 16 requests together approaches the utilization levels that justify its cost.

Static batching waits until a batch is full before processing it. This maximizes GPU utilization but introduces queuing latency: requests that arrive when a batch is nearly full must wait for the next batch. Static batching is appropriate for asynchronous workloads where latency is not a constraint: nightly batch processing of documents, daily analytics runs, or any use case where the caller submits requests and retrieves results later.

Continuous batching (also called iteration-level scheduling) processes requests as they arrive, dynamically adding new requests to the batch as existing requests complete. vLLM's implementation of continuous batching is particularly efficient because PagedAttention allows it to share KV cache memory across requests in the same batch rather than allocating fixed KV cache per request. For enterprise serving workloads with mixed request sizes and real-time latency requirements, continuous batching provides better performance than static batching.

Health, Monitoring, and Failover

An enterprise inference serving layer requires the same operational instrumentation as any other critical service: health endpoints, latency metrics, error rate monitoring, and a failover plan. The inference server is a single point of failure for every application that depends on it. In regulated industries where AI is integrated into workflows with clinical, financial, or compliance consequences, an inference server outage is not a minor inconvenience: it interrupts regulated workflows and may leave staff without tools they rely on.

Both vLLM and llama-server expose health check endpoints that load balancers and monitoring systems can poll. These should be wired into the same monitoring infrastructure as other critical services. Latency metrics should be collected per request (not just aggregated averages) to detect tail latency issues that averages obscure. A p99 latency alarm is more useful than an average latency alarm for detecting the slow requests that affect real user workflows.

For regulated industries, the failover plan for inference infrastructure deserves explicit design. Options include: a second GPU instance on standby (active-passive failover), a load balancer distributing across multiple GPU instances (active-active), a circuit breaker that falls back to a cloud API endpoint for non-sensitive tasks when local inference is unavailable, and degraded-mode operation where regulated tasks queue until local inference is restored rather than routing to cloud. The right design depends on the data sensitivity, the latency tolerance, and the regulatory risk of the fallback options.

The Evaluation Problem in Air-Gapped Environments

Enterprise AI evaluation has developed largely in cloud contexts, where the most capable available models serve as judges and evaluators. A typical cloud evaluation pipeline submits model outputs to a frontier API, which assesses quality against a rubric and returns scores. This approach leverages the best available evaluation capability but is structurally incompatible with air-gapped regulated deployments: the outputs being evaluated are regulated data that cannot be transmitted to any external API.

The solution is local LLM-as-judge evaluation: using a larger or differently fine-tuned local model as the judge. This approach is not a fallback; it is a first-class evaluation methodology with a strong research foundation. Zheng et al. (arXiv:2306.05685) introduced the MT-Bench and Chatbot Arena methodology for LLM-as-judge evaluation, demonstrating that large language models can assess output quality against a rubric with high agreement with human expert evaluations on most open-ended tasks. The key condition is that the judge model must be strong enough relative to the evaluated model to provide meaningful discrimination.

For regulated industries, local LLM-as-judge evaluation has an additional advantage: the judge can be fine-tuned on domain-specific evaluation rubrics. A judge model fine-tuned on examples of correct and incorrect clinical extraction outputs, reviewed by clinical experts, produces more domain-relevant quality signals than a general-purpose cloud judge that has no domain calibration.

RAGAS for Local Evaluation

RAGAS (Retrieval Augmented Generation Assessment), introduced in Es et al. (arXiv:2309.15217), provides a framework for evaluating the output quality of retrieval-augmented generation systems along three dimensions: faithfulness (does the response faithfully represent the retrieved context?), answer relevance (does the response correctly address the query?), and context precision (does the retrieved context contain the information needed to answer the query?). RAGAS was designed to run against any LLM-as-judge, including local models.

For regulated industries deploying RAG systems over internal document repositories (clinical guidelines, compliance manuals, regulatory filings), RAGAS-style evaluation provides a structured quality signal that can be automated into the CI pipeline. Each metric is computed by prompting the judge model with the input query, the retrieved context, and the generated response, and asking it to assess one dimension of quality. The result is a numeric score that can be tracked over time and used as a gate on model updates.

Building the Evaluation Stack

An evaluation stack for a regulated local AI deployment has four components: a version-controlled test dataset, a judge model configuration, a scoring script, and a CI gate. Each component has governance requirements that parallel those of the training pipeline.

The test dataset is a held-out set of input-output pairs that represents the full range of variation in the production task distribution. It must include easy examples (where even a weak model succeeds), hard examples (where quality differences between model versions are detectable), and edge cases (unusual inputs that probe the model's failure modes). The test dataset must be version-controlled with the same rigor as source code: every update must be reviewed, labeled with the update reason, and tagged with the model version it was added for. A test dataset that drifts away from the current production distribution provides misleading quality signals.

The judge model configuration specifies which local model serves as the judge, the rubric it is prompted with, and the aggregation method for multi-criterion evaluations. For regulated industries, the rubric should be reviewed and approved by domain experts, not authored by AI engineers alone. A clinical extraction rubric that does not reflect actual clinical documentation standards will produce evaluation scores that do not predict clinical utility.

The scoring script combines the test dataset, the evaluated model, and the judge model into a repeatable, automated quality assessment. It should report individual test case results in addition to aggregate metrics, so that reviewers can inspect which cases failed and understand whether the failures represent systematic weaknesses or isolated edge cases. The script should exit non-zero when the overall quality score falls below the defined threshold, enabling CI systems to enforce the quality gate automatically.

MEDFIT-LLM: Domain-Specific Evaluation

The MEDFIT-LLM evaluation methodology (Rao, Jaggi, Naidu, IEEE RMKMATE 2025, DOI: 10.1109/RMKMATE64574.2025.11042816) demonstrates that comparing fine-tuned domain-specific models against general-purpose frontier models on standardized domain task suites reveals systematic performance differences that aggregate benchmarks obscure. The core finding: task-specific evaluation on the actual deployment task distribution is a more reliable predictor of deployed performance than general capability benchmarks.

Applied to regulated industry local AI deployments, the MEDFIT-LLM methodology suggests: construct evaluation suites on your actual task distribution; compare local fine-tuned models against general-purpose baselines on those tasks specifically; and use the results to make deployment decisions rather than relying on generic benchmark rankings. A 7B model fine-tuned on clinical extraction may outperform a general-purpose frontier model on the specific extraction task you are deploying for, even if the frontier model outperforms it on every standard benchmark.

Network Isolation

The first and most important security control for a local inference endpoint is network isolation. The inference server should be bound to a private network interface and accessible only from authorized internal systems. In practice this means binding to localhost (127.0.0.1) for single-host deployments or to a private VLAN address for deployments that serve multiple internal clients. It means firewall rules that prevent any external network path to the inference port. And it means treating any deviation from these defaults, such as enabling the inference endpoint for external access for a demo or a remote access session, as a security exception that requires explicit approval and a time limit.

The network isolation of the inference endpoint is what separates local AI from cloud AI in the regulatory sense. Data that is processed by an endpoint inside the network perimeter, reachable only from internal systems, and never transmitted to an external host satisfies the data residency requirement that cloud AI cannot satisfy. Compromising this by making the endpoint externally accessible, even temporarily, collapses the regulatory distinction that justified local deployment in the first place.

Where multiple teams need access to the inference endpoint, a reverse proxy with authentication is the right architecture. The inference server binds to localhost and is accessible only via the proxy. The proxy enforces authentication, rate limits, and access logging before forwarding requests to the model. This centralizes access control and logging while allowing the inference server itself to focus on serving model requests efficiently.

Prompt Injection

OWASP LLM Top 10 (v1.1, 2023) lists prompt injection as the top risk category for LLM-based applications. A prompt injection attack occurs when user-supplied text incorporated into a model prompt contains instructions that override or redirect the model's behavior. A local model is equally vulnerable to prompt injection as a cloud model: the vulnerability is in how the application constructs prompts, not in how the model is accessed.

For regulated industries, the consequences of prompt injection can extend beyond privacy violations to safety incidents. A clinical decision support system that accepts user-supplied patient notes and incorporates them into a diagnostic prompt is vulnerable to an injection attack that redirects the model to provide incorrect medical guidance. A financial risk assessment tool that accepts user-supplied context is vulnerable to an injection that suppresses risk flags.

Prompt injection controls operate at the input layer. Structured separation of system content and user content, using the chat completions API's role-based message structure rather than string concatenation, is the baseline control. The system prompt content should never be user-controllable. User input should be validated and sanitized before incorporation: at minimum, HTML encoding of control characters and rejection of inputs that contain recognizable instruction patterns.

Model Integrity

Open-weight model files downloaded from public repositories represent a supply chain risk. A model file with a verified hash from the original publisher is a known artifact. A model file whose hash has not been verified could have been modified in transit, replaced on the distribution host, or substituted during the download process. Model tampering that embeds malicious behaviors or biases into weights is technically feasible and has been demonstrated in research settings.

The mitigation is straightforward: maintain a registry of approved model files identified by their SHA-256 hash, verify the hash of every model file after download and before loading into the inference server, and reject any model whose hash does not match the approved value. This process should be automated and enforced at the model loading step, not left as a manual verification step.

The Regulatory Landscape

Running AI inside the perimeter transfers the governance obligation from the vendor to the deployer. When data goes to a cloud AI vendor, some portion of the governance obligation rests with the vendor under the BAA or DPA. When inference runs locally, the deployer bears the complete obligation. For regulated industries, this is not a reason to prefer cloud AI: it is a reason to build robust governance infrastructure for local deployments.

The EU AI Act (Regulation EU 2024/1689) establishes risk-based obligations that apply to deployers of high-risk AI systems. High-risk categories include AI systems used in medical devices, critical infrastructure, educational access, employment, essential services, law enforcement, migration, and administration of justice. Healthcare, financial services, and many government deployments fall within these categories. Article 26 obligates deployers to use the AI system in accordance with its instructions, monitor its performance, ensure human oversight, and maintain logs sufficient to reconstruct the system's inputs and outputs over the applicable retention period.

The EU AI Act penalty structure has two tiers. Violations involving prohibited AI practices carry penalties up to 7% of global annual turnover. Violations of deployer obligations, including logging failures and inadequate human oversight, carry penalties up to 3% of global annual turnover. For large organizations, these are not rounding errors in the risk calculation.

The Audit Schema

The audit trail for local AI inference is a structured log of every inference call. At minimum, each log entry must include: a unique invocation identifier; the timestamp of the request and response; the identity of the calling system or user; the model identifier and version; the data classification label of the input; the token count of the input and output; the latency; and the exit code. For HIPAA-covered processing, the log must also include the fact that PHI was processed (without logging the PHI itself in the audit record, which would create a secondary PHI exposure).

The audit log must be stored in append-only storage. Log entries that can be modified or deleted do not constitute a reliable audit trail. For regulated industries, consider using a HMAC (hash-based message authentication code) to sign each log entry at write time, enabling later verification that the log has not been tampered with. The signing key must be managed separately from the inference infrastructure under the NIST SP 800-57 key management framework.

NIST AI RMF (NIST AI 100-1, 2023) provides the operational structure for AI governance across four functions: Govern (establish accountability and policies), Map (identify AI risks), Measure (analyze and assess risks), and Manage (prioritize and treat risks). Local AI deployments must address all four functions, not just the Measure function (evaluation) that most AI teams focus on. Govern requires assigning accountability for each AI system and establishing change control processes. Map requires documenting the intended use case, the data used, and the applicable regulatory context. Manage requires ongoing monitoring and a process for responding to incidents.

The Case for Hybrid Architecture

A purely local AI architecture limits your organization to the capabilities of models you can run on hardware you own. A purely cloud AI architecture creates the data perimeter risks that regulated industries cannot tolerate. The hybrid architecture threads this needle: regulated data routes to local inference unconditionally, while tasks that involve non-regulated data and require frontier model capabilities route to the cloud. The routing policy, enforced programmatically, is what makes the hybrid architecture both compliant and capable.

FrugalGPT (Chen, Zaharia, Zou, arXiv:2310.11409) provides the theoretical and empirical foundation for cascade routing: the insight that most enterprise tasks do not require the most expensive model, and that a routing layer that matches tasks to the cheapest capable model achieves 60 to 80 percent inference cost reduction with no measurable quality loss on the evaluated task distributions. Applied to local-cloud routing, the same principle applies: most regulated tasks that can be handled by a fine-tuned local 7B or 13B model should be. Only the tasks that require frontier model capabilities, and whose data permits cloud transmission, should route to cloud APIs.

The economics of hybrid architecture improve with scale. The capital expenditure for local GPU infrastructure is fixed; the per-inference cost decreases as utilization increases. At low task volumes, cloud APIs are more cost-effective (no capital expenditure, pay per request). At high task volumes, local inference becomes cheaper once the fixed cost is amortized across enough requests. The break-even point depends on the GPU hardware cost, the utilization rate, and the cloud API pricing. Organizations should calculate this break-even for their specific workload before committing to local infrastructure investment.

Routing Policy Design

A routing policy has three dimensions: data sensitivity, task complexity, and quality requirement. Data sensitivity is the first gate and must be enforced unconditionally: any request involving regulated data routes to local inference regardless of task complexity or quality considerations. This is not a suggestion; it is a compliance requirement. The routing layer must enforce it programmatically, not rely on human judgment at request time.

For requests that do not involve regulated data, task complexity and quality requirement determine the routing decision. Simple extraction and classification tasks route to local inference: these are exactly the task types where fine-tuned local models match or exceed frontier model performance, and the cost advantage of local inference is most pronounced. Complex, open-ended reasoning tasks that require frontier model capability, and whose inputs do not contain regulated data, may route to cloud APIs.

The quality requirement adds a quality floor: if the local model does not meet the quality threshold for a given task type (as measured by the evaluation process described in Chapter 7), the request should route to the cloud regardless of data sensitivity, provided the data permits it. This is the cascade routing principle applied to the local-cloud dimension: start with local, escalate to cloud only when local does not meet the threshold.