Running LLMs Locally: Open Weights, Quantization, and Getting Started with Zorp

The Open-Weight Revolution

Until 2023, running a capable large language model required access to either an API from a frontier lab or a research affiliation that provided compute access. The release of LLaMA (Touvron et al., arXiv:2302.13971) changed this. LLaMA released weights for models ranging from 7 to 65 billion parameters under a research license, and within weeks the open-source community had produced instruction-tuned derivatives, quantized versions runnable on consumer hardware, and the inference tooling needed to deploy them.

LLaMA 2 and Mistral 7B followed with more permissive licenses that allowed commercial use. By the time Mixtral 8x7B was released in late 2023, the ecosystem had tools for every part of the deployment pipeline: model quantization, CPU and GPU inference, serving infrastructure, and evaluation frameworks. The gap in capability between open-weight models and frontier proprietary models on standard benchmarks narrowed substantially, and for many domain-specific tasks the gap narrowed further still after fine-tuning.

For organizations with data privacy requirements, regulatory constraints on cloud processing, or high-volume inference needs where per-token API costs become significant, open-weight local deployment is now a viable operational choice rather than a research exercise.

Quantization Methods

A 70 billion parameter model in full bfloat16 precision requires roughly 140 GB of GPU memory, which exceeds the capacity of all but the largest single-GPU configurations. Quantization compresses the weight representation from 16-bit floating point to 8-bit or 4-bit integers, reducing memory requirements proportionally. The quality cost depends on the quantization method and the bit width.

GPTQ (Frantar et al., arXiv:2210.17323) is a post-training quantization method that applies layer-by-layer quantization with second-order weight updates to minimize the reconstruction error introduced by quantization. A 4-bit GPTQ quantization of a 70 billion parameter model requires roughly 35 GB of GPU memory and typically loses a small but measurable amount of quality on standard benchmarks compared to the full-precision version.

AWQ (Lin et al., arXiv:2306.00978) identifies the subset of weights that are most influential for model quality (the salient weights) and protects them during quantization while aggressively compressing the rest. This produces better quality at 4-bit than naive uniform quantization and is particularly effective for instruction-following tasks where specific knowledge must be preserved precisely.

GGUF (the format used by llama.cpp) supports a range of quantization levels from Q2_K (2-bit, significant quality degradation) through Q8_0 (8-bit, near-lossless). The llama.cpp runtime can offload layers between GPU and CPU, allowing models too large to fit entirely in GPU memory to run at reduced throughput. For a 70 billion parameter model at Q4_K_M quantization, GPU memory requirements drop to roughly 40 to 48 GB depending on the quantization variant, which fits within a dual-GPU workstation configuration.

Hardware Requirements at Each Scale

The practical hardware requirements for local deployment scale with model size and quantization choice. Directional ranges (not sourced from a single study; these are practitioner estimates widely reported in the open-source community):

• 7B models at Q4 run on 6 to 8 GB GPU memory, available in most consumer gaming GPUs from the last two to three years
• 13B models at Q4 require 8 to 10 GB GPU memory, achievable on a single mid-range GPU
• 34B models at Q4 require 20 to 24 GB, requiring either a high-end professional GPU or two consumer GPUs
• 70B models at Q4 require 40 to 48 GB, achievable on a single high-end professional GPU (A100 80GB or H100) or two high-end consumer GPUs

CPU inference is possible with GGUF models but slow. A 7B model running on a modern CPU produces tokens at roughly 3 to 10 tokens per second depending on the processor, compared to 40 to 100 tokens per second on a capable GPU. For interactive use cases, GPU inference is strongly preferred. For batch offline tasks where throughput matters more than latency, CPU inference with multi-threading can be cost-effective if GPU hardware is unavailable.

Getting Started with Zorp

Zorp (by Aviskaar) is a local LLM runtime that handles the operational complexity of running open-weight models: model downloading, quantization format selection, GPU memory management, and serving behind an OpenAI-compatible API. For developers and teams who want to run local LLMs without becoming experts in llama.cpp configuration or CUDA memory management, Zorp provides a layer of abstraction that makes the process approachable.

The key operational decisions Zorp handles automatically include quantization selection given your available GPU memory, layer offloading configuration when the model exceeds GPU capacity, context window management, and batching for concurrent requests. These are decisions that previously required reading through llama.cpp documentation and experimenting with command-line flags. Presenting them as sensible defaults with optional overrides reduces the time from "I want to try this model" to "it is running and serving requests" from hours to minutes for most configurations.

Privacy and Data Residency

The most compelling enterprise use case for local LLM deployment is not cost optimization but data residency. Organizations in regulated industries, particularly healthcare, finance, and legal, often cannot send sensitive data to third-party API endpoints without complex compliance arrangements. Running inference locally on hardware within your own control eliminates this constraint.

MEDFIT-LLM (Rao, Jaggi, Naidu, IEEE RMKMATE 2025, DOI:10.1109/RMKMATE64574.2025.11042816) demonstrates this in the healthcare context: fine-tuned local models evaluated against medical benchmark datasets can achieve performance competitive with cloud-hosted general models on domain-specific tasks, while keeping patient data on premises. The combination of a capable open-weight base model, domain-specific fine-tuning on locally held data, and a local runtime like Zorp represents a complete stack for regulated enterprise deployment without cloud data exposure.

Evaluating Whether Local Deployment Is Right for Your Use Case

Not every use case benefits from local deployment. The decision involves tradeoffs across several dimensions that should be evaluated explicitly rather than assumed.

Capability tradeoffs. The best open-weight models at any given time trail frontier proprietary models on complex reasoning, instruction following, and knowledge breadth. For tasks requiring frontier-level capability, local deployment at current model sizes involves a quality cost. The gap is smaller for domain-specific tasks where fine-tuning can recover much of the general capability deficit.

Cost tradeoffs. Local deployment has fixed hardware costs and low marginal inference costs. API deployment has zero fixed costs and variable per-token costs. At low volumes, API is cheaper. At high volumes, the crossover point depends on hardware amortization and token volume. FrugalGPT (Chen, Zaharia, Zou, arXiv:2310.11409) provides a formal framework for evaluating this crossover under different cost and quality constraints.

Operational tradeoffs. Running local models requires infrastructure management: hardware maintenance, model updates, reliability engineering, and performance monitoring. API services handle all of this in exchange for per-token fees and data residency constraints. Organizations without existing ML infrastructure should factor in the operational overhead, not just the hardware cost.

The series began with how LLMs work at the token prediction level and ends here, at the point where a practitioner can download an open-weight model, quantize it, deploy it locally, and begin evaluating it against their own use case. The path from transformer attention to Zorp running a 70B model on a workstation is shorter than it appears from the outside, and the tooling continues to improve. The ability to run capable models locally, privately, and at low marginal cost is one of the more consequential shifts in enterprise AI infrastructure of the last several years.