GPT-4 launched in March 2023 scoring roughly 86.4% on MMLU. Today, Meta's Llama 3.1 70B (a model you can download for free, run on your own hardware, and modify without restriction) scores 86.0% on the same benchmark. The frontier has moved, but the floor moved with it. This is the complete framework for deciding when the $0 stack is the right stack, when it isn't, and exactly how to build each layer.
There are three distinct versions of the $0 AI stack. They have different architectures, different constraints, and different failure modes. Confusing them is the most common error teams make when scoping an AI initiative.
Free weights (open-source): Models released under open licenses where the weights are freely downloadable. You own the deployment, the cost, the latency, and the privacy guarantees. Meta's Llama 3 family, Mistral, Google's Gemma, and Microsoft's Phi-3 are the canonical examples. The inference cost is not zero: it is the cost of the hardware you run these models on. But the model itself costs nothing.
Free inference tiers: Hosted API access to models, sometimes frontier, sometimes open-source, provided free within rate limits. Google AI Studio offers access to Gemini models at no cost under documented rate limits. Groq provides a free developer tier for several open-source models. Together AI, Hugging Face Inference API, and others offer similar access. These are real and usable for prototyping and low-volume applications. They stop working the moment your traffic exceeds their limits.
Free tier and open source hybrid: Run open-source models on cloud infrastructure at near-zero cost by taking advantage of spot pricing, university compute credits, free tiers from GPU cloud providers, or your own existing hardware. This is often the most durable version of the $0 stack for a team that needs consistent throughput.
Open-source means zero licensing cost, not zero inference cost. The question is not "is this free?" but "is this free given my volume, my hardware, and my latency requirements?" The answers diverge sharply once you move from prototype to pilot.
The clearest way to understand what the $0 stack can do today is to read the benchmarks directly from the papers. MMLU (Massive Multitask Language Understanding) tests knowledge across 57 subjects from elementary to professional level. It is not a perfect proxy for real-world task performance, but it is one of the most consistently reported metrics across model releases.
MMLU is a multiple-choice benchmark evaluated in a controlled setting. Real-world task performance on enterprise workloads (document analysis, code generation, structured extraction) differs from benchmark performance. The numbers below establish relative capability, not a guarantee of fit for your specific task.
The Meta Llama 3.1 Herd of Models paper (arXiv:2407.21783) reports MMLU scores of 73.0% for the 8B parameter model and 86.0% for the 70B model. The 405B model, also freely available for download under Meta's Llama 3 Community License, scores 88.6%, above every commercially released model as of its launch date.
Mistral AI's original Mistral 7B (arXiv:2310.06825) reports 64.2% MMLU, a meaningful result for a 7B parameter model. The Mistral family uses grouped-query attention and a sliding window attention mechanism that improves inference efficiency significantly relative to models of the same parameter count.
Microsoft's Phi-3-Mini 3.8B (arXiv:2404.14219) reports 68.8% MMLU. This is an outlier: a 3.8 billion parameter model beating many 7B models on a knowledge benchmark. Microsoft attributes this to high-quality synthetic training data. For edge deployment or environments where hardware is genuinely constrained, Phi-3 Mini is the current benchmark leader by parameter efficiency.
Google's Gemma 2 9B (Google Gemma 2 Technical Report, 2024) scores 72.3% MMLU. The Gemma 2 family is notable for its permissive Apache 2.0 license, its strong code performance relative to its general language score, and its availability through the Keras and Hugging Face ecosystems with minimal setup friction.
Several providers offer meaningful free access to hosted inference in 2026. Each has documented rate limits, and those limits define the ceiling of what you can build with them before cost appears.
Google AI Studio provides free access to Gemini models under its documented free tier. The rate limits are published at ai.google.dev and are sufficient for prototyping, internal tools, and low-volume applications. The free tier operates on Google's infrastructure, which means you get production-grade latency without managing a cluster. The constraint is that your data moves through Google's systems, which creates data-handling implications for regulated industries.
Groq operates a free developer tier on its Language Processing Unit (LPU) inference hardware, providing access to several open-source models including Llama 3.1 and Mistral variants. Groq's hardware delivers token generation speeds substantially faster than standard GPU inference. The free tier is rate-limited by requests per minute and day, with limits documented at console.groq.com.
Together AI provides an initial credit balance for new accounts that is sufficient for several hundred thousand tokens of inference. This is not a sustained free tier: it is a trial credit. Teams that want ongoing free access should plan around open-source weights on their own hardware, not on Together's hosted API.
Hugging Face Inference API provides rate-limited free access to many models in the Hugging Face Hub. The free tier is intentionally constrained and is designed for experimentation, not production traffic. The ecosystem's primary value for the $0 stack is its model hosting, dataset hosting, and tooling integrations, all of which are free.
Every free inference tier will throttle you at scale. This is not a bug: it is the provider's cost management. If your application needs consistent throughput above a few hundred requests per hour, the $0 inference tier path closes. The open-weights path on your own hardware opens.
Ollama is the simplest path to running open-source models locally. It is an open-source tool (github.com/ollama/ollama) that handles model download, quantization selection, hardware detection, and a local REST API that mirrors the OpenAI API surface.
The hardware requirements are worth stating precisely, because "runs locally" means different things at different parameter counts:
Knowing where the free path fails is as important as knowing where it works. There are four categories of limitation that consistently surface in enterprise evaluations.
Instruction-following consistency at scale. Open-source models at the 7B–13B range show higher variance in instruction adherence compared to frontier models. For tasks where every response must follow a precise format (structured JSON extraction, legal template completion, compliance-sensitive outputs), the lower tail of the distribution matters. A model that produces correct output 92% of the time and malformed output 8% of the time is not a production tool without significant post-processing scaffolding.
Long-context performance. Llama 3.1's architecture supports a 128K token context window. In practice, performance on tasks requiring reasoning across very long documents degrades well before the context limit is reached. This is consistent with findings in long-context evaluation research. If your application requires sustained coherent reasoning across 50,000 tokens or more, plan for retrieval-augmented patterns rather than long-context inference.
Multimodal tasks. The most capable open-source vision models in 2026 lag frontier multimodal systems on complex document understanding and diagram interpretation. For workloads that are fundamentally about images (invoice processing, medical imaging analysis), the open-source path requires careful benchmark evaluation against your specific task before committing.
Operational burden. A $0 model is not zero total cost of ownership. Someone on your team must provision hardware, manage model versions, handle quantization decisions, monitor inference latency, and respond when the GPU runs out of memory. For a team without an ML infrastructure function, the hidden cost of operating the local stack often exceeds the cost of paying for a hosted API.
| Use Case | Recommended Tier | Privacy | $0 Path | Key Constraint |
|---|---|---|---|---|
| Internal document Q&A | 7B–13B local | Full control | Primary | Retrieval quality, not model quality |
| Code review / completion | 13B–70B | Full control | Primary | Context length for large files |
| Customer-facing chatbot | 70B or frontier | Depends on data | Secondary | Instruction consistency at 7B–13B |
| Structured data extraction | 70B preferred | Full control | Primary (70B) | Lower reliability at smaller sizes |
| HIPAA / PHI workloads | Local only | Full control | Required | Compliance mandates local |
| Complex multimodal analysis | Frontier | Hosted required | Blocked | Open-source vision capability gap |
| Real-time voice / streaming | Frontier or Groq | Depends on provider | Groq free tier | Latency above 7B on consumer GPU |
| Batch processing (overnight) | 7B–70B local | Full control | Primary | Hardware provisioning |
| Agentic / multi-step reasoning | 70B minimum | Full control | Secondary (70B) | Error compounding at smaller sizes |
Llama 3 Community License: Meta's Llama 3 models are released under a custom community license, not a standard OSI-approved open-source license. Commercial use is permitted. However, organizations with more than 700 million monthly active users must request a separate license from Meta. Derivative models must carry the Llama 3 name. The weights may not be used to train other foundation models that compete with Meta's products. For the vast majority of enterprise use cases, these restrictions are non-binding, but they must be understood.
Apache 2.0 (Mistral, Phi-3, Gemma): Mistral 7B, Microsoft's Phi-3 family, and Google's Gemma 2 models are released under the Apache 2.0 license. This is a permissive license with no commercial use restrictions and no copyleft provisions. For enterprise commercial deployment, Apache 2.0 is the cleaner licensing path.
Move to a frontier paid API when: instruction-following consistency below 95% is causing downstream pipeline failures, the task requires multimodal capability the open stack cannot match, or the operational overhead of running your own inference exceeds the cost of hosted inference for your volume. Not before any of those conditions are true.
Fine-tuning: Running inference on open-source models is free. Fine-tuning them is not, unless you have access to significant compute. A full fine-tuning run on Llama 3.1 70B requires GPU clusters that cost hundreds to thousands of dollars to run. Parameter-efficient fine-tuning methods like LoRA (Hu et al., arXiv:2106.09685) reduce this substantially, but not to zero. The MEDFIT-LLM work (Rao, Jaggi, Naidu, IEEE RMKMATE 2025, DOI:10.1109/RMKMATE64574.2025.11042816) demonstrates that fine-tuning smaller open-source models on domain-specific data can match much larger general models in specialized tasks, but the fine-tuning compute is a real cost that must be planned for.
Safety and alignment: Llama 3.1 Instruct models include safety fine-tuning; Llama 3.1 base models do not. For customer-facing applications, using a base model without safety fine-tuning is not appropriate. The Constitutional AI framework (Bai et al., arXiv:2212.08073) and related alignment techniques are available for teams that want to add safety layers to base models, but this requires additional engineering investment.