Running LLMs Without a GPU: What's Actually Possible

The honest numbers on CPU, NPU, and iGPU inference — what runs, how fast, and when to stop pretending.

Last year a friend who runs a small consulting firm in Dubai asked me what GPU he needed to "run AI locally." He had a budget in mind and had already been quoted a four-figure price tag for a workstation. I asked him what he actually wanted to run. A document chatbot over his firm's contracts. He owned a three-year-old office laptop with 16 GB of RAM and no discrete GPU.

I told him not to buy anything yet. Two hours later, I had a 7B-parameter model running on that laptop at about 11 tokens per second, answering questions from his own PDFs. Not fast enough for a chat product serving a thousand users. Fast enough for a private assistant that costs nothing per query and keeps every contract on his machine.

The market wants you to believe local AI requires expensive hardware. The truth is more interesting and much cheaper. Here is what is actually possible — with real numbers, real architecture, and an honest map of where CPU-only inference stops being viable.

Why Everyone Thinks You Need a GPU

The confusion comes from a conflation of two different workloads. Training and fine-tuning a model is brutally compute-hungry: it means doing forward passes over billions of parameters millions of times, and backpropagation on top. That genuinely wants GPUs, and a lot of them.

Inference — running the model to produce an answer — is a different animal. Generating one token means one forward pass through the network. It is memory-bandwidth bound far more than compute bound, especially at the small batch sizes a single user generates. That is the single most important fact in this entire article: for single-user inference, what matters is how fast you can stream weights from memory, not how many FLOPS you have. A CPU with lots of fast RAM can do a surprisingly respectable job, and modern processors have neural accelerators that help even more.

What Runs, and How Fast: The Honest Numbers

Here are the real token rates I have measured on actual hardware, not the marketing ones. Your mileage varies, but these are the right ballparks for current consumer machines:

For context: comfortable reading speed is roughly 20 tokens per second. Below 8 tokens per second, interactive chat starts to feel sluggish, though a batch job — summarize these 400 documents overnight — does not care about interactivity at all, which changes the calculus completely.

One more distinction that explains why CPU inference feels uneven in practice: the two phases of generation behave very differently. Prompt processing (reading the input, aka prefill) is compute-bound and can be slow on CPU for long inputs — a 2,000-token prompt might take a second or two before the first output token appears. Token generation (decode) is memory-bandwidth-bound and steady — once the model is warmed up, each output token streams at a consistent rate. So the experienced quality of a CPU model is dominated by prompt length: short prompts with long answers feel fine; long prompts with short answers feel sluggish, and that is pure prefill time, not generation speed. Capping context and keeping inputs tight is not a quality compromise, it is a latency lever.

The magic ingredient on the Apple side is unified memory: the same memory serves CPU and GPU, and it is fast memory. That is why a fanless MacBook Air outperforms a bigger Windows laptop for llama.cpp. On the Windows/Intel side, the NPUs shipping in 2024+ laptops are the emerging story — they are built for low-power token generation, and software support is maturing quickly.

The Stack That Makes This Work

Every serious CPU inference setup is built from the same four pieces:

1. Quantized models (GGUF). The breakthrough that made CPU inference practical. You take a model whose weights are 16-bit floats and shrink them to 4-bit integers, trading a little quality for a ~4x memory and bandwidth reduction. The GGUF format (from the llama.cpp project) is the standard container. The Q4_K_M variant is the sweet spot I default to — decent quality, ~5 GB for an 8B model, fits in a laptop's budget. Q8_0 is better quality at ~9 GB; Q3_K_M fits in 4 GB but you will feel the quality loss.

2. llama.cpp. The reference runtime. A pure C/C++ implementation that runs on CPU, and offloads layers to GPU/NPU where available. It is the engine behind virtually every local-LLM tool you have heard of, and it is fast on CPU because it was written to be.

3. A server layer. llama.cpp ships a server binary that speaks the OpenAI-compatible chat/completions API. This is the detail that makes CPU inference actually usable: any tool that talks to the OpenAI API — including your existing app, if you point its base_url at your local server — will work against your CPU model with a one-line change.

4. Ollama (or llama-cpp-python) for the glue. Ollama wraps llama.cpp with model management and a dead-simple CLI; llama-cpp-python gives you the same engine as a Python package. Both are legitimate; you are choosing convenience over control.

Let's Actually Run One

Here is the whole setup on a Mac or Linux box. First, install Ollama and pull a quantized model:

# install ollama, then:
ollama pull llama3.1:8b-q4_K_M
ollama serve

That single command is now a running LLM endpoint on http://localhost:11434, OpenAI-compatible. Test it:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.1:8b-q4_K_M",
       "messages": [{"role": "user", "content": "Explain quantization in 50 words."}]}'

If you prefer raw llama.cpp for control — say, you want to offload specific layers to an iGPU or NPU — you clone it and run the server directly:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build --config Release
./build/bin/llama-server \
  -m llama-3.1-8b-instruct-Q4_K_M.gguf \
  --n-gpu-layers 12 \
  -c 4096

The --n-gpu-layers flag is the whole game on hybrid hardware: you push the layers that benefit from the GPU/NPU and keep the rest on CPU, trading power draw for speed. On the Apple side the equivalent is Metal layers (-ngl with Metal enabled).

Now the part people forget. Point your OpenAI-compatible client at it:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
resp = client.chat.completions.create(
    model="llama3.1:8b-q4_K_M",
    messages=[{"role": "user", "content": "Summarize the attached contract risk clauses."}],
)
print(resp.choices[0].message.content)

That is it. Your app now runs on a local model with zero per-query cost and zero data leaving the machine. The entire "I need a GPU" story collapses into one base_url change.

If you do not know which model to start with, here is the shortlist I use. For a first experiment, a 7–8B model at Q4_K_M (Llama 3.1, Qwen 2.5, Gemma) is the right default on 16 GB of RAM — big enough to feel genuinely capable, small enough to be usable. If you are on 8 GB or an older machine, step down to a 3–4B model at Q4_K_M; the quality gap is smaller than the speed gap suggests. And if you only need classification or extraction, a 1–3B model at 40+ tok/s will beat an 8B at 10 tok/s for the job. The rule: pick the smallest model that passes your acceptance test, then measure — most people overestimate what they need by one size.

What Makes CPU Inference Go Fast (or Slow)

If you are on CPU-only and the speed is not acceptable, the fix is almost never "get a GPU." Work through this order:

1. Check your quantization. Going from Q8_0 to Q4_K_M nearly doubles throughput at modest quality cost. For chat, most people cannot reliably tell the difference.
2. Cap your context. A long conversation balloons the prompt-processing phase. Dropping -c from 8192 to 4096 makes the first token arrive much faster.
3. Use your accelerators. Offload layers to the iGPU/NPU where available. Even a partial offload cuts power draw and wall time.
4. Match the model to the job. A 3B model at 40 tok/s is better than an 8B at 9 tok/s for classification, extraction, or summarization. Use a small model for the routine work and escalate only when needed.
5. Cache. KV-cache reuse and prompt caching cut latency dramatically on repeated queries. llama.cpp supports both.

And here is the counterintuitive one: for batch work, a slow model is a non-issue. If you need to tag 10,000 support tickets overnight, 5 tok/s is plenty. You are optimizing for throughput over time, not first-token latency, and CPU inference wins that comparison on cost by a mile.

The same logic extends to embeddings, which people forget entirely. An embedding model like a small 300M–1B parameter encoder runs happily on CPU and is the backbone of local RAG: it turns your documents into vectors on your own machine, with no API calls and no data leaving the building. I have run a full local retrieval pipeline — embeddings plus a small chat model — on a single 16 GB laptop, and the retrieval step, which is the part that actually makes answers good, is nearly instant. If your goal is a private document assistant, the LLM is the part you have to accept as "good enough"; the embeddings are the part that genuinely shines.

Where It Breaks: The Honest Limits

I want to be equally honest about the wall. CPU-only inference fails in four specific places, and you should not pretend otherwise:

1. Large models. Anything above ~14B at Q4 needs 10+ GB just for weights, and you will hit 5–8 tok/s — below the interactive threshold. Models that need 32 GB of RAM are off the table entirely on consumer hardware.
2. High concurrency. A CPU serving five users at once just divides its tokens between them. This is a single-user story, or a low-concurrency internal tool, not a public product.
3. Long-generation tasks. Drafting a full article or a 2,000-token email response is painful at 10 tok/s — 3+ minutes. Plan for it or use a small, fast model for drafting.
4. Agent workloads. When an agent runs ten chained model calls with tool calls in between, each one pays the full prompt-processing tax. On CPU, a five-step agent run can take minutes. Local-first agents on CPU exist, but you are optimizing for privacy and cost, not speed.

When You Actually Do Need a GPU

You need the GPU when the workload is compute-bound or concurrent: fine-tuning, large-scale embedding jobs, or serving many simultaneous users. If you are building the next consumer chatbot product, you are not the CPU story. You are the cloud API story.

But here is the framing I use now, after years of building: the GPU question is not "should I buy one" — it is "what is the workload, what is the concurrency, and what are the latency requirements?" A private document assistant, a code-completion tool running on a single developer's machine, an on-prem compliance filter that must never see the internet, a batch tagging pipeline — all of these are CPU-viable today, and the hardware you already own is probably enough to start.

The Practitioner's Checklist

When someone asks me to stand up local LLMs on the hardware they have, I work through this list:

• Define the workload: interactive chat, batch processing, or agent loops
• Measure RAM: weights (Q4 ≈ 0.6–0.7 GB per billion params) plus context, plus OS
• Pick GGUF quantization: Q4_K_M default, Q8_0 if quality matters more than speed
• Start with the biggest model that fits in RAM at the target speed; drop a size if under ~8 tok/s for chat
• Set the server up OpenAI-compatible so existing code works with a base_url change
• Cap context to what the task actually needs
• Offload layers to iGPU/NPU/Metal where available
• For batch jobs, accept low token rates and let throughput win
• For agents, budget for multi-call latency or drop to a smaller model
• Keep a cloud API as the escape hatch for the jobs that outgrow the box

The Real Answer

My friend in Dubai runs his contract assistant on that same laptop today. It does not write poetry and it is not answering a thousand concurrent users. It reads contracts, answers questions in context, and costs him nothing per query — and his documents never leave his machine. That is the actual value proposition of CPU inference: not "local models beat the cloud," but "local models are good enough for a specific class of work, and that class is much bigger than people think."

Start with what you already own. Quantize, cap context, offload what you can, and match the model to the job. You will be surprised how far a laptop goes — and you will only spend GPU money when the workload genuinely demands it.

*Gulshan Yad

CPU vs GPU Inference Trade‑offs

Modern CPUs have evolved to include large core counts, deep cache hierarchies, and advanced vector instruction sets such as AVX‑512. These features allow a single CPU to execute many parallel matrix multiplications, the core of transformer inference. However, GPUs still offer higher raw throughput for dense linear algebra due to their massively parallel architecture and higher memory bandwidth. For workloads that demand low per‑request latency rather than high throughput, a CPU can be preferable, especially when the cost of GPU instances outweighs the performance benefit. CPU inference also benefits from lower power consumption and simpler deployment pipelines, as most servers already host CPUs.

When evaluating CPU viability, consider the model size, batch size, and token length. Models under 4 B parameters can often be accommodated on a 32‑core CPU with 64 GB of RAM, achieving token latencies in the 200–300 ms range with 8‑bit quantization. Larger models require sharding or hybrid CPU‑GPU pipelines. Additionally, CPU inference can be more predictable in terms of latency, as it avoids the scheduling overhead and GPU queue contention present in shared GPU environments.

Model Quantization Techniques

Quantization reduces the precision of weights and activations, shrinking memory footprints and accelerating arithmetic on integer units. In practice, 8‑bit integer quantization is the most common approach for CPU inference, as it preserves a high degree of model fidelity while enabling efficient use of SIMD instructions. 4‑bit quantization further cuts memory usage but can introduce noticeable degradation in nuanced tasks such as long‑form generation or fine‑grained sentiment analysis.

Weight‑sharing, where multiple attention heads or layers share the same weight matrices, can reduce parameter count without altering architecture. Activation pruning, which zeroes out low‑importance activations, can be combined with sparse attention patterns to lower both memory usage and compute. Frameworks like QLoRA or bitsandbytes provide tooling to apply these techniques with minimal code changes. It is essential to validate the quantized model on a representative dataset, as some tasks may be more sensitive to precision loss.

Batch Inference and Pipelining

Batching is a powerful way to amortize the overhead of kernel launches and memory transfers. On CPUs, larger batch sizes improve cache utilization and allow vector units to operate at full capacity. However, latency‑sensitive applications must balance batch size against response time. A common strategy is to use a dynamic batching queue: incoming requests are grouped until a threshold is reached or a timeout expires, then processed together.

Pipelining can also be employed by overlapping token generation with I/O. While the CPU processes the current token, the next token can be fetched from a pre‑computed cache or streamed from disk. This approach is particularly effective when combined with asynchronous I/O libraries such as libaio or the async features of modern frameworks. Profiling tools can help identify the optimal batch size and pipeline depth for a given workload.

Edge Deployment Strategies

Deploying LLMs to edge devices—such as smartphones, IoT gateways, or embedded systems—poses unique constraints. Limited RAM (often 2–8 GB) and lower compute capabilities mean that only very small models (≤200 M parameters) are feasible. Aggressive quantization to 4‑bit or even binary weights is often necessary, and the use of lightweight transformer variants (e.g., DistilBERT, MobileBERT) can further reduce memory demands.

Edge deployment also benefits from model pruning and knowledge distillation. By training a smaller student model to mimic a larger teacher, one can achieve comparable performance with a fraction of the parameters. Additionally, offloading intermediate tensors to flash storage or using memory‑mapped files can allow the device to process longer sequences without exhausting RAM.

Memory Management and Offloading

Memory constraints are the primary barrier to CPU inference for large models. Effective strategies include:

• Tensor offloading – Store intermediate activations on disk or a secondary memory pool, loading them on demand. This technique is supported by frameworks that provide a “checkpoint” API.
• Sparse attention – Replace full‑sequence attention with local or block‑sparse patterns, reducing the number of required multiplications.
• Buffer reuse – Allocate a single large buffer for activations and reuse it across layers, minimizing fragmentation.
• Just‑in‑time (JIT) compilation – Compile the inference graph for the specific CPU architecture, eliminating generic overhead.

By combining these techniques, it is possible to run a 6 B parameter model on a high‑core CPU with 64 GB of RAM, albeit with increased latency compared to GPU execution.

Future Hardware Trends and Hybrid Solutions

CPU manufacturers are incorporating AI‑specific instruction sets and accelerators. Intel’s upcoming AI Acceleration Engine and AMD’s AI cores aim to accelerate matrix operations directly on the CPU die. These developments will narrow the performance gap between CPUs and GPUs for transformer inference.

Hybrid inference pipelines, where a lightweight CPU model performs initial token filtering or prompt‑tuning and a heavier GPU model handles the final decoding, provide a pragmatic compromise. This approach reduces GPU memory pressure while still leveraging the GPU’s speed for the most compute‑intensive part of the pipeline. As hardware evolves, increasingly sophisticated multi‑device orchestration—using frameworks that support device placement at the tensor level—will enable seamless scaling across CPUs, GPUs, and emerging AI accelerators.

Key Takeaways

• CPU inference can be viable for small‑to‑medium LLMs when paired with aggressive quantization and efficient token‑level batching, reducing latency to acceptable ranges for many production workloads.
• Model size is the primary bottleneck—models under 4 B parameters can comfortably run on a single high‑core CPU with 32 GB RAM; larger models require sharding or hybrid CPU‑GPU setups.
• Quantization to 4‑bit or 8‑bit precision, combined with weight‑sharing and activation pruning, cuts memory usage by up to 75 % while preserving most downstream performance.
• Effective memory management—including off‑loading intermediate tensors to disk or using sparse attention patterns—enables inference of models that would otherwise exceed available RAM.
• Hybrid inference pipelines, where a lightweight CPU model handles routing or prompt‑tuning and a heavier GPU model performs final decoding, balance cost and speed for latency‑sensitive applications.
• Monitoring and profiling tools (e.g., Intel VTune, Linux perf, or vendor‑specific profilers) are essential to identify CPU bottlenecks and guide optimization decisions.