# Model quantization techniques: use LLMs on your CPU

What if running a language model did not require an expensive GPU? Quantization changes the memory requirements, but the details matter.

By Paras Madan | 2024-02-28 | Updated 2026-09-14


Originally published on [Medium](https://medium.com/@parasmadan.in/model-quantization-techniques-use-llms-on-your-cpu-34d805da84b1) on February 28, 2024. Technical definitions have been corrected for republication. The original code example uses older libraries and a GGML model; it has not been retested against current releases.

Large language models are useful for language understanding, code generation, and question answering. But their size demands substantial memory and compute, which can make running them on everyday hardware difficult.

Quantization offers a way to reduce those requirements. This article explains what it does, why it helps, and how the original example loaded a quantized model for local inference.

## What is quantization?

Quantization represents values using a smaller set of possible values. For neural networks, that often means converting model weights from higher-precision representations, such as 32-bit floating-point, to lower-bit representations.

The goal is to reduce memory use and computational cost while keeping the model's accuracy acceptable for the task. Lower precision introduces approximation, so the resulting model still needs evaluation.

## Why quantize LLMs?

1. **Smaller model weights.** Moving from 16-bit weights to a 4-bit representation reduces the bits needed to store each weight. Quantization metadata and other runtime memory still add to the total, but the reduction can make a model fit on hardware that could not hold its higher-precision version.
2. **More accessible hardware.** A suitably sized quantized model can run on a CPU with a compatible inference runtime. Whether it is practical depends on available RAM, the model, context length, and the speed your application needs.
3. **Potentially faster inference.** Lower-bit representations can reduce memory traffic and improve performance with suitable kernels. Smaller weights do not guarantee a speedup on every processor or runtime, so benchmark the configuration you intend to use.

## The original code example

The February 2024 example used LangChain's `CTransformers` integration to load a quantized Llama 2 model:

```python
from langchain.llms import CTransformers

# Local CTransformers model
llm = CTransformers(
    model='TheBloke/Llama-2-7B-Chat-GGML',
    model_type='llama',
    config={
        'max_new_tokens': 256,
        'temperature': 0.01,
    },
)

prediction = llm.predict("Hi!, who is Narendra Modi")
print(prediction)
```

1. Import `CTransformers` from the LangChain integration used in the original example.
2. Initialize it with the model repository and generation settings.
3. Send a prompt to the local model.
4. Print the generated response.

This loads an already-quantized model; it does not perform quantization itself. A replacement model must have an architecture and file format supported by the runtime. You cannot substitute an arbitrary Hugging Face model and expect it to work.

## Quantization methods, formats, and runtimes

These names often appear together, but they refer to different parts of the process:

- **GPTQ** is a post-training weight-quantization method that uses approximate second-order information to reduce quantization error. It was introduced by Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. [GPTQ paper](https://arxiv.org/abs/2210.17323).
- **ExLlama** is an inference implementation designed to run Llama models with quantized weights efficiently. It is a runtime, rather than a quantization-aware training method. [ExLlama repository](https://github.com/turboderp/exllama).
- **NF4**, or NormalFloat 4, is a 4-bit data type introduced in the QLoRA paper for normally distributed weights. It is not a neural architecture search method. [QLoRA paper](https://arxiv.org/abs/2305.14314).
- **bitsandbytes** is a library providing quantization-related operations, including low-bit model support. It is a tool for implementing these techniques rather than a single quantization format. [bitsandbytes repository](https://github.com/bitsandbytes-foundation/bitsandbytes).
- **GGML** is a tensor library used in local inference tooling, with support for quantized operations. The original code uses an older model artifact carrying the GGML name; GGML itself is not one quantization algorithm. [GGML repository](https://github.com/ggml-org/ggml).

Choosing a quantization method and choosing a runtime are related decisions. They need to be compatible with each other, the model, and your hardware. Not every item in this list is intended for CPU inference.

## Final thoughts

Quantization can make local LLMs more accessible by reducing their memory requirements. That creates useful options for experimentation and applications that need local processing.

The practical test is whether the model fits, runs fast enough, and remains accurate enough for your use case. Check all three before treating a smaller model file as a successful deployment.