LLM Quantization Formats: INT8, INT4, GPTQ, AWQ, GGUF — Which to Choose?

Why LLM quantization breaks answer quality

LLM quantization converts weights from a 16-bit representation to 8 or 4 bits. Compressing model weights reduces its size by two to four times: it no longer requires multiple accelerators, fits in the memory of one, and responds noticeably faster.

The price is rounding each weight, and rounding errors accumulate across the entire depth of the network.

The average metric barely moves: a difference of tenths of a percent looks safe on a reporting chart.

Degradation in answer quality settles not in the average but on rare tokens — numbers, dates, SKUs, names, units of measurement — and on long reasoning chains, where an error from an early step only gets multiplied further.

The first to suffer are predictable classes of requests: extracting fields from contracts, arithmetic on tables, code generation relying on rare calls, multi-step instructions, and working with long context closer to its end.

The answer format is maintained at low bit width — the model confidently returns the required structure, but the facts inside it are replaced with plausible ones.

For a product, this is the most expensive class of failures. The user does not see the error or the limits of applicability; they receive a neatly formatted incorrect answer and lose trust in the entire system, not just one request.

Therefore, a lack of memory and model response time cannot be solved with a single slider move toward lower bit width — we examine how this is addressed without a drop in accuracy.

What weight compression provides: response speed and memory footprint

Model weights occupy the main part of memory when working in inference — and are compressed first. Moving from full precision to INT8 reduces it by approximately half; to INT4, by three to four times.

The freed volume goes to the context cache and increased request batch size: memory savings turn into the headroom that was usually lacking at peak load.

Response time drops not because computations have become "faster," but because less data is read from memory at each generation step.

While memory subsystem bandwidth remained the bottleneck, adding accelerators barely helped; after compression, the same fleet serves more simultaneous requests, and tail latencies stop growing during spikes.

Model placement density grows by multiples: where one large model fit, two to four fit, including separate ones for narrow tasks — classification, entity extraction, summarization.

Instead of one overloaded model for everything, you get a set of specialized models, each with its own memory budget and load profile.

Stability under load is a consequence of the same headroom. When memory is sufficient with a margin, the scheduler does not evict contexts or restart instances, and queues do not grow avalanche-like.

With aggressive compression without calibration on your domain, answer quality may degrade, so the format — GPTQ, AWQ, GGUF — is chosen for the task and traffic profile, not by the principle "the fewer bits, the better."

Comparison of weight compression formats: accuracy and limitations

Quantization reduces weight bit width so the model fits into accelerator memory and produces more tokens per second. But choosing a format is not only about the number of bits: methods reconstruct weights differently and therefore transfer loss to answer quality differently. The error manifests not at load time, but later — on long reasoning and code generation.

Below are the real alternatives among which a choice is made on the test bench.

In practice, two or three builds are kept: one with accuracy headroom for complex tasks, the second more compressed for bulk requests. The selection criterion is not an averaged metric but the divergence of answers from the full model on your task set.

Calibration sets and method descriptions are convenient to take from Hugging Face — there you can also see what data the format was tested on.

How implementation proceeds: audit, scheme, validation, handover

LLM quantization implementation begins not with choosing a format but with measuring what the model currently outputs.

Each subsequent stage relies on the previous one: without baseline metrics, there is nothing to compare the compressed variant against, and without validation on real requests, you cannot promise preserved quality. Therefore, the implementation stages follow a fixed order.

1. Model audit. We capture baseline metrics at full precision: the share of correct answers on your request set, response time, memory consumption. We separately mark scenarios where an error costs the most — they will become control scenarios.
2. Selection of compression mode. We compare quality requirements and available resources: INT8 is the cautious option, INT4 with group quantization (GPTQ, AWQ) when memory matters more, GGUF for local deployment.
3. Compression setup. We assemble a calibration set from real requests, set the group size, and leave sensitive layers at higher precision — this is cheaper than losing quality across the entire range.
4. Quality testing. We run the compressed variant through control scenarios and compare with the baseline: match rate, answer coherence, behavior on long context. If divergence is above the threshold, we return to setup rather than attributing it to compression specifics.
5. Safeguards and rollback. We keep a full-weight copy and version the weights so that returning to the previous build takes minutes.
6. Solution handover. We provide weights, configurations, build scripts, and procedures: how to update the model, what to use to measure quality, at which metrics to roll back. Your engineers reproduce the build without us.

We record all measurements and selected parameters in a report — it shows why this particular compression mode was suitable and what will change when switching to another.

List of artifacts: configurations, schemes, procedures

A built model without a fixed configuration is reproduced only on the test bench where it was first launched: the compression result depends on the calibration set and pinned dependency versions.

Therefore, together with the solution, a package is handed over, according to which your team can bring up, validate, and roll back the model itself.

• Build and compression configurations — bit width, calibration set, pinned dependency versions: a rebuild produces the same artifact.
• Deployment schemes — model layout across accelerators, memory requirements, balancing points, and peak load headroom.
• Access policies — roles for reading and changing parameters, procedure for issuing keys to the inference service, access log.
• Operational procedures — what to do when quality degrades, how to roll back the previous build, when a rebuild is needed.
• Team instructions — how to bring up the test bench, reproduce quality measurements, add a new compression format to the tooling.
• Documentation — model passport, accuracy measurements before and after compression, change log with the owner of each edit.
• Reference validation sets — questions that reveal degradation after switching to INT4: formats are compared on your data, not by general reviews.

The package is designed for independent work: you can rebuild the model at a different bit width or restore the previous version without our involvement.

Case study: reduced hardware fleet without quality loss

How it was measured and what happened

The open-weight Llama model at full precision occupied four cards, and peak traffic forced horizontal scaling.

Load testing was assembled on a recording of real traffic with two profiles — short typical questions and long requests with attachments: memory consumption and response time behave differently on them.

Metrics were captured before quantization and after each format, from INT8 to group quantization with calibration, on the same hardware and the same request set.

The mixed scheme — INT4 on linear layers, INT8 on attention and embedding layers — reduced response time by almost three times and increased throughput by 2.6 times on the same cards.

The peak that previously spread across two nodes began to fit on one, and the model fit into two cards instead of four: the fleet for this scenario was halved without replacing hardware.

What did not work the first time

The first run was uniform INT4 across the entire network: no difference was seen on short questions, but on long contexts the model lost conditions from the middle of the dialogue and confused clauses of the procedure.

Operator-labeled answer evaluation dropped from 91% to 82% — more noticeably and faster than the speed drop.

We returned to the mixed scheme: calibration for group quantization was assembled from a month of real requests, critical layers were left in INT8, and a regression set was introduced that runs before every weight update. The result is 90% match with full-model answers at throughput 2.6 times higher than the original.

How does LLM quantization differ for a service and for a device?

The difference is not in which format is more accurate, but in which resource hits the limit first. Server deployment keeps weights in accelerator memory constantly and serves dozens of parallel requests — throughput and memory per session are critical.

Local deployment responds to one user in the device's shared memory pool, so what matters is not speed but the very ability to fit the model.

This determines the format choice. On a server, INT8 remains the baseline: memory for weights is halved with an almost imperceptible drop, and the freed volume goes to parallel sessions and long input sequences.

INT4 is used when the model does not fit otherwise — then the group size and calibration set are selected for the task.

On a device, the logic is reversed: first fit into memory, then look at quality. GGUF allows partial offloading of layers to the CPU — the model launches where it does not fit entirely, at the cost of latency.

The decision is made based on measurements on your own question set, not on tables from the internet: ready-made builds in the Hugging Face catalog save time, but quality loss is specific to each scenario.

Common technical doubts before starting

Before starting, the same questions come up: will answer quality drop, what to do when the model is updated, will the compressed version withstand peak load.

The weight format — INT8, INT4, GPTQ, AWQ, or GGUF — changes the compression method, but the doubts themselves remain common.

We validate the answers below on your data, not on averaged metrics.

What happens when the model is updated?

Compression is not transferred between versions: calibration coefficients are not suitable for new weights. An update is re-quantization and a regression set run that catches degradation before deployment to the production environment. Artifacts are versioned, so the previous build remains available for rollback.

How does the compressed model behave under peak load?

The memory gain provides headroom for the context cache and concurrent processing size, so throughput on one server increases. But there is a limit: when the queue overflows, response time grows. We capture the load profile and show the point after which a second model copy is added.

A full version is already deployed — will everything have to be redone?

The compressed model is placed alongside the full one, and traffic is switched gradually: first shadow mode with answer comparison, then a share of requests, then full migration.

Until matching on your data is confirmed, the full version remains in operation — rollback takes one step.

Let's discuss the task and return with an estimate and plan

To make the estimate substantive rather than "by eye," three things are enough to start: a request profile (task type, average and peak context length, ratio of generation to short answers), quality requirements (which answers are considered unacceptable and whether there is a reference question set for comparison), and hardware constraints — what the model should ultimately run on.

If you already have measurements of response time and model size, attach them: they remove an entire stage of coordination.

We return in response:

• task breakdown: where the bottleneck is — memory, response speed, or accuracy requirements — and which compression schemes fit it;
• stage-by-stage work plan: preparation of calibration data, selection of bit width, quality measurement before and after, implementation;
• timeline estimate and team composition for your volume;
• acceptance criteria — quality and response time metrics by which we compare options against each other;
• list of source data and access needed from your side;
• risks and rollback points: how to return to the original precision without rebuilding the environment from scratch.

Describe the task in free form — we will send a timeline estimate, work plan, and proposal for starting the project. A short description and one or two examples of real requests are enough.