Edge LLM inference: quantization and deployment on Raspberry Pi and Jetson

Deploying LLMs on edge devices: where inference breaks

Deploying LLMs on edge devices almost always breaks against the same thing: a model that responds instantly in the cloud hits a memory wall on the device.

The model doesn't fit in memory at full precision, part of the computation goes to swapping — and inference on the device grows by multiples, not percentages.

The second reason is thermal conditions. Under prolonged load, the accelerator throttles, and response latency on the device stops being predictable: the same request is processed sometimes quickly, sometimes with a noticeable drop. In the cloud this effect is smoothed by scale; on a single device it is visible immediately.

Next is memory for context. The cache grows with every dialogue message, and consumption grows with it: in a short test everything fits, but in a real scenario with long documents and chat history, the device starts evicting data to disk. Speed drops a second time, and already nonlinearly.

And the toughest constraint is operation without an external network. As soon as the channel to the data center disappears, the cloud loop stops responding entirely, while the local one must continue: in a warehouse, in the field, at a production site.

This is where it surfaces that the model is not quantized, the weight format doesn't match the accelerator, and the configuration was written for a test bench, not for the device.

We break down how this is closed: selecting bit depth and weight format, profiling the real load, measuring response time and memory consumption — and only then rollout with a predictable rollback to the previous build.

What local inference provides: latency, autonomy, privacy

Local inference removes the external channel and server-side queue from the response chain. Response time starts to be determined only by the device: predictable fractions of a second regardless of network load and how many requests run in parallel.

Response time variance is reduced by several times, and this is visible where the answer is embedded in a live process — the operator waits for a hint, the line waits for a decision, and any pause slows down the entire shift.

Autonomous operation of the model is the second consequence. The device continues computing when communication is lost, in a closed loop, at a field site without stable coverage.

This is the reduction of the failure class: instead of a complete service stop, degradation remains — the answer is shorter, some functions are unavailable, but the workflow does not stop and the dispatcher does not handle the incident manually.

Data privacy is covered by the same solution. Requests, documents, and processing results do not leave the device: they do not go to the external loop and do not settle in third-party logs.

For medical, HR, and production scenarios, this is often not a plus, but a condition under which the project is approved at all.

We keep the balance between quality and platform resources: the model is compressed, and accuracy on target tasks is verified before and after — we compute on compact platforms such as NVIDIA Jetson.

The question remains of the price of such autonomy: which model can actually be carried on the device and with what quality loss. We break it down below.

Comparison of LLM quantization modes: what suits your task

The compression mode is selected not by the principle "the more compact, the better," but by acceptable quality degradation on your set of requests.

In LLM quantization, three things diverge: the amount of memory occupied, quality on long contexts, and the ability to return to the original model without retraining.

Below are four modes that are actually chosen between when deploying LLMs on the edge: from single-board platforms such as Raspberry Pi and Jetson to industrial modules. The comparison is built on constraints, not promises.

We fix the choice by measurement on the target device: we run one set of requests in each mode and compare not only speed, but also the share of answers that passed semantic validation. We keep the full-precision configuration as a reference — rollback to it takes minutes.

Implementation stages: from model audit to handover to operation

The order of work is the same for Raspberry Pi and Jetson — only the depth of optimization for the target accelerator changes. We start with a model audit and finish with handover to operation: at each step, an artifact remains that can be checked.

1. Model audit. We compute the profile: context length, distribution of weights across layers, sensitivity of blocks to precision reduction. The output is a decision on what we convert to 8-bit format and what we leave at full precision.
2. Deployment scheme. We fix where the weights are stored, who initiates inference, how the device receives updates, and what it serves when communication with the server is lost.
3. Device image build. We build an image with pinned dependencies and the model inside — deployment comes from a single artifact, reproducibly and without manual steps on site.
4. Inference mode setup. We select batch size, number of threads, and distribution of layers between the accelerator and CPU so as to meet the response time requirement without overheating the enclosure.
5. Measurements. We capture time to first response and per subsequent token, memory consumption, and temperature under prolonged load — on a request profile close to production, not synthetic.
6. Validation on real data. We run a control set and compare quality before and after quantization by task metrics, not just by matching strings with the reference.
7. Handover to operation. Configurations, update scheme, rollback regulations to the previous image, and clear instructions for the on-duty shift.

What remains in hand: configurations, schemes, regulations

Deploying a model on a device is not just a running inference process. At the output, the customer receives a set of artifacts by which the solution can be reproduced, handed over to their team, and updated without our involvement. Below is exactly what is documented.

• Deployment configurations — versioned files with quantization parameters, memory limits for weights and buffers, number of concurrent requests, and accelerator operating mode. The configuration can be rolled back to the previous revision with one command.
• Data flow scheme — the path from the event source to the model response: where preprocessing happens, where the queue sits, what goes to the external loop, and what does not leave the device.
• Access policies — keys and roles for invoking inference, logging rules, and the list of fields that are masked before being written to the log.
• Model update regulations — the procedure for rolling out new weights, running on the control set, the threshold at which an update is blocked, and the procedure for returning to the previous build.
• Monitoring and thresholds — which metrics are captured continuously: response latency, board temperature, failure rate, quality divergence after retraining.
• Solution documentation — deployment scheme, dependencies, failure points, and step-by-step installation instructions on a clean device.

Each item is a working document, not a report for the sake of a report: from it, a new engineer brings up the bench, and operations understands what can be changed and when. Before handover, we go through this list together with your team — on a test device.

Case: local inference on a production line

The quality control line produced several dozen parts per minute, and each passed through a stream of frames from cameras. Inference ran in the cloud: the channel hit peak hours, about two-thirds of frames reached the model, and the verdict returned with a delay.

The operator manually rejected what the model had already seen — by that time the part had moved further down the line.

What changed

The model was moved to a compute module next to the conveyor — NVIDIA Jetson in an industrial version.

The weights were converted to integer formats and compressed: recognition quality was checked on a labeled sample of frames from the line, comparing predictions with the reference model for each defect class.

Inference was split across threads, frames were processed locally, and only disputed cases went to an engineer for review.

What became

Verdict latency dropped from seconds to fractions of a second — the decision arrives before the part leaves the control zone. The share of lost frames went to zero, and the failure class "verdict was late" closed entirely.

The line works without the cloud: if the external channel is lost, quality control does not stop, and model updates go on schedule during a downtime window with rollback to the previous build.

How quantization during training differs from post-training

The difference is at what stage the model learns about limited bit depth. With compression during training, the weights are immediately brought to the target precision, and training goes through rounding simulation — the network adapts and compensates for losses.

With post-training compression, the model is trained at full precision and then the weights are recalculated using a calibration sample: without retraining, only scale selection.

The choice of mode is set by two factors: quality headroom on the target set and frequency of retraining.

If fine-tuning is planned regularly, post-training compression costs less in support — but each weight update requires a new calibration, otherwise losses accumulate from iteration to iteration.

In practice, we start simple: first post-training compression as a baseline and measurement of drop on the validation set.

If the divergence from full precision exceeds the acceptable threshold, we move to training with rounding simulation — more expensive in time, but it removes degradation and simplifies repeated runs.

FAQ: memory, accuracy, model updates

Before starting a project, the customer usually has three technical doubts: whether the model will fit in the device memory, how much quality will drop after quantization, and how to update the fleet without stopping the line.

Below are short answers; we do the exact calculation on target hardware, not on a demonstration bench.

We fix the memory budget before deployment: weight size in the chosen format, peak consumption for intermediate tensors, I/O buffers, and system services.

If the model does not fit — we shorten the context, move sensitive layers to a more aggressive quantization scheme, or split inference across several nodes, for example on Jetson.

How is accuracy degradation measured after quantization?

We take a reference set of tasks from the domain and compare the answers of the full and compressed models by task metrics: share of correct extractions, answer completeness, format stability. We agree on the acceptable divergence in advance — usually a few percent.

How to update a fleet of devices and roll back a version?

We roll out the model in waves: first a canary group, then the rest. Two model artifacts are stored on the device, and switching is done by version tag — so rollback takes seconds and does not require reflashing.

What is included in post-launch support?

Monitoring quality drift and memory consumption, scheduled model updates, regulations for responding to degradation, and replacement of retired devices.

Let's discuss the task: we'll select a mode and assemble an implementation plan

So that the project estimate is based on facts, not on "plus or minus," we need input data along four axes: model, scenario, device fleet, and constraints.

Describe the task in free form — the more specific the inputs, the more accurately we will name the quantization mode, target weight format, and scope of work.

• Model. Architecture and size, license for commercial use, acceptable quality degradation after compression, context length requirements.
• Scenario. Which requests are processed, whether a stream or queue is needed, what response latency is acceptable, whether there is communication with external services.
• Device fleet. How many nodes, which platform classes, RAM size, whether there is a hardware accelerator.
• Constraints. Power and thermal conditions, enclosure, autonomous operation without network, procedure for updating the model on nodes.
• Integrations. What we connect inference to: sensors, local services, exchange bus, where the result goes.
• What already exists. Prototypes, measurements, current bottlenecks — this saves iterations when selecting a configuration.

In response, we send not general words, but an implementation plan: a quantization scheme for your device class, the target build, a pilot plan on one device, and acceptance metrics.

A pilot on one device answers the main question — whether the model maintains the required quality at the given speed and thermal conditions — after which the configuration is replicated across the entire fleet.