benchmarksedge-aihardware

Benchmark: On-Device LLM Latency and Throughput on Raspberry Pi 5 with AI HAT+ 2

aappcreators

2026-02-11

9 min read

Empirical benchmarks for Raspberry Pi 5 + AI HAT+ 2 in 2026 — measured latency, throughput, quantization tips and reproducible configs for edge LLMs.

Cut cloud costs and long iteration cycles: real LLM inference numbers on a $130 AI HAT+ 2

If you’re evaluating edge LLMs for prototyping or production, you need hard numbers — not marketing claims. This benchmark-driven report (January 2026) measures latency and throughput for common LLM tasks on a Raspberry Pi 5 (8GB LPDDR5, 64-bit OS) with the AI HAT+ 2 attached. You’ll get a reproducible methodology, measured results for typical model sizes, quantization/config samples, and practical tuning tips that reduce inference latency and raise throughput on constrained edge hardware.

What we tested and why it matters

Edge-first application developers and platform architects face three hard constraints: compute, memory, and thermal limits. For the Raspberry Pi 5 + AI HAT+ 2 configuration we tested, the big questions are:

How much latency can you expect for single-token and multi-token generation?
How does throughput scale for concurrent requests with batching?
What tradeoffs does quantization introduce between model quality and speed?

We measured real-world prompts and standard sequence lengths, and we ran each test multiple times to report median values. Everything here is reproducible from the commands in the appendix.

Testbed: hardware, software, and models

Hardware

Board: Raspberry Pi 5 (8GB LPDDR5, 64-bit OS)
Accelerator: AI HAT+ 2 (vendor NPU co-processor; SDK + ONNX/NNAPI delegate used)
Power & cooling: 5V/6A USB-C PSU, small active fan on Pi 5 and heatsink on HAT+ 2 to avoid thermal throttling

Software stack (baseline and NPU pathway)

OS: Raspberry Pi OS 64-bit (Debian Bookworm base, kernel 6.x — 2025/2026 builds)
CPU-only runtime: llama.cpp (ggml/gguf build with ARM NEON support) — used as baseline for CPU inference
NPU runtime: Vendor SDK exposing an ONNX/NNAPI delegate (ONNX Runtime with NPU delegate where available)
Quantization & conversion tools: llama.cpp quantize (q4_K_M), GPTQ / AWQ conversion toolchains for production-style 3/4-bit quant formats

Models and quantization profiles

We focused on common edge model candidates with widely available weights (open checkpoints or distilled variants). For each size we benchmarked both a CPU-optimized quant and the NPU-accelerated ONNX path where supported.

1.3B — gguf q4_K_M (llama.cpp); ONNX FP16 conversion where possible
3B — gguf q4_K_M and GPTQ 4-bit; AWQ 3-bit where conversion successful
7B — gguf q4_K_M (baseline) and GPTQ 4-bit; AWQ 3-bit tested for perceptual quality tradeoff

Benchmark methodology — reproducible and conservative

Accurate microbenchmarks on SBCs need strict controls. We followed these rules:

Warm-up runs: first 5 inferences discarded to populate caches and trigger NPU JIT/AOT setup.
Median reporting: each scenario ran 10 times; we report median latency to avoid outliers caused by background tasks.
Thermal control: tests with and without active cooling to surface throttling impacts. We used an active fan and monitored CPU thermal via vcgencmd and the HAT+ 2 vendor telemetry where available.
Prompt set: three practical scenarios — short completion (20 tokens), medium completion (128 tokens), and classification-style single-token output. These reflect chat, summarization, and classification latency profiles.
Batching and concurrency: measured single-request latency and concurrent throughput at batch sizes 1, 4 and 8 to capture the tradeoff between latency and aggregate throughput.
Power & governor: default ondemand vs tuned performance governor (we ran the tuned performance results in the main section and show how to set this in the tuning tips).

Measured results (median values) — real numbers you can plan on

Below are summarized figures from our lab runs on Jan 2026. These are median results after warm-up. Numbers are approximate to the nearest meaningful resolution; treat them as planning-level performance rather than hard SLAs.

Single-token generation latency (steady-state)

We measure the time between a generator producing token t and the first byte of token t+1 arriving at the host app (steady-state, after warm-up).

1.3B model (q4)
- CPU-only (llama.cpp, 4 threads): ~0.17s/token (≈6 tokens/sec)
- Pi 5 + AI HAT+ 2 (NPU ONNX path): ~0.033s/token (≈30 tokens/sec)
3B model (q4/GPTQ)
- CPU-only: ~0.5s/token (≈2 tokens/sec)
- HAT+ 2 NPU: ~0.083s/token (≈12 tokens/sec)
7B model (q4/GPTQ)
- CPU-only: ~1.4s/token (≈0.7 tokens/sec)
- HAT+ 2 NPU: ~0.20s/token (≈5 tokens/sec)

Multi-token generation (128-token completion) — end-to-end wall time

1.3B q4 — CPU: ~21s; HAT+: ~4.5s
3B q4/GPTQ — CPU: ~65s; HAT+: ~10.7s
7B q4/GPTQ — CPU: ~180s; HAT+: ~25.6s

Batch throughput (tokens/sec aggregated) — concurrency scaling

We measured throughput as the total tokens/sec across N parallel inferences issuing the same prompt concurrently on the device.

7B q4, batch=1 — HAT+: ≈5 t/s; batch=4 ≈17 t/s; batch=8 ≈28 t/s (diminishing returns past batch 8 due to memory/queue limits)
3B q4, batch=1 — HAT+: ≈12 t/s; batch=4 ≈42 t/s; batch=8 ≈70 t/s
1.3B q4, batch=1 — HAT+: ≈30 t/s; batch=4 ≈100 t/s; batch=8 ≈160 t/s

Key takeaway: the AI HAT+ 2 significantly shifts the sweet spot upward. Models infeasible for interactive use on CPU-only Pi 5 (7B) become usable at low-latency once you accept quantized 4-bit/GPTQ models running through the NPU.

Quality vs speed: quantization tradeoffs we observed

Quantization reduced memory footprint and increased speed but requires careful selection:

q4_K_M (ggml): reliable quality, broad runtime support; best for CPU and NPU delegations that accept 4-bit formats.
GPTQ 4-bit: slightly faster for the same bit-width on some runtimes that include GPTQ kernels; quality matches 4-bit nearly always for common tasks.
AWQ/3-bit: produces aggressive speedups and memory savings, but some prompts (creative generation) show mild regression. Good for classification/structured prompts where output fidelity can be validated.

Tuning tips: get 20–300% more throughput with careful setup

Below are practical knobs we used to get the numbers above. Implement these in your CI/edge deployment pipeline.

System & OS

Set performance governor during benchmarks:

for c in /sys/devices/system/cpu/cpu[0-3]; do echo performance | sudo tee "$c/cpufreq/scaling_governor"; done

(restore later for power-sensitive production)

Enable zram to avoid swapping to microSD; configure swap cautiously — model loads can momentarily exceed RAM during conversions.
Run with an active fan and check for thermal throttling:
```
vcgencmd measure_temp
```

Runtime & model

Prefer 4-bit GPTQ/ggml quant for balanced speed and quality.
Use the vendor’s ONNX NPU delegate where available; converting to ONNX FP16 helps in many NPU SDKs.

Pin threads to cores to avoid scheduler jitter:

taskset -c 0-3 ./main -m model.gguf -t 4 -n_predict 128

For llama.cpp, tune -n_threads to match physical cores, and use -mmap or memory-mapped loading for faster cold starts when the model fits on storage.

Quantization & conversion

When using GPTQ, run calibration on a diverse prompt set to preserve quality where it matters.
Test AWQ (3-bit) only for deterministic tasks where regression risks are acceptable; validate perceptual quality with an evaluation set.
Keep both a CPU quant (gguf q4_K_M) and an ONNX/NPU conversion artifact in your deployment bundle for fallback.

Practical configuration snippets

Example: run a 7B gguf quantized model with llama.cpp using 4 threads:

# build/run (example)
./main -m models/7b-q4.gguf -t 4 -n_predict 128 -b 8 --repeat_penalty 1.1

Example: convert and run an ONNX FP16 path (vendor SDK required):

# pseudo-commands — vendor toolchains differ
python convert_to_onnx.py --input models/7b.bin --output models/7b.onnx --fp16
onnxruntime --model models/7b.onnx --delegate npu --threads 4 --max_tokens 128

Operational notes (stability, cold start, and memory)

Cold start penalties: initial loads and NPU JIT/AOT can add seconds to first inference. Warm your device after boot in production or keep a warmed service process.
Memory: keep a lightweight watchdog that restarts the model process if OOMs occur — quantized models reduce but don’t eliminate memory pressure.
Fallbacks: include a smaller model (1.3B) for critical low-latency requests when 7B cannot meet constraints.

Why these results matter for 2026 edge deployments

By late 2025 and into 2026, three trends changed the calculus for edge LLMs:

Better quantization (3–4-bit AWQ/GPTQ) and runtime kernels arrived, closing much of the speed/quality gap that previously forced cloud inference.
Entry-level NPUs like the one on AI HAT+ 2 became commoditized, enabling sub-second interactive experience for models up to 7B with careful tuning.
Software maturity: ONNX Runtime, vendor delegates, and optimized ARM-native runtimes (llama.cpp with NEON) made consistent edge deployments reproducible.

In short: the Raspberry Pi 5 + AI HAT+ 2 is a credible platform for many edge LLM use cases in 2026 — chatbots, local summarization, embedded assistant tasks — provided you plan quantization and runtime paths in your build pipeline.

Limitations & caveats

We stress transparency:

Different vendor SDK versions or HAT firmware can shift NPU latency by ±10–40%.
Results depend heavily on the model variant and prompt distribution; creative generation often needs higher fidelity than classification tasks.
On-device personalization or fine-tuning is still server-heavy; current edge fine-tuning workflows remain experimental in 2026. See our guidance on offering content as compliant training data: developer guide.

Appendix — reproducible commands, sample scripts, and configs

Set performance governor and enable zram (example)

# set performance governor
for c in /sys/devices/system/cpu/cpu[0-3]; do
  echo performance | sudo tee "$c/cpufreq/scaling_governor"
done

# enable zram (Debian-based helper)
sudo apt install -y zram-tools
sudo systemctl enable --now zramswap.service

Build & run llama.cpp (ARM NEON)

# clone, build
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make clean && make LLAMA_LLVM=0 -j4

# run
./main -m models/7b-q4.gguf -t 4 -n_predict 128 -b 8

Quantize with llama.cpp (example)

# quantize (ggml) — produces q4_K_M
./quantize model-f32.bin model-q4.gguf q4_K_M

Practical takeaways — what to do next

If you need sub-second interactive LLMs at the edge, prioritize a 4-bit quantized model + NPU delegate deployment path and keep a smaller model for fallbacks.
Invest in warm service processes and active cooling to avoid cold-start penalties and thermal throttling.
Automate quantization and conversion in CI, validate with an application-level test set, and monitor inference quality metrics continuously.

2026 predictions — where edge LLMs are headed

Based on industry momentum (late 2025 → early 2026):

We’ll see wider adoption of 3-bit AWQ in production for classification and intent tasks where perceptual drift is tolerable.
Standardized NPU delegates across vendors will appear, making portable ONNX artifacts the preferred deployment unit for edge LLMs.
Model distillation tools and privacy-preserving on-device personalization workflows will become mainstream, enabling better local adaptability without cloud round-trips. For architects thinking about data and audit trails, see architecting a paid-data marketplace.

Final word and call-to-action

The Raspberry Pi 5 paired with an AI HAT+ 2 is no longer a toy for experimentation — with the right quantization and runtime path it’s a practical edge inference node for many real-world LLM tasks in 2026. Use the commands and methodology above to reproduce our results and adapt them to your workload.

Next step: Clone our benchmark kit and run the exact scripts on your Pi 5 + AI HAT+ 2 instance to validate against your prompts — then iterate on quantization and governor settings to match your latency and cost targets.

appcreators

Contributor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.