Generation Speed

[CarlosS7]

Device & OS

• Hardware: Raspberry Pi 3B+
• OS: Raspberry Pi OS 64-bit, Debian 1:6.12.62-1+rpt1 (2025-12-18) aarch64 GNU/Linux
• Compiler: gcc 14.2.0

Model

• Model file: tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf
• Quantization: Q4_K_M

What happened?
I am getting nowhere near the 4 tk/s for the Raspberry Pi 3B+

Command you ran

picolm models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf -p "The capital of France is" -n 10 -t 0 -j 4

Expected output
Generation speed that's close to 4 tk/s

Actual output

Model config:
  n_embd=2048, n_ffn=5632, n_heads=32, n_kv_heads=4
  n_layers=22, vocab_size=32000, max_seq=2048
  head_dim=64, rope_base=10000.0
Allocating 1.17 MB for runtime state (+ 44.00 MB FP16 KV cache)
Tokenizer loaded: 32000 tokens, bos=1, eos=2
Prompt: 6 tokens, generating up to 10 (temp=0.00, top_p=0.90, threads=4)
---
 Paris.

2. B.C. The
---
Prefill: 6 tokens in 166.62s (0.0 tok/s)
Generation: 11 tokens in 278.45s (0.0 tok/s)
Total: 445.07s
Memory: 45.17 MB runtime state (FP16 KV cache)

[zoor29]

I can only agree. I have the same setup (only difference is Raspian is used in my case).

picolm is REALLY slow. I hope its the setup on my side...

I've opened htop at the same time just to see if it drains resources but thats not the case.

CPU (all 4 threads) is at around 6%-8%. Memory is at approx 278 MB out of 906 MB.
Swap = 8,5 MB.

[gunakkoc]

Same for my Rasberry Pi 4B with 4 GB RAM.

Instead of the advertised ~8 tokens/s I am getting 1.7 tokens/s. Using llamafile works much better at the moment.

I also checked temps and there is no thermal throttling.