Measurements

The first message and the next ones

Most of a commit is loading the model. Kept loaded (model.keep_loaded, on by default), only the first commit after a pause pays for it.

End to end through the CLI, on a Ryzen 5 5500 with a GeForce RTX 3050 8 GB:

A GPU spends most of the time loading, so it gains the most. The CPU spends it reading the prompt.

Five commits in a row

qwen2.5-coder-1.5b, five commits in a row. The first one starts the server (an estimate).

What it costs, and when it is not used

• It is kept only if it takes at most 15% of the RAM (model.keep_loaded_max_ram).
• The NPU and the Intel GPU have no server. On the NPU the first load compiles the graph (11.7s); later loads read the cache (1.7s, 246 MB).
• It stops when idle, or when a model.* setting changes.

pionus server status
# what is kept loaded, and how long it has been idle
pionus server stop
# free its memory now
pionus config set model.keep_loaded 30
# idle minutes; 0 turns it off

By device

On a Core Ultra 5 225H (Arc iGPU, NPU), per message, end to end. Here openvino-cpu was still called cpu.

The integrated GPU is the fastest. The NPU is the slowest but uses far less CPU.

On a Ryzen 5 5500 with a GeForce RTX 3050 8 GB, pionus bench on four prompts sized like real ones:

Most of the Vulkan load is copying the weights to the GPU.

By model

Per message on the Ryzen 5 5500 with a GeForce RTX 3050 8 GB. The fine-tunes write shorter replies, so they are faster.

The 7B model spends 7.8s of its 8.3s copying 4.4 GB to the GPU. Kept loaded, the next commit takes 0.57s.

Where the time goes

Reading the prompt dominates: about 90% of a message on the CPU. Milliseconds per message on a Core Ultra 5 225H (Arc iGPU, NPU), over 6 real prompts:

• One thread per core halves the prefill, so pionus sets model.threads_batch.
• Halving model.max_prompt_tokens halves the prefill, at a cost in quality.
• No gain from flash attention, weight repacking or speculative decoding.

Fast load

Two steps of starting llama-server are rarely needed for a 1 GB model: repacking the weights and fitting them to GPU memory. pionus bench keeps whichever is faster.

Asking for the type

The more confident the model is in the type, the more often it is right. Overall it matched the author 0.700 of the time, on 250 commits:

model.ask_type costs one more reading of the prompt per commit. More in the measurements document.

Machines and method

• Ryzen 5 5500 (6 cores), GeForce RTX 3050 8 GB with the proprietary driver 595.104, 23 GiB RAM, Pop!_OS 24.04, kernel 7.2.
• Core Ultra 5 225H (Arrow Lake-H: Arc iGPU, NPU 37xx), 15 GiB RAM, Pop!_OS 24.04, kernel 7.2.
• Your times will differ: pionus bench measures your machine.

How each number was measured, and how to repeat it, is in the measurements document.