servingmemoryvLLMQwen3.5-4BDraft

Where the GPU memory goes

Calculate the memory needed for model weights and active requests, then use vLLM's logs and metrics to see what fits on an A10.

Daniel MolenaarsQwen

Beginner·45–60 minutes·Updated Oct 8, 2026

Recommended setup

Model

Qwen/Qwen3.5-4B

Hardware

1x NVIDIA GPU, 24 GB VRAM

Engine

vLLM 0.27.0

Benchmark

EvalScope 1.9.1

Python

Python 3.12

Tested environment

Hardware

1x NVIDIA A10, 24 GB VRAM

System

Ubuntu 22.04 container

Driver

NVIDIA 580.105.08

In the previous recipe, we saw requests start to queue as we increased concurrency. We will now look at the memory those requests need: this helps us understand how many can run together, and what happens when we give the server a longer prompt.

Prerequisites

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install 'vllm==0.27.0'

source .venv/bin/activate

hf download Qwen/Qwen3.5-4B \
  --local-dir ./models/qwen3.5-4b

Start with the weights

The first recipe estimated the space needed for the model's weights. We can calculate this by multiplying the number of parameters by the bytes used to store each one. With BF16, each parameter takes two bytes.

For Qwen3.5-4B, the checkpoint contains roughly 4.66 billion parameters, giving us about 9.32 GB of stored tensors. The name “4B” gives us a useful starting point, but the checkpoint includes several components: the language model, a vision encoder and a module for predicting additional tokens.

You can inspect the checkpoint's recorded tensor size without loading the model:

python - <<'PY'
import json
from pathlib import Path

index = json.loads(
    Path("models/qwen3.5-4b/model.safetensors.index.json").read_text()
)
size = index["metadata"]["total_size"]
print(f"Checkpoint tensors: {size / 1e9:.2f} GB")
print(f"Checkpoint tensors: {size / 2**30:.2f} GiB")
PY

For this checkpoint, the output is:

Checkpoint tensors: 9.32 GB
Checkpoint tensors: 8.68 GiB

Both lines describe the same amount of memory. GB uses a billion bytes, while GiB uses 1,073,741,824 bytes. vLLM's memory logs use GiB, so in the future we will only use this unit for our calculations.

Understanding the server's memory profile

Run the following command:

vllm serve ./models/qwen3.5-4b \
  --served-model-name qwen3.5-4b \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 2048 \
  --dtype bfloat16 \
  --no-enable-prefix-caching

Go back to the terminal running vLLM and look for these messages during startup:

1. Model loading took ... GiB memory shows the memory attributed to loading the model.
2. Available KV cache memory shows the budget vLLM has calculated for the request-state pool.
3. Graph capturing finished ... took ... GiB reports memory used when preparing CUDA graphs. These help the server replay GPU work efficiently.
4. GPU KV cache size and Maximum concurrency for ... tokens per request describe the capacity calculated by the engine.

The following excerpts show the output to look for with this model and A10 configuration. Timestamps and process prefixes are omitted; memory and timing can vary slightly between runs.

Model loading took 8.61 GiB memory and 1.703832 seconds
Available KV cache memory: 8.88 GiB
GPU KV cache size: 273,563 tokens
Maximum concurrency for 32,768 tokens per request: 8.35x
Graph capturing finished in 5 secs, took 0.54 GiB

With this command, the server loads the language and vision weights, using about 8.46 GiB for parameter tensors. Good to note is that even though our requests contain only text, the default model still loads its vision encoder (which you can optionally disable with the --language-model-only flag to save on more memory).

As you can see above, the startup log reports 8.61 GiB for model loading. This also includes other memory allocated during the loading process.

Keep the vision encoder enabled to follow the memory figures in this recipe.

vLLM also profiles activations: intermediate results produced while running the model. For example, a prompt must pass through each layer before we can generate its answer, and those calculations need temporary space. Larger batches can increase that requirement.

Note the two explicit scheduler settings:

• --max-num-seqs 256 limits the number of sequences the scheduler can process in a step. Lowering it can reduce the number of requests competing for cache space, but can also leave more requests waiting in the queue.
• --max-num-batched-tokens 2048 limits the tokens processed in that step, including prompt and decoding work. A larger budget can improve prompt-processing throughput; a smaller budget can shorten the time that ongoing decoding waits for prompt work. The effect depends on the workload.

Prefill Activation Memory vs. KV Cache Memory

Let's take a closer look at activation memory and how it relates to the KV cache memory. When vLLM starts up, it loads the model weights, profiles working memory and accounts for runtime overhead. It uses the remaining space within the configured memory budget for the cache. Two parts of this memory use are:

With Qwen3.5, the request-state pool also holds DeltaNet state. We will calculate both kinds of stored state below.

Why is there a trade-off?

As we will see below, vLLM allocates its cache pool at startup. It profiles working memory to leave room for the configured batch sizes before sizing that pool. This reduces the risk of running out of memory (OOM), although profiling cannot guarantee that every workload will fit.

To see how they interact, look at what happens during a single prompt's prefill phase, when its tokens fit in the available budget for one step:

1. You send a prompt of 1,000 tokens.
2. The engine executes a forward pass. The massive math required to process all 1,000 tokens simultaneously requires Activation Memory.
3. As the model processes these layers, it generates the historical keys and values for those 1,000 tokens. These are saved into the KV Cache so they don't have to be recalculated later.
4. The step finishes. Temporary Activation Memory can be reused for the next batch, while the KV Cache retains the stored token information. GPU memory readings can stay high because the allocator keeps memory reserved.

If you raise --max-num-batched-tokens, the profiled working-memory requirement can grow and leave less space for the KV cache. A larger token batch can improve prefill throughput, but a smaller cache can mean fewer active requests or more preemptions, where a request releases its cache and later repeats work.

If you lower it, the working-memory requirement can shrink and leave more space for the cache. Long prompts may need more processing steps because vLLM splits them into smaller chunks.

Keep both scheduler settings fixed for the rest of this recipe. This lets you compare memory budgets and request lengths with the same scheduler configuration.

Tweaking vLLM's memory budget

Our server uses --gpu-memory-utilization 0.90, allowing the executor to budget around 90% of the device memory visible to CUDA. This includes the weights and runtime work as well as the cache.

For the A10 configuration shown here, CUDA reports 22.06 GiB of total memory. Use the capacity reported by your runtime when checking this budget; the GPU's advertised 24 GB and its usable capacity use different units and can also differ because of device reservations.

Stop the server with Ctrl+C and repeat the server command above with --gpu-memory-utilization 0.70, then --gpu-memory-utilization 0.80. Keep the other settings the same. Each startup should print a new cache budget and capacity estimate.

Compare the cache-budget messages from each startup (timestamps and process prefixes omitted):

# --gpu-memory-utilization 0.70
Available KV cache memory: 4.47 GiB

# --gpu-memory-utilization 0.80
Available KV cache memory: 6.68 GiB

# --gpu-memory-utilization 0.90
Available KV cache memory: 8.88 GiB

The chart shows the resulting allocations once the server is ready. The allocated pool can be slightly smaller than the logged budget because it must fit a whole number of blocks.

At 70%, the chart shows a cache pool of 4.46 GiB. At 80%, it grows to 6.67 GiB, and at 90% it reaches 8.88 GiB. The weights stay at the same size throughout, so the additional budget gives the server substantially more room for request state!

The total device usage after startup is about 14.92, 17.13 and 19.35 GiB, respectively. The remaining portion includes CUDA graphs, working buffers, allocator reservations and other runtime memory. This explains why estimating the weights alone leaves us well short of the full serving requirement!

How much memory does a request need?

The weights are shared by all requests, but you can imagine that each active request also needs its own state so that the model can continue generating the answer. A layer is one processing stage in the model that contains this request-specific state. For Qwen3.5, there are two pieces to calculate: full attention layers, and Gated DeltaNet layers.

Full-attention K/V cache

In the previous recipe, we followed an attention layer as it created keys and values for each token. The model keeps those vectors so that later tokens can use the information already processed.

Qwen3.5-4B has eight full-attention layers in total. Each uses four key/value heads with 256 numbers per head. We store both a key and a value, with two bytes per number at BF16:

K/V bytes per token = layers × 2 × K/V heads × head size × bytes
                   = 8 × 2 × 4 × 256 × 2
                   = 32,768 bytes
                   = 32 KiB

These values can be looked up in the model's configuration:

{
  "text_config": {
    "head_dim": 256,
    "num_key_value_heads": 4
  }
}

A sequence containing 8,192 tokens, for example, therefore needs 256 MiB for its full-attention K/V vectors. The bigger our context, the bigger the space required to store our token sequences, and the smaller the total amount of sequences shared across users can be stored simultaneously.

DeltaNet state

Qwen3.5-4B's other 24 layers use Gated DeltaNet. These layers update a compact recurrent state as tokens pass through them. You can think of this as a working summary that is updated as the model reads and writes. The interesting part is that the size of this state stays fixed as a single request grows!

These values can again be looked up in the model's configuration:

{
  "text_config": {
    "linear_num_value_heads": 32,
    "linear_value_head_dim": 128,
    "linear_num_key_heads": 16,
    "linear_key_head_dim": 128,
    "mamba_ssm_dtype": "float32",
    "linear_conv_kernel_dim": 4
  }
}

Let’s unpack what these variables mean. Each DeltaNet layer divides its recurrent state across 32 value heads, which process information in parallel.

Each head then keeps a matrix: a grid with 128 rows and 128 columns, containing 16,384 numbers. As the layer processes another token, it updates the numbers in that grid.

The model’s trained weights determine how these updates happen, and the state holds information accumulated while processing the current request, so each active request needs its own copy.

To calculate the memory needed, we therefore count the numbers and multiply these by their storage size. In our configuration, the recurrent state uses a 32-bit floating point number (float32).

Recurrent state per layer
    = 32 heads × 128 rows × 128 columns × 4 bytes (32 bits)
    = 2,097,152 bytes
    = 2 MiB

Each DeltaNet layer also keeps a small rolling buffer. The linear_conv_kernel_dim property is 4, so its convolution combines information from the current token with the three previous tokens: 4 − 1 = 3 previous positions to retain.

Our buffer uses BF16, which stores each number in two bytes.

For each previous token, the buffer stores 8,192 intermediate numbers. These are the query, key and value representations produced inside the layer:

Numbers per position
    = query numbers + key numbers + value numbers
    = (16 × 128) + (16 × 128) + (32 × 128)
    = 8,192

Buffer size
    = (4 − 1) previous positions × 8,192 numbers × 2 bytes
    = 49,152 bytes
    = 48 KiB

As the model processes another token, the buffer replaces its oldest entry. Its size stays fixed, giving us another 1.125 MiB per request across the 24 DeltaNet layers.

Across all 24 DeltaNet layers, we get:

Recurrent matrices    = 24 × 2 MiB  = 48 MiB
Convolution buffers   = 24 × 48 KiB = 1.125 MiB

Total DeltaNet state per request    = 49.125 MiB

A longer prompt changes the contents of this memory, but its size stays the same! A second active request will need exactly another 49.125 MiB for its own DeltaNet state.

We can now put the two pieces together:

Request state ≈ stored tokens × 32 KiB + 49.125 MiB

For an 8,192-token sequence, the result is about 305 MiB. Four such sequences need about 1.19 GiB of stored values. Try changing the number of active requests below:

As you can see, longer sequences increase the full-attention cache, while the DeltaNet portion stays the same for a given number of requests. Increasing the request count multiplies both pieces.

You can repeat the numbers for your own workload with this small Python script:

stored_tokens = 8192  # Prompt plus answer tokens stored so far
active_requests = 4

kv_bytes = stored_tokens * 8 * 2 * 4 * 256 * 2
recurrent_bytes = 24 * (32 * 128 * 128 * 4 + 3 * 8192 * 2)
request_bytes = active_requests * (kv_bytes + recurrent_bytes)

print(f"Stored request state: {request_bytes / 2**30:.2f} GiB")

Understanding how vLLM allocates memory

vLLM allocates memory space in blocks, allowing requests to acquire and release pieces of the cache as they run.

As we've seen, Qwen3.5 needs room for both full attention and recurrent state. The engine groups these layers and adjusts their allocation sizes to share the cache pool. We can inspect the resulting block size in the startup logs:

...

Setting attention block size to 528 tokens to ensure that
attention page size is >= mamba page size.

Padding mamba page size by 0.76% to ensure that mamba page size
and attention page size are exactly equal.

...

During startup, vLLM automatically chooses an allocation layout that accommodates both the attention cache and DeltaNet state. With this configuration, it selects 528-token attention blocks.

Across the eight full-attention layers, each block takes 16.5 MiB (32 KiB x 528). vLLM also adds a small amount of unused space, called padding, to align the DeltaNet allocation (referred to as 'mamba'), with the attention allocation. This brings the DeltaNet state to 49.5 MiB per active request.

Let's give a simple example: imagine that we have a request with 640 tokens, which exceeds the attention block size allocated by vLLM. As a result, vLLM will allocate not one, but two memory blocks for this request! The total memory used will therefore be as follows:

Attention cache = ceil(640 / 528) × 16.5 MiB = 33 MiB
DeltaNet state  = 49.5 MiB
Total          = 82.5 MiB per request

Watch the cache fill

Let's put these numbers into practice: we will now use EvalScope to increase concurrency while watching the cache. First, restart the server with the original 0.90 memory setting and 32768 context limit.

From another terminal, you can watch the cache and request counts update:

watch -n 1 "curl --silent http://127.0.0.1:8000/metrics | grep -E '^vllm:(kv_cache_usage_perc|num_requests_running|num_requests_waiting)\{'"

The cache counter is a fraction from zero to one; multiply it by 100 for a percentage. The other counters show running and waiting requests. Leave this monitor running. In a third terminal, activate the benchmark environment from the previous recipe and run:

source .bench-venv/bin/activate

evalscope perf \
  --model qwen3.5-4b \
  --url http://127.0.0.1:8000/v1/chat/completions \
  --api openai \
  --dataset random \
  --tokenizer-path ./models/qwen3.5-4b \
  --min-prompt-length 512 \
  --max-prompt-length 512 \
  --prefix-length 0 \
  --min-tokens 128 \
  --max-tokens 128 \
  --temperature 0 \
  --seed 42 \
  --num-workers 1 \
  --parallel 1 4 16 64 \
  --number 8 8 32 64 \
  --warmup-num 4 \
  --in-flight-task-multiplier 1 \
  --total-timeout 300 \
  --read-timeout 300 \
  --connect-timeout 10 \
  --stream \
  --extra-args '{"ignore_eos": true, "chat_template_kwargs": {"enable_thinking": false}}' \
  --name qwen3.5-4b-memory-p512 \
  --outputs-dir ./memory-results

Repeat the benchmark with both prompt-length flags set to 8192, then 16384, optionally giving each run a different --name if you want to store the results. The results will likely be like below:

Starting with Cache used, we can see that for 512-token prompts, usage grows from about 0.9% at concurrency one to 58.2% at concurrency 64. Our estimate is 82.5 MiB per request. At 64 active requests, that becomes 5.16 GiB, which aligns with the cache usage shown in the chart. Remember that the total cache pool is 8.88 GiB?

Longer prompts fill this pool much sooner. At concurrency 16, the 8,192-token prompts use 55.3% of the cache, and 16,384-token prompts already reach 95.5% utilization at this level!

Switch to Running requests to see why. With the longest prompts, the chart shows at most 15 requests running together at both concurrency 16 and 64. Each needs about 577.5 MiB once its prompt and answer are stored. Sixteen would need 9.02 GiB, exceeding the available pool. The server holds the remaining requests in its queue and admits them as space becomes available, which we already went over in the previous recipe.

Next, switch to Waiting requests. Some waiting also appears with short prompts, even when there is room in the cache, as the scheduler still has to process the arriving prompts within its 2,048-token per-step budget. The 256-sequence limit is an upper bound; the available cache and token budget can keep the actual running count much lower.

Finally, select GPU memory. All three lines stay close to 19.35 GiB, even as the cache fills! The reason for this is that vLLM already allocated its pool at startup, and requests only use and release blocks inside that allocation.

The chart uses server samples every 250 ms and GPU samples about once per second. The one-second monitor above gives you a live view, but can miss brief peaks. Each plotted peak can occur at a different instant.

Changing the context limit

--max-model-len limits the combined prompt and generated answer for one request. Setting it higher allows the server to accept longer requests. When requests actually use that extra length, they need more cache and fewer can fit together!

Stop the server and change --max-model-len in the server command to 8192, 32768 and 65536 in turn, keeping the memory setting at 0.90. Check the cache budget, capacity estimate and GPU usage after each startup.

In a second terminal, check the GPU usage with:

nvidia-smi --query-gpu=name,memory.total,memory.used \
  --format=csv

Compare these startup messages for each context limit (timestamps and process prefixes omitted):

# --max-model-len 8192
Available KV cache memory: 8.87 GiB
Maximum concurrency for 8,192 tokens per request: 28.95x

# --max-model-len 32768
Available KV cache memory: 8.88 GiB
Maximum concurrency for 32,768 tokens per request: 8.35x

# --max-model-len 65536
Available KV cache memory: 8.88 GiB
Maximum concurrency for 65,536 tokens per request: 4.30x

With this configuration, expect the cache pool to stay close to 8.88 GiB at all three limits. Device memory after startup should also stay close to 19.35 GiB.

The capacity estimate changes: vLLM reports room for about 29 full-length requests at 8K, 8 at 32K and 4 at 64K. Each estimate assumes that requests reach the configured maximum. These are memory-capacity estimates; the workload and scheduler also affect how many requests run together.

Repeat the same short requests

After each restart, run the monitor and repeat the earlier EvalScope command, replacing the corresponding options with:

--min-prompt-length 512 \
--max-prompt-length 512 \
--parallel 4 \
--number 8 \
--name qwen3.5-4b-memory-short-control

Keep the output length at 128 tokens. In the monitor, look for four running requests and cache usage near 0.0364, or 3.64%. The same short requests occupy about 330 MiB at each context limit:

Four requests × (two attention blocks × 16.5 MiB + 49.5 MiB DeltaNet state)
    = 4 × 82.5 MiB
    = 330 MiB

Raising the allowed context length leaves these requests' memory needs unchanged because their actual lengths stay the same.

Recover from a memory misconfiguration

vLLM normally estimates the cache budget for us. However, it can happen that it runs out of memory, for example through any manual flags that we have set.

One of these flags is --kv-cache-memory-bytes, which lets us choose an explicit cache size. A cache setting equal to the entire GPU's capacity will therefore ask for more than the device can provide.

Stop the running server with Ctrl+C, then try adding the following flag to the vLLM startup command:

--kv-cache-memory-bytes 25769803776

This requests 24 GiB for the cache alone. On the A10 configuration used here, this exceeds the available memory. Expect an allocation error like the following; the exact failed allocation and free-memory figures can vary:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.00 GiB.
GPU 0 has a total capacity of 22.06 GiB of which 1.01 GiB is free.

To recover from an error like this, you can do the following:

1. Stop the failed server and check nvidia-smi to confirm its memory has been released.
2. Remove --kv-cache-memory-bytes and restart with the original 0.90 utilization setting and 32768 context limit.

Once restarted, your server should return to an 8.88 GiB cache pool. Repeat the short-request command above to check that it can answer requests again.

Wrapping up

That was a lot of arithmetic! You can now make a more useful memory estimate before choosing a GPU or increasing the load:

1. You counted the model weights and compared them with the server's loaded memory.
2. You calculated the full-attention cache and recurrent state needed by active requests for models which adopt the Qwen3.5 architecture.
3. You watched the cache fill as prompt length and concurrency increased.
4. You changed the memory and context settings and recovered from an allocation failure.

Next, we will use our endpoint to build a classification model. We will start by giving Qwen a question and a set of possible answers, then measure how reliably it chooses between them.