Skip to content
servingvLLMQwen3.5-4B

Getting started with self-hosting Qwen

Serve Qwen3.5-4B behind a local OpenAI-compatible endpoint on one NVIDIA GPU, with pinned software and a clear VRAM budget.

Daniel MolenaarsQwen

Beginner30 minutesUpdated Oct 8, 2026

Model
Qwen/Qwen3.5-4B
Hardware
1x NVIDIA GPU, 24 GB VRAM
Engine
vLLM 0.27.0
Python
Python 3.12

Tested environment

Hardware
1x NVIDIA A10, 24 GB VRAM
System
Ubuntu 24.04 x86_64
Driver
NVIDIA 580.126.09
On this page
What you'll do
  • Choose a practical GPU and context budget for serving Qwen3.5-4B.
  • Start vLLM and verify a local OpenAI-compatible endpoint.

Size the GPU

Build a VRAM budget

For every model that you choose, you need to assess how much VRAM you need. The following components provide some guidance on a first estimate:

  1. Weights. These are the model's parameters, which are loaded once at startup. We can calculate this as follows: parameter count times bytes per parameter. Qwen3.5-4B has roughly 4.66 billion parameters: at BF16 (16 bits, or 2 bytes) that costs about 9.3 GB (4.66 billion * 2 bytes ≈ 9.3 GB). 8-bit quantization will cut that in half to 4.65 GB, and 4-bit quantization will reduce it to only 2.32 GB.
  2. Key-Value (KV) cache. These are cached tokens that are accumulated over time. It grows with context length and with concurrency.
  3. Overhead. This consists of activations, CUDA context, engine workspace.

For our baseline the prediction is therefore: about 9.3 GB of weights, plus the KV cache for our 32.768-token context budget, plus overhead.

Count the model parameters and weight memory

You can count the exact parameters and required VRAM for the model weights using the following Python script:

Python
from transformers import AutoModelForCausalLM
import torch

model_id = "Qwen/Qwen3.5-4B"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16)

# 1. Precise parameter count
total_params = sum(p.numel() for p in model.parameters())
print(f"Total parameters:     {total_params:,}")

# 2. Actual weight memory
weight_bytes = sum(p.numel() * p.element_size() for p in model.parameters())
print(f"Weight VRAM (fp16):   {weight_bytes / 1e9:.2f} GB")

Estimate with whichllm

To make things easier, you can use whichllm, a CLI tool that estimates a model's VRAM footprint based on the estimated weights, KV cache, activations, overhead, plans hardware for a given model.

Execute the below command to run whichllm. Note that you need to have uv installed. For more information on installing uv, see here.

Bash
# What GPU does Qwen3.5-4B need at our context budget?
uvx whichllm plan "Qwen/Qwen3.5-4B" \
  --context-length 32768 \
  --refresh

You'll get something like the following:

Output
╭───────────────── Model Info ─────────────────────╮
│ Model:  Qwen/Qwen3.5-4B                          │
│ Params: 4.7B | Arch: qwen3_5 | Context: unknown  │
│ License: apache-2.0                              │
╰──────────────────────────────────────────────────╯
     VRAM Required (context: 32768)
┏━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Quant    ┃       VRAM ┃ Quality Loss ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ Q2_K     │     4.2 GB │         -25% │
├──────────┼────────────┼──────────────┤
│ Q3_K_M   │     4.7 GB │          -8% │
├──────────┼────────────┼──────────────┤
│ Q4_K_M ★ │     5.3 GB │          -5% │
├──────────┼────────────┼──────────────┤
│ Q5_K_M   │     5.8 GB │          -3% │
├──────────┼────────────┼──────────────┤
│ Q6_K     │     6.3 GB │          -2% │
├──────────┼────────────┼──────────────┤
│ Q8_0     │     7.4 GB │          -1% │
├──────────┼────────────┼──────────────┤
│ F16      │    11.5 GB │           0% │
└──────────┴────────────┴──────────────┘

The Quant column indicates the quantization level of the model. For now, we'll focus on deploying F16: the unquantized, full-precision 16-bit model. You'll notice that the VRAM estimation is higher than our earlier 9.3 GB. This is because whichllm takes into consideration our context length, the KV cache size associated with that context length, as well as any overhead.

Warning

Always pass the full repo ID (such as Qwen/Qwen3.5-4B) and add the --refresh flag before trusting the output. whichllm keeps a local model index which might not contain your desired model by default, and as a result it may pick a different model completely.

Install and validate the runtime

A typical vLLM installation has three layers:

  1. The NVIDIA driver. This determines the highest CUDA runtime version the machine can support. It needs to be installed separately from vLLM.
  2. The CUDA runtime bundled with PyTorch. The normal vLLM wheel brings a compatible PyTorch build and CUDA runtime, so you typically will not have to worry about installing a runtime separately.
  3. vLLM. This provides the model engine and OpenAI-compatible HTTP server, and sits on top of torch.

First, check if your current provider already pre-installed the driver for you. Most platforms will already have done this for you. You can verify this with the following command:

Bash
nvidia-smi

The version you'll see here will determine the CUDA ceiling: the maximum version of CUDA your driver supports. You will see something like this:

Output
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.09             Driver Version: 580.126.09     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A10                     On  |   00000000:00:07.0 Off |                    0 |
|  0%   28C    P8             10W /  150W |       0MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

In the above example, we see that our driver's major version is 580, and our CUDA version is 13.0. The CUDA value printed by nvidia-smi is the newest CUDA compatibility level supported by the installed driver.

If your GPU driver is not yet installed, you need to do it yourself. You can view which version you should be using by going to the Drivers page of NVIDIA and using the Driver Search tool to quickly verify which driver will work for your card.

Once you have verified the driver is installed, create a clean Python environment and install vLLM:

Bash
python -m venv .venv
source .venv/bin/activate
python -m pip install vllm==0.27.0

Then inspect all three layers:

Bash
vllm --version
python -c "import torch; print(torch.version.cuda)"
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv

The PyTorch CUDA runtime must be supported by the installed driver, so ensure that the CUDA runtime version emitted by PyTorch is equal to or lower than the driver's supported version.

Start the server

Start the local vLLM server:

Bash
vllm serve Qwen/Qwen3.5-4B \
  --served-model-name qwen3.5-4b \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --dtype bfloat16

This what we are telling vLLM:

  • --served-model-name qwen3.5-4b creates a stable API alias. This is the model name clients send in requests.
  • --max-model-len 32768 caps the combined prompt and output length. Our 3.5-4B model can actually support much longer contexts, but this will also increase the size of the KV-cache!
  • --gpu-memory-utilization 0.90 lets this vLLM instance use up to 90% of GPU memory for the model executor. vLLM profiles the model and uses the available portion for weights, runtime work, and cache allocation.
  • --dtype bfloat16 informs vLLM that the model's precision is 16-bit.

Wait for the startup logs to settle, this may take several minutes. A successful server eventually reports something like the following:

Output
(APIServer pid=3938) INFO [launcher.py:105] API server: HTTP server started

In a second terminal, confirm that the server advertises the alias and model you expect:

Bash
curl http://127.0.0.1:8000/v1/models

The response should contain these values:

JSON
{
  "object": "list",
  "data": [
    {
      "id": "qwen3.5-4b",
      "root": "Qwen/Qwen3.5-4B",
      "max_model_len": 32768
      ...
    }
  ]
}

The actual response contains additional fields, but the important thing is that id matches the served alias you specified above.

Read the memory profile

While you are here, you can now verify the exact amount of VRAM being utilized. In the server startup logs, look for the following output logs:

  1. Model loading took X GiB memory and Y seconds. This is the loaded weights footprint.
  2. Actual usage is X GiB for consumed memory (weights + non-torch), Y GiB for peak activation, and Z GiB for CUDAGraph memory. The memory usage of weights, peak activations, and CUDA graphs.
  3. Available KV cache memory: X GiB. What's left for KV after weights + activations + graphs.
  4. GPU KV cache size: X tokens. The allocated token capacity for the full-attention cache. This is the unit you can use to determine your maximum cache size.
  5. Maximum concurrency for X tokens per request: ...x. How many full-context requests theoretically fit at once.

Dissect a request

Qwen3.5 thinks by default. Disable thinking through the chat-template option in the request:

Bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5-4b",
    "messages": [
      {
        "role": "user",
        "content": "Explain in one sentence why KV cache memory grows with context length."
      }
    ],
    "max_tokens": 128,
    "temperature": 0.0,
    "chat_template_kwargs": {
      "enable_thinking": false
    }
  }'

The response comes back in an OpenAI-compatible shape:

JSON
{
  "id": "chatcmpl-96e4d1659944adbb",
  "object": "chat.completion",
  "created": 1786453580,
  "model": "qwen3.5-4b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "KV cache memory grows with context length because the model must store the entire sequence of token embeddings and attention weights for every token in the input sequence simultaneously to perform efficient attention and matrix multiplication operations.",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "reasoning": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null,
      "token_ids": null,
      "routed_experts": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": "vllm-0.27.0-70c2f18b",
  "usage": {
    "prompt_tokens": 30,
    "total_tokens": 69,
    "completion_tokens": 39,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "prompt_text": null,
  "kv_transfer_params": null,
  "ec_transfer_params": null,
  "metrics": null
}

Here are some interesting response fields to take note of:

  • model should be qwen3.5-4b, the alias configured on the server.
  • choices[0].message.content contains the direct answer to our question.
  • choices[0].message.reasoning should be absent or null because thinking was disabled.
  • choices[0].finish_reason is normally "stop", like in the above response; "length" means generation reached max_tokens before finishing.
  • usage reports the prompt, completion, and total token counts for this request. We will use this later to measure metrics such as throughput.

What happened under the hood

When serving a request like ours, a few things happen under the hood. For now, it's important to distinguish between the two phases that take place, prefill and decoding:

  • Prefill essentially consists of reading and understanding the entire question prompt at once. In our case, that is Explain in one sentence why KV cache memory grows with context length.The goal is to build a complete internal understanding of what you asked, and creates the initial KV cache that will be reused later. The speed of prefill is compute-bound. It processes many tokens at once using massive matrix multiplications, and fully utilizes the GPU's compute power. This phase determines the time to first token (TTFT), the very first output token of the model. The bigger your input, the more computation will need to be done and the longer it takes to generate the first token.
  • Decoding on the other hand is about writing the answer to our prompt, one word at a time. This happens repeatedly, token by token, until the response is finished. The model generates one new token at a time, and each new token depends on all previous tokens. Since only one token is processed per step, the GPU's compute cores sit mostly idle waiting for data to be fetched from memory that is constantly growing as the context grows, which is why this stage is memory-bound. This phase determines the throughput, measured as tokens per second (TPS), as well as the inter-token latency (ITL).
request arrivesPrefillall prompt tokens, in parallelcompute-boundfirst token12345…Decode — one token per step · memory-bandwidth-boundTTFT — time to first tokenITL — inter-token latencytime
Prefill processes the whole prompt in parallel, then decode emits one token per step.

Wrapping up

That wraps up the first recipe. To mirror the goals we set out with:

  1. You sized a practical VRAM budget and reviewed the actual memory profile from vLLM's startup logs.
  2. You validated the driver, PyTorch runtime, and pinned vLLM release.
  3. You served Qwen3.5-4B as a local endpoint on one NVIDIA A10 or equivalent.
  4. You sent a non-thinking request and inspected the response fields.

Next, we will benchmark our newly deployed model under a controlled workload to create a baseline.