servingbenchmarkingEvalScopeQwen3.5-4BDraft

How to benchmark

Build a repeatable serving baseline by controlling prompt length, output length, caching, concurrency, and the tokenizer used to count tokens.

Daniel MolenaarsQwen

Beginner·35 minutes·Updated Oct 8, 2026

Recommended setup

Model

Qwen/Qwen3.5-4B

Hardware

1x NVIDIA GPU, 24 GB VRAM

Engine

vLLM 0.27.0

Benchmark

EvalScope 1.9.1

Python

Python 3.12

Tested environment

Hardware

1x NVIDIA A10, 24 GB VRAM

System

Ubuntu 24.04 x86_64

Driver

NVIDIA 580.126.09

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install 'vllm==0.27.0'

Quick start

In the previous recipe, we served Qwen3.5-4B through a local OpenAI-compatible endpoint. We will now use that endpoint to build our first benchmark.

Start the vLLM with prefix caching explicitly disabled:

source .venv/bin/activate

vllm serve Qwen/Qwen3.5-4B \
  --served-model-name qwen3.5-4b \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --dtype bfloat16 \
  --no-enable-prefix-caching

Leave the server running. In a second terminal, create a separate environment for the benchmark client and install the tested EvalScope release:

python -m venv .bench-venv
source .bench-venv/bin/activate
python -m pip install 'evalscope[perf]==1.9.1'

evalscope --version

Then, run the performance benchmark:

evalscope perf \
  --model qwen3.5-4b \
  --url http://127.0.0.1:8000/v1/chat/completions \
  --api openai \
  --dataset random \
  --tokenizer-path Qwen/Qwen3.5-4B \
  --min-prompt-length 512 \
  --max-prompt-length 512 \
  --prefix-length 0 \
  --min-tokens 128 \
  --max-tokens 128 \
  --temperature 0 \
  --seed 42 \
  --parallel 1 \
  --number 100 \
  --warmup-num 5 \
  --stream \
  --extra-args '{"ignore_eos": true, "chat_template_kwargs": {"enable_thinking": false}}' \
  --name qwen3.5-4b-a10-c1-512-128 \
  --outputs-dir ./benchmark-results

This sends five warmup requests, followed by 100 sequential request that will be benchmarked. Each request has a 512-token prompt and asks the server to generate exactly 128 tokens.

Why benchmarking is difficult

The hardest part about benchmarking is that results need to be comparable. This means controlling a lot of different variables that can easily change if you're not careful.

For an inference benchmark, we need to essentially control four things:

1. The server: the model itself, the precision of the model, serving engine version, launch flags, and the hardware.
2. The request: prompt content, prompt length, output length, chat template, thinking mode, and decoding parameters.
3. The load: concurrency, request count, and whether requests are sent in a closed or open loop.
4. The measurement: warmup behavior, streaming mode, tokenizer used, the reported percentiles, and the failed requests.

Defining the workload

For this post, our baseline will use a synthetic workload of 512 input tokens, 128 output tokens, and one concurrent request.

Synthetic means that we won't use an existing corpus of instructions or questions for our model. Rather, we will generate random data with a tool. EvalScope's random dataset can produce synthetic token sequences at the requested length. This gives us a stable instrument for comparing server configurations without allowing prompt content to change between experiments. However, if you have a custom dataset for your specific workload that you can use, that is even better!

For more detail on the benchmark client, see the official EvalScope performance-testing guide and parameter reference.

Matching tokenizers

The following option tells EvalScope which tokenizer should construct the requests:

--tokenizer-path Qwen/Qwen3.5-4B

The tokenizer ideally must point to the same model family as the one running on our server, as a tokenizer from another model or family can split identical text differently. The benchmark will still run, but if we generate 512 tokens client side with our tokenizer, these may not translate to 512 tokens on the server side.

As its name indicates, a tokenizer breaks raw text into tokens, often subwords, and converts these into numerical IDs that an AI model can process. The tokenizer file that we refer to in tokenizer-path stores the vocabulary and splitting rules needed to perform this encoding consistently, acting as a shared dictionary between human-readable text and the numbers the model understands.

To repeat, the two model-related flags have different jobs:

• --model qwen3.5-4b is the API name sent to vLLM. It matches the alias configured with --served-model-name.
• --tokenizer-path Qwen/Qwen3.5-4B identifies the tokenizer files used by the benchmark client. These are always located in a model repository.

Fix the output length

By default, a model can emit an end-of-sequence (EOS) token before reaching max_tokens. That is desirable in a real application: the model should stop when its answer is complete, and you don't want it to be wasteful with tokens. It becomes a problem in a synthetic performance test because different requests then perform different amounts of decode work, as typically the synthetic data just consists of random words.

We can control the output like follows:

--min-tokens 128 \
--max-tokens 128 \
--extra-args '{"ignore_eos": true, "chat_template_kwargs": {"enable_thinking": false}}'

• min_tokens forces the model to keep generating until it reaches that count, suppressing the EOS token even if the model naturally wants to stop earlier.
• max_tokens caps generation at a fixed length regardless of whether the model has finished its thought.
• ignore_eos disables the EOS token entirely, so the model never stops on its own and always runs until max_tokens is hit.

You might be wondering why there is a need for ignore_eos if we already have the min-tokens flag enabled. In short, min_tokens only suppresses the EOS token until the minimum token count is hit. After that point, the model regains its normal ability to stop whenever it predicts an EOS token. ignore_eos, by contrast, disables EOS detection entirely for the whole generation run, meaning the model can never stop on its own and will always generate until it hits max_tokens.

You can remove the ignore_eos flag only if min_tokens are set it equal to max-tokens, which forces the model to run to its max token limit just like ignore_eos does. Otherwise, min-tokens guarantees a minimum length while still allowing natural stopping afterwards.

Keep decoding deterministic

We also set temperature to zero and use a fixed seed:

--temperature 0 \
--seed 42

Temperature and seed control decoding in complementary ways. One shapes what the model is likely to pick, and the other ensures which pick is reproducible.

Temperature scales the raw probability distribution over the vocabulary before a token is sampled. A low temperature (e.g., 0.1) sharpens the distribution, heavily favoring the most likely tokens for deterministic, focused output. A high temperature (e.g., 1.5) flattens it, boosting the odds of less likely tokens and producing more diverse or creative text. At exactly 0, most frameworks skip sampling entirely and just pick the top token (greedy decoding).

Seed initializes the random number generator used during that sampling step. Because temperature still leaves multiple tokens with non-zero probabilities, the actual choice among them is random. Fixing the seed makes that randomness deterministic: running the same prompt with the same temperature and seed will yield the exact same sequence of tokens every time.

In short: temperature controls the shape of the probability curve, while the seed controls the repeatability of the roll on that curve.

Remove hidden shortcuts

Warm the server before measuring

The first request after startup is rarely representative. The runtime may still be initializing kernels, allocating working memory, or populating internal caches. If that first request is included, a short benchmark might just mostly measure startup effects.

EvalScope handles our warmup with:

--warmup-num 5

These requests use the same workload as the measured run but are excluded from the reported metrics. For now, we'll set this to 5 as a practical starting point for our initial baseline, but you can increase this if you feel that latency has not yet settled.

Disable prefix caching

Automatic prefix caching allows vLLM to reuse work when requests begin with token sequences it has already processed. This is useful in production, especially for long shared system prompts, and coding scenarios where we might store the conversational history with repository files. However, once you start comparing models where in one set-up the caching has been enabled and in another it has been disabled, you will get a skewed view of the performance baseline, making the caching-enabled server serve faster when repeated prefixes receive cache hits.

For that reason, our vLLM startup includes:

--no-enable-prefix-caching

We also set --prefix-length 0, which tells EvalScope not to deliberately create a shared random prefix.

Stream the response

Streaming changes what the client can observe. With a streaming response, EvalScope receives the first generated token before the request is complete and can measure time to first token (TTFT). With a non-streaming response, the entire answer arrives at once, so the client cannot separate the wait for the first token from the time spent generating the rest.

We therefore pass --stream explicitly. If a report contains no TTFT values, streaming is typically the first setting to check (if your request arrived at all!).

Closed-loop and open loop testing

--parallel 1 means EvalScope keeps at most one request in flight. When it receives the complete response, it sends the next request. This is called a closed-loop test: the client advances at the speed of the server.

At concurrency one, the GPU will have little opportunity to batch independent requests together. The result is useful as a latency baseline, but it does not tell us the maximum throughput of the server.

Throughput and latency typically move in opposite directions. Increasing concurrency generally gives vLLM more work to batch, which can raise the total number of tokens produced per second. Simultaneously, the total time it takes to complete a single response will increase as the server is getting more busy processing the increased requests.

This is called an 'open loop' test, on which we will focus in the next recipe. For now, we will keep the load fixed at one active request.

Send enough requests

Even on an idle machine, latency can vary due to small differences in request handling. Secondly, we're interested in the statistic distribution of the requests: out of our requests, were there any outliers? What is the median response time?

Our measured run therefore uses 100 requests:

--number 100

Reading the results

EvalScope prints a benchmark summary and stores the run artifacts under ./benchmark-results. Before looking for the fastest number, validate the experiment in this order:

1. Successful and failed requests. Our benchmark results become untrustworthy if requests failed or timed out, as these will skew both the response time (which will become longer due to timeouts) and throughput (which will become lower due to failed requests).
2. Average input and output tokens. Confirm that the workload was actually close to 512 input tokens and exactly 128 output tokens per successful request.
3. TTFT and ITL. These describe different parts of the user experience.
4. Output throughput. At concurrency one, this is the output rate of one sequential client.
5. Percentiles. Compare at least p50 and p99, as an average can hide a small number of very slow requests!

What this baseline can tell you

This benchmark can answer a narrow but useful question:

How does Qwen3.5-4B on this server handle a fixed 512-token prompt followed by 128 decode steps, with one request active and no prefix-cache reuse?

It cannot yet tell you:

• How much traffic the server can sustain.
• How latency changes as concurrency increases.
• How long prompts affect time to first token.
• Whether the model's answers are useful or correct.

Wrapping up

You now have the first version of a trustworthy measuring instrument!

1. You fixed the workload at 512 input tokens and 128 output tokens.
2. You removed hidden optimization shortcuts by warming the server and disabling prefix caching.
3. You ensured token measurement is precise with the matching Qwen tokenizer.
4. You measured through a streaming endpoint so TTFT remains observable.

Next, we will increase concurrency and plot TTFT, ITL, and throughput, and see the tradeoffs when optimizing for each of these metrics.