servingperformanceEvalScopeQwen3.5-4BDraft

Reading the curves: TTFT, ITL, and throughput

Understand what happens during prefill and decode, then connect those phases to the latency and throughput curves you measure.

Daniel MolenaarsQwen

Beginner·45–90 minutes·Updated Oct 8, 2026

Recommended setup

Model

Qwen/Qwen3.5-4B

Hardware

1x NVIDIA GPU, 24 GB VRAM

Engine

vLLM 0.27.0

Benchmark

EvalScope 1.9.1

Python

Python 3.12

Tested environment

Hardware

1x NVIDIA A10, 24 GB VRAM

System

Ubuntu 22.04 container

Driver

NVIDIA 580.105.08

Quick start

In the previous recipe, we built a benchmark with one active request. We will now send several requests at once. This gives vLLM an opportunity to batch work together, and lets us see what happens to each user's response as the server gets busier.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install 'vllm==0.27.0'

Run the following in your first terminal:

source .venv/bin/activate

# Download the same checkpoint and tokenizer used for this experiment
hf download Qwen/Qwen3.5-4B \
  --local-dir ./models/qwen3.5-4b

vllm serve ./models/qwen3.5-4b \
  --served-model-name qwen3.5-4b \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --dtype bfloat16 \
  --no-enable-prefix-caching

Leave the server running. In a second terminal, create or activate the separate benchmark environment and verify the endpoint:

python -m venv .bench-venv
source .bench-venv/bin/activate
python -m pip install 'evalscope[perf]==1.9.1' \
  'transformers==5.15.0' 'tokenizers==0.22.2'

curl http://127.0.0.1:8000/v1/models

We will keep the same 512-token prompt target and 128-token output like before, and increase the number of active requests:

evalscope perf \
  --model qwen3.5-4b \
  --url http://127.0.0.1:8000/v1/chat/completions \
  --api openai \
  --dataset random \
  --tokenizer-path ./models/qwen3.5-4b \
  --min-prompt-length 512 \
  --max-prompt-length 512 \
  --prefix-length 0 \
  --min-tokens 128 \
  --max-tokens 128 \
  --temperature 0 \
  --seed 42 \
  --parallel 1 2 4 8 16 32 64 128 256 \
  --number 2048 2048 2048 2048 2048 2048 2048 2048 2048 \
  --duration 120 \
  --in-flight-task-multiplier 1 \
  --warmup-num 5 \
  --total-timeout 120 \
  --stream \
  --extra-args '{"ignore_eos": true, "chat_template_kwargs": {"enable_thinking": false}}' \
  --name qwen3.5-4b-a10-concurrency \
  --outputs-dir ./curve-results

This runs nine separate benchmarks at different concurrency levels. Once finished, Evalscope will produce a report for you.

What are we measuring?

Imagine that you ask a question in a chat application. There are three different things you might notice:

1. Time to first token (TTFT). How long do you wait before the answer starts appearing? This includes request handling, any wait in the server's queue, and the work needed to process the prompt and produce the first token.
2. Inter-token latency (ITL). Once the answer starts, how quickly does the rest appear? A lower ITL generally means a smoother, faster stream.
3. Output throughput. How many output tokens does the server produce per second across all requests? This tells us about the server's total work serving multiple clients, rather than the speed of one person's answer. Imagine your model could only serve one user at a time: that would not be very useful at all!

Each of these metrics describes a different part of the total end user experience. A server can produce more tokens per second overall, while making each person wait longer for their own response. Similarly, a server can produce not that many tokens per second in order to prioritize individual user request speed.

Look at the median and the slower requests

Let's talk a bit about statistics. The p50, or median, is the middle of the sorted measurements: roughly half were faster and half were slower. The p95 (or p99, which is commonly used) is near the slow end: roughly 95% were at or below that value. Comparing both helps us see whether a change affects typical requests or mainly a smaller group of slow ones.

For TTFT, this means that we are looking at how fast the typical initial response is each time. If we see that our p99 is high, that means that at least 1% of our requests are hitting this number. This can be undesirable in scenarios where we want the model to act fast.

For ITL, each gap in the stream contributes a measurement. The ITL p95 therefore describes slow gaps overall, not the slowest 5% of whole requests like we see with TTFT.

Read the concurrency curve

We will first change only concurrency: the number of requests that the client keeps in flight. Every request still has a 512-token input target and produces 128 output tokens. Execute the following command with Evalscope:

evalscope perf \
  --model qwen3.5-4b \
  --url http://127.0.0.1:8000/v1/chat/completions \
  --api openai \
  --dataset random \
  --tokenizer-path ./models/qwen3.5-4b \
  --min-prompt-length 512 \
  --max-prompt-length 512 \
  --prefix-length 0 \
  --min-tokens 128 \
  --max-tokens 128 \
  --temperature 0 \
  --seed 42 \
  --parallel 1 2 4 8 16 32 64 128 256 \
  --number 2048 2048 2048 2048 2048 2048 2048 2048 2048 \
  --duration 120 \
  --in-flight-task-multiplier 1 \
  --warmup-num 5 \
  --total-timeout 120 \
  --stream \
  --extra-args '{"ignore_eos": true, "chat_template_kwargs": {"enable_thinking": false}}' \
  --name qwen3.5-4b-a10-concurrency \
  --outputs-dir ./curve-results

Here are the settings that changed from our first benchmark:

• --parallel lists the concurrency levels to test, separated by spaces.
• --duration 120 gives each level a two-minute window for scheduling measured work, after warmup. Already scheduled requests finish afterward.
• --number the request count for each concurrency level, separated by spaces.
• --in-flight-task-multiplier 1 limits the client's scheduled tasks to the chosen concurrency, avoiding an extra group waiting inside the client.
• --warmup-num 5 sends five warmup requests before each point. Warmup is excluded from the timed window.
• --name names the overall benchmark run. EvalScope will automatically create a separate result directory for each concurrency/request-count pair.

Once your run has finished, EvalScope will produce a table with the benchmark data. The data will contain something like the chart below:

Let's start with Throughput. As you can see, the server throughput shifts from about 49 output tokens per second at concurrency one to 855 at 64. We then see that it hits a performance ceiling: about 890 at 128 and 877 at 256, already indicating that the maximum throughput has been achieved.

Now switch to TTFT. At concurrency one, the median wait is about 91 ms. At 64, it is 872 ms. At 128, the median reaches 2.18 seconds, while p95 reaches 8.29 seconds. At 256, the median is 19.0 seconds and p95 is 25.4 seconds, a 250x increase! In other words, a single GPU can only do so much, and by increasing the concurrency we are asking users to wait much longer for almost the same total output generated by the card.

Finally, look at ITL. The median gap grows from 19.8 ms at concurrency one to about 55 ms at 128–256. The p95 gap reaches roughly 317 ms at 256, so some pauses are much longer than the typical gap.

This is the tradeoff we are trying to see: batching more requests can increase the server's total output, while making each individual answer take longer. Once throughput levels off, any additional concurrency mostly adds waiting.

Changing the prompt length

Next, we will look at how the size of our prompts can impact generation speed. For this test, we will keep the concurrency fixed at four and increase the input from 128 to 8,192 tokens. Every response still contains 128 output tokens.

Start from the Evalscope command in the previous section. Remove --duration and --in-flight-task-multiplier, then replace both sweep lists and the corresponding settings with:

--parallel 4 \
--number 128 \
--warmup-num 5 \
--min-prompt-length 2048 \
--max-prompt-length 2048 \
--min-tokens 128 \
--max-tokens 128

Repeat this with input targets 128, 512, 1,024, 2,048, and 8,192, giving each run a different name.

Once the results are in, they should look something like this:

Starting, with TTFT, we can already see that moving from a 128-token prompt target to 8,192 raised the median wait from 118 ms to 2,338 ms, a 20x increase! The reason for this is that the server has much more input to process before it can start answering. It has to be added, though, that in practice, TTFT will also includes queueing and request handling, so keep that in mind!

Switch to ITL to see what happened once the answer started. The typical streaming gap changed much less, from 21.0 ms to 23.1 ms. However, the p95 gap at the 8K target reached 286.9 ms! Looking only at the median would miss those longer pauses.

The Throughput view shows essentially the inverse of our previous chart: the higher the input, the lower the throughput, with a sharp decline at the higher end of our inputs. The reason is quite simple: each request still generates the same answer length, but the server must spend more work on input before it can produce that output, making prompt length part of any meaningful tokens-per-second comparison!

We can now see that for long inputs and short outputs, results are prefill-bound: our server spends most of the time processing the inputs, rather than generating the output. More on that below.

Compare a longer answer

In contrast, a longer answer mean that our server will need more time for decoding. Return the prompt target to 512 and compare output lengths 32, 128, and 512, keeping concurrency four:

--min-prompt-length 512 \
--max-prompt-length 512 \
--min-tokens 512 \
--max-tokens 512

Increasing the output from 32 to 512 tokens barely changed median TTFT: 308 ms versus 306 ms. The prompt-processing work was similar. But median total response time grew from 0.96 seconds to 11.12 seconds, a 10x increase, because the model had many more decode steps to perform!

Aggregate throughput, however, increased from 133.3 to 184.2 tokens per second. The server now spends a larger share of each request generating output, spreading the prompt-processing cost over more tokens. A higher throughput number can therefore come from changing the workload, without actually improving the server itself.

What happens when requests keep arriving?

Our concurrency sweep above was a closed-loop test: the client waited for a response to finish before replacing it. If the server slowed down, replacements arrived more slowly too.

Real users will probably not want to wait for somebody else's answer before sending their own question. An open-loop test models this by scheduling arrivals independently of completions. We will use 2, 4, 8, and 12 requests per second, with the same 512-token input target and 128-token output.

Return to the quick-start command and replace its concurrency, duration, request-count, warmup, and name settings with:

--open-loop \
--rate 2 4 8 12 \
--number 240 480 960 1440 \
--warmup-num 16 \
--name qwen3.5-4b-a10-arrivals

Remove --parallel, --duration, and --in-flight-task-multiplier.

Before starting the test, run this script in another terminal, which records the running and waiting request counts once per second. Leave it running until the benchmark finishes and the queue drains, then press Ctrl+C:

python - <<'PYTHON'
import csv
import re
import time
import urllib.request

names = ["num_requests_running", "num_requests_waiting"]
with open("queue-samples.csv", "w", newline="") as file:
    writer = csv.writer(file)
    writer.writerow(["elapsed_seconds", *names])
    started = time.monotonic()
    try:
        while True:
            with urllib.request.urlopen("http://127.0.0.1:8000/metrics", timeout=10) as response:
                metrics = response.read().decode()
            counts = []
            for name in names:
                values = re.findall(r"^vllm:" + name + r"(?:\{[^}]*\})? ([^\s]+)",
                                    metrics, re.MULTILINE)
                if not values:
                    raise ValueError(f"Missing server metric: {name}")
                counts.append(sum(float(value) for value in values))
            writer.writerow([round(time.monotonic() - started, 2), *counts])
            file.flush()
            print(dict(zip(names, counts)), flush=True)
            time.sleep(1)
    except KeyboardInterrupt:
        pass
PYTHON

We will store all samples within the queue-samples.csv file, which should look roughly like our chart below:

Follow the queue over time

This chart displays the running and waiting requests as time goes on. At 2 and 4 requests/s we can clearly see that our server can keep up: there is no queue, and every request sent to our server is picked up right away.

It becomes more interesting at 8 requests/s, where 45 seconds into our test where we are able to visibly sample a growing amount of queued requests by the server, reaching more than 110 at its peak.

At twelve requests per second, we can now also see the maximum requests that vLLM can serve given our current hardware: give or take 110. More incoming work keeps accumulating until we finally stop sending out requests.

Note also the time vLLM takes to drain all outstanding requests. At two requests/s, the last response arrived 2.8 seconds after the last request started. At eight, draining took 20.8 seconds; at twelve, it took 79.5 seconds!

What vLLM does with those requests

vLLM does not wait for a whole batch of answers to finish before accepting another batch. It uses continuous batching: the scheduler decides which requests can make progress at each step.

For this configuration, the process is roughly:

1. A new request enters the waiting queue. It needs a place in the running set and enough cache space for its state.
2. The scheduler gives running requests work. This can include ongoing generation and unfinished prompt processing. Each step has a token budget.
3. New requests join when resources allow. With chunked prefill enabled, a long prompt can be processed across several steps instead of requiring all of its prompt tokens to fit in one step.
4. Finished requests release resources. Other requests can then use the freed capacity. Under cache pressure, vLLM can also interrupt a running request and return it to the waiting queue; this is called preemption and can add work and delay.

The default scheduling policy is first come, first served, but that does not mean requests finish in arrival order. Prompt length, output length, the work already running, and resource limits all matter. If you're interested, you can take a look at the vLLM scheduler source code to dive into the details.

This also explains why a growing queue will affect TTFT: more requests spend time waiting before their first text can appear. It also explains why a server can keep producing plenty of tokens while becoming unpleasant to use: good aggregate throughput does not remove the wait experienced by an individual user!

What happened under the hood

The first recipe introduced prefill and decode. Prefill processes the prompt, while decode generates the continuation one token at a time. Both affect our curves, but in different ways.

Ingesting the prompt

During prefill, the model processes many prompt positions together. This creates enough parallel work to make good use of the GPU's compute units. As explained before, a longer prompt means more work before the answer can start, which is why prompt length often has a strong effect on TTFT.

However, TTFT is not just prefill time. As we have seen just now, if too many requests arrive together, some may have to be queued up before the server can process them!

Generating the answer

During decode, each request needs its previous token before it can produce the next one. At low concurrency, reading the model's weights from GPU memory can take more time than calculating the next token. We call this being memory-bound: memory movement limits the speed.

Batching helps because the model can use the same weights to process a step for several requests together. This can raise total throughput substantially. However, it also means a larger step, more active state, and more work for the GPU, so each individual stream can become slower.

One decode step

After prefill, vLLM repeats the same loop for every generated token:

1. Embed the current token: its token ID becomes a hidden vector.
2. Run all 32 layers: the vector passes through eight groups of 4 layers for Qwen3.5-4B. Qwen3.5 introduced Gated DeltaNet layers, a special layer that does not require us to store full attention. We'll get into that in another recipe.
3. Score and select the next token: the output head produces one score for every token in the vocabulary. vLLM applies the request's decoding settings, selects one token ID, and repeats the loop.

What the attention layer does

So what is an attention layer, and what does hidden vector mean? Hidden simply means internal to the model, nothing else! Each token is represented by a list of numbers called a hidden vector. The embedding creates the first version, and every layer updates it to encode more of what that token means in its current context. In Qwen3.5-4B, each hidden vector contains 2,560 numbers and is not directly readable as text.

When it comes to attention, think of a full-attention layer as a small search engine. It creates three different views of each hidden vector:

1. Query: “What am I looking for?” The query is like a search request made by the current position. It represents the features that would be useful to retrieve right now.
2. Key: “What can I be matched on?” A key is like a catalog entry for a visible token position. The query is compared with these keys to decide which positions are relevant.
3. Value: “What information can I contribute?” A value is like the content associated with a catalog entry. If its key is a strong match, more of this value is retrieved.

Don't worry if this seems confusing at first: these three items are just numbers in the end, which can make it difficult to understand their contribution to the model!

The attention calculation then follows four steps:

1. Match: compare the current query with every visible key.
2. Weight: turn the match scores into attention weights; stronger matches get more weight.
3. Retrieve: multiply each value by its weight and combine the results.
4. Update: use that mixture to enrich the current hidden vector with relevant context.

Consider the sentence “The dog chased the ball because it was red.” The following diagram slows down one illustrative attention head as it processes the final token, “red”:

During decoding, the query is used for the current lookup, while the newest key and value are appended to the K/V cache. The next decode step can reuse those keys and values instead of recomputing them for the whole sequence. Qwen3.5-4B has eight full-attention layers that use this cache; its other 24 layers update a compact DeltaNet recurrent state. The model weights themselves stay unchanged.

How the next token is chosen

We've also used the word vocabulary in the context of token generation. A model's vocabulary refers to the tokenizer's inventory of text pieces, rather than a full-blown dictionary. This is because a token can represent not just a whole word, but also a part of a word, punctuation, whitespace, or even a special control marker. In fact, Qwen3.5-4B has 248,320 possible token IDs in its vocabulary!

If you're curious, you can view Qwen3.5-4B vocabulary in its vocab.json file on Huggingface. Below are the first 10 entries, covering symbols and punctuation:

{
    "!": 0,
    "\"": 1,
    "#": 2,
    "$": 3,
    "%": 4,
    "&": 5,
    "'": 6,
    "(": 7,
    ")": 8,
    "*": 9,
    "+": 10,
    ...
}

After the final decoder layer, the output head compares the hidden vector with a pre-learned vector for every token in that vocabulary. The result is one raw score, called a logit, per token. A higher logit means the model considers that token a better continuation in the current context; logits are relative scores, not confidence percentages.

vLLM's sampler turns those logits into a decision. With temperature: 0.0, it performs greedy decoding and chooses the token with the highest final logit. With a nonzero temperature, vLLM can reshape the scores, restrict the candidates with settings such as top_p or top_k, and sample from the resulting probability distribution. The selected token is appended to the sequence, decoded into text, and fed into the next iteration.

Choose a useful operating point

The highest throughput number is not automatically the configuration you want. Start with what your application needs:

1. For an interactive application, decide how long users can wait for the first token and how slowly the answer can stream. Find a region of the curve that meets both needs.
2. For background work, higher throughput may matter more than the latency of any individual request. Batching more requests can be useful even when each one takes longer.
3. For traffic from real users, follow the concurrency sweep with an arrival-rate test and representative prompt/output lengths. Leave room for bursts rather than treating a short successful run as a capacity guarantee.

If throughput is still rising at the final point, we have not found its maximum. If it levels off while latency rises, additional concurrency is buying little extra completed work. In either case, check failures and actual token counts before drawing a conclusion.

Wrapping up

You now have more than a single tokens-per-second number:

1. You varied concurrency while keeping the workload fixed.
2. You compared TTFT, ITL, and total throughput, and saw why they answer different questions.
3. You changed prompt and output length separately to connect the results to prefill and decode.
4. You distinguished closed-loop and open-loop testing, so you can choose a test that matches how requests reach your application.

Next, we will look at memory: what the active requests store, how the cache grows, and how that limits the workloads we can serve on one GPU.