It accurately replays real-world request traces with original timing or dynamically generates requests using popular datasets. The tool delivers comprehensive performance metrics—including Time to First Token (TTFT), Time Per Output Token (TPOT), Inter-Token Latency (ITL), End-to-End Latency, Goodput, etc.
The Trace Replay feature mainly includes request generation, request sending and response receiving, as well as result calculation and saving. It can reproduce historical requests based on MoonCake's trace file and strictly send the requests according to the timestamps recorded in the trace. After execution, Trace Replay calculates key performance metrics such as Time to First Token (TTFT) and Time Per Output Token (TPOT), then outputs the results to the terminal and saves them to an Excel file.
Mooncake traces consist of two types of trace data：
- Conversation and Tool&Agent trace: Sampled from one hour of online request data from different clusters.
- Synthetic trace: Generated synthetically from other publicly available datasets.
For more information, please refer to the Mooncake paper: Mooncake-FAST25.pdf .
Trace Replay supports two request-generation methods:
- Hash ID-based: Input tokens are generated based on the input_length and hash_ids recorded in the trace file. Each hash_id corresponds to a block, with each block containing 512 tokens. The same hash_id always maps to the identical token sequence.
- Dataset-based: Prompts are generated by invoking vLLM's benchmark module using the input_length from the trace file and the user-specified dataset name. This approach does not rely on the hash_ids present in the trace file.
Depending on the request generation method, Trace Replay offers two modes: Trace Mode and Benchmark Mode, which can be configured by the user via the --trace-mode parameter.
| Argument | Default | Help |
|---|---|---|
| --backend | None | backend framework type |
| --model | None | model path |
| --host | localhost | IP address of the inference server |
| --port | None | Port number of the inference server |
| --trace-path | None | trace jsonl file path |
| --trace-mode | trace | 'trace' to replay requests from cached trace files, 'benchmark' to generate requests dynamically using the benchmark module |
| --dataset-name | sharegpt | if enable benchmark mode, you must specify a dataset， refer to the vLLM benchmark documentation |
| --save-prompts | False | save generated prompts with timestamp for reuse |
| --save-result | False | save the benchmark metrics to excel file |
| --result-dir | None | path to save results |
You need to download the trace jsonl file from Mooncake traces. In the trace, each line is a JSON object representing a single request:
{
"timestamp": 1696000000123, // ms since epoch
"input_length": 512, // number of input tokens
"output_length": 128, // expected output tokens
"hash_ids": [123, 456, 789] // seed list for deterministic prompt generation
}
Trace Replay depends on vLLM's benchmark module, which you need to download separately. Before running Trace Replay, you must set the path to the benchmark module via an environment variable.:
export BENCHMARK_PATH="/vllm-workspace/benchmarks"Execute the Python script to replay a trace against a local vLLM instance:
python3 /trace_replay.py \
--model /home/models/dsv2-lite \
--backend vllm \
--trace-path /conversation_trace.jsonl \
--trace-mode trace \
--host 127.0.0.1 \
--port 8000 \
--save-result \
--save-promptsSuccessful execution results in output similar to the following:
============ Serving Benchmark Result ============
Successful requests: 510
Benchmark duration (s): 301.46
Total input tokens: 7201515
Total generated tokens: 185502
Request throughput (req/s): 1.69
Output token throughput (tok/s): 615.34
Total Token throughput (tok/s): 24504.02
---------------Time to First Token----------------
Mean TTFT (ms): 20931.33
Median TTFT (ms): 19119.63
Std TTFT (ms): 17324.45
P25 TTFT (ms): 4057.98
P50 TTFT (ms): 19119.63
P75 TTFT (ms): 33284.55
P99 TTFT (ms): 64592.68
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 187.71
Median TPOT (ms): 200.69
Std TPOT (ms): 63.08
P25 TPOT (ms): 144.17
P50 TPOT (ms): 200.69
P75 TPOT (ms): 234.55
P99 TPOT (ms): 312.87
---------------Inter-token Latency----------------
Mean ITL (ms): 181.20
Median ITL (ms): 169.18
Std ITL (ms): 133.70
P25 ITL (ms): 86.63
P50 ITL (ms): 169.18
P75 ITL (ms): 230.91
P99 ITL (ms): 647.04
----------------End-to-end Latency----------------
Mean E2EL (ms): 86656.79
Median E2EL (ms): 89218.82
Std E2EL (ms): 43454.94
P25 E2EL (ms): 53935.13
P50 E2EL (ms): 89218.82
P75 E2EL (ms): 120761.34
P99 E2EL (ms): 171262.27
==================================================