Enable multi-image support benchmarking for serving #21145
Conversation
Allows the online benchmark script to correctly process and send requests containing multiple images per prompt, which is required for datasets like MUIRBENCH. Example on how to run this: ```sh python3 benchmarks/benchmark_serving.py \ --backend openai-chat \ --endpoint /chat/completions \ --base-url http://127.0.0.1:8688/v1 \ --model gemma/gemma-3-27b-it \ --request-rate "inf" \ --percentile-metrics ttft,tpot,itl,e2el \ --ignore-eos \ --num-prompt 128 \ --port 8688 \ --dataset-path MUIRBENCH/MUIRBENCH \ --dataset-name hf \ --max-concurrency 128 \ --save-result ``` And here is the results: ```sh ============ Serving Benchmark Result ============ Successful requests: 128 Benchmark duration (s): 116.25 Total input tokens: 2808 Total generated tokens: 16384 Request throughput (req/s): 1.10 Output token throughput (tok/s): 140.93 Total Token throughput (tok/s): 165.09 ---------------Time to First Token---------------- Mean TTFT (ms): 73745.15 Median TTFT (ms): 71047.68 P99 TTFT (ms): 111490.03 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 194.42 Median TPOT (ms): 162.71 P99 TPOT (ms): 425.83 ---------------Inter-token Latency---------------- Mean ITL (ms): 192.97 Median ITL (ms): 40.98 P99 ITL (ms): 1227.04 ----------------End-to-end Latency---------------- Mean E2EL (ms): 98436.22 Median E2EL (ms): 88044.50 P99 E2EL (ms): 116240.25 ================================================== ``` Signed-off-by: Stanley Phoong <[email protected]>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces multi-image support for serving benchmarks. It adds the MuirBenchDataset and updates the request handling logic for the OpenAI Chat backend. There are two potential issues that should be addressed to ensure the code functions correctly.
Signed-off-by: Stanley Phoong <[email protected]>
|
@DarkLight1337 , may you help to review this PR? |
Signed-off-by: Stanley Phoong <[email protected]>
Purpose
This PR updates the online benchmark script (
benchmarks/benchmark_serving.py) to support multi-image prompts. This capability is necessary for accurately benchmarking multimodal models using dataset that support more than one image e.g. MUIRBENCH.Test Plan
To validate the changes, run the online benchmark script pointing to a multi-image dataset. The following command was used for testing with the MUIRBENCH dataset:
python3 benchmarks/benchmark_serving.py \ --backend openai-chat \ --endpoint /chat/completions \ --base-url http://127.0.0.1:8688/v1 \ --model gemma/gemma-3-27b-it \ --request-rate "inf" \ --percentile-metrics ttft,tpot,itl,e2el \ --ignore-eos \ --num-prompt 128 \ --port 8688 \ --dataset-path MUIRBENCH/MUIRBENCH \ --dataset-name hf \ --max-concurrency 128 \ --save-resultTest Result
The results from the test command are provided below: