Add an optimization doc on TPU #21155
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
mergify
bot
added
documentation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds a new documentation page with optimization tips for using vLLM on TPUs. The content is valuable and well-structured. I've identified a few issues, mainly related to broken or fragile links and a missing image, which could impact the user experience. I've also found a typo in a command-line argument that could cause issues for users who copy-paste it. Please see my detailed comments.
|
|
||
| ### **Get started** | ||
|
|
||
| Looking for setup and installation instructions? Find them [here](https://vllm--19708.org.readthedocs.build/en/19708/getting_started/installation/google_tpu.html). |
There was a problem hiding this comment.
The link to the setup and installation instructions points to a temporary ReadTheDocs build for this PR. This link will become invalid after the PR is merged. Please update it to a permanent link, likely pointing to the official vLLM documentation.
| Looking for setup and installation instructions? Find them [here](https://vllm--19708.org.readthedocs.build/en/19708/getting_started/installation/google_tpu.html). | |
| Looking for setup and installation instructions? Find them [here](https://docs.vllm.ai/en/latest/getting_started/installation/google_tpu.html). |
|
|
||
| When selecting the ideal number of chips for a single serving instance, it's important to account for both the model size and the average request context length. Adequate HBM for the KV cache is essential to ensure a sufficient number of concurrent requests can be processed. | ||
|
|
||
| The following colab [calculator](https://colab.sandbox.google.com/drive/1M_f3xZm-_Ce2D-UMAyGNyacEIN-6rUbf) will tell you: |
There was a problem hiding this comment.
The link to the Colab calculator uses a sandbox.google.com domain, which may not be accessible to all users. Please use a public, shareable link (e.g., from colab.research.google.com) to ensure everyone can access the calculator.
|
|
||
| #### *max model len vs. most model len* | ||
|
|
||
|  |
There was a problem hiding this comment.
The image most_model_len.png is referenced here, but it doesn't seem to be included in this pull request. This will result in a broken image in the documentation. Please add the image file to the repository, for example in the same directory as this README.
|
|
||
| ### **Tune your workloads!** | ||
|
|
||
| Although we try to have great default configs, we strongly recommend you check out the [vLLM auto-tuner](https://github.com/vllm-project/vllm/pull/20779/files?short_path=f9b273a#diff-f9b273a10e0688ba63c38bd93a2e64ceb54d4fdd7ff7b82d347df06d0d34e39c) to optimize your workloads for your use case. |
There was a problem hiding this comment.
Linking to a pull request is not ideal for documentation as PRs are not permanent resources in the same way that documentation pages are. The link is also very specific and might break easily. It would be better to link to the official documentation for the auto-tuner feature once it's available, or to a more stable resource.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.Purpose
Test Plan
Test Result
(Optional) Documentation Update