[WIP] backed_Size_oblivious + pytorch 2.8 #20719

laithsakka · 2025-07-09T23:54:31Z

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Signed-off-by: Huy Do <[email protected]>

github-actions · 2025-07-09T23:54:40Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Summary of Changes

Hello @laithsakka, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request primarily focuses on upgrading the project's core PyTorch dependency to version 2.8.0, along with its associated libraries like torchvision and torchaudio. It also introduces a specific optimization for torch.compile using backed_size_oblivious mode, and includes necessary adjustments to build processes and dependencies to ensure compatibility with the new PyTorch version, including temporary workarounds for unreleased components.

Highlights

PyTorch Version Update: I've updated the project's PyTorch dependency from version 2.7.0 to 2.8.0 across all relevant configuration files, including CMakeLists.txt, pyproject.toml, and various requirements/*.txt files (build, CPU, CUDA, ROCm, test).
PyTorch Test/RC Index Integration: To support the PyTorch 2.8.0 update, I've configured pip-compile and Docker builds to use the PyTorch test/release candidate (RC) index (https://download.pytorch.org/whl/test/) for dependencies like torch, torchaudio, torchvision, and flashinfer. This is reflected in .pre-commit-config.yaml, docker/Dockerfile, and requirements/test.in.
XFormers Compatibility Workaround: Since xformers 0.0.30 is not yet officially released for PyTorch 2.8.0, I've added a temporary workaround in docker/Dockerfile to install xformers directly from its GitHub repository. The requirements/cuda.txt file also reflects this temporary change by commenting out the direct version dependency.
Torch.compile Enhancement: I've modified vllm/worker/model_runner.py to wrap the torch.compile call with torch.fx.experimental._config.patch(backed_size_oblivious=True). This change likely aims to leverage a new feature in PyTorch 2.8 for improved memory management or performance during model compilation.
Dependency Updates: Beyond PyTorch, I've updated torchvision to 0.23.0, torchaudio to 2.8.0, and triton to 3.4.0. Additionally, various nvidia-cuda related packages in requirements/test.txt have been updated to newer versions compatible with PyTorch 2.8.
Documentation and Test Script Updates: I've updated the docs/contributing/ci/update_pytorch_version.md to include instructions for using PyTorch RC in tests, and applied this change directly to tests/standalone_tests/python_only_compile.sh for immediate testing.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request updates the project to use PyTorch 2.8 and introduces the backed_size_oblivious feature. The changes across various configuration files and dependency lists are largely consistent with this goal. My review focuses on ensuring dependency management is consistent across all installation methods and removing temporary debugging code before merging. I've identified a critical issue in requirements/cuda.txt where a necessary dependency is commented out, and a few minor cleanup items in the Dockerfile and a test script.

gemini-code-assist · 2025-07-09T23:56:08Z

requirements/cuda.txt

+# TODO (huydhn): Re-enable this once xformers is released for 2.8.0
 # https://github.com/facebookresearch/xformers/releases/tag/v0.0.30
-xformers==0.0.30; platform_system == 'Linux' and platform_machine == 'x86_64'  # Requires PyTorch >= 2.7
+# git+https://github.com/facebookresearch/xformers@v0.0.30; platform_system == 'Linux' and platform_machine == 'x86_64'  # Requires PyTorch >= 2.7


The xformers dependency is commented out, which will likely cause installation issues for users who rely on this requirements file. To ensure consistency with the Dockerfile, which installs xformers from git, this line should be uncommented. I've also updated the comment to reflect the new PyTorch version.

# TODO (huydhn): Re-enable this once xformers is released for 2.8.0 # https://github.com/facebookresearch/xformers/releases/tag/v0.0.30 git+https://github.com/facebookresearch/[email protected]; platform_system == 'Linux' and platform_machine == 'x86_64' # Requires PyTorch >= 2.8

gemini-code-assist · 2025-07-09T23:56:08Z

docker/Dockerfile

+    # DEBUG
+    python3 -m xformers.info


This debugging line should be removed before the pull request is merged. It's good for development, but shouldn't be in the final Docker image.

gemini-code-assist · 2025-07-09T23:56:09Z

tests/standalone_tests/python_only_compile.sh

@@ -18,7 +18,9 @@ apt autoremove -y

 echo 'import os; os.system("touch /tmp/changed.file")' >> vllm/__init__.py

-VLLM_TEST_USE_PRECOMPILED_NIGHTLY_WHEEL=1 VLLM_USE_PRECOMPILED=1 pip3 install -vvv -e .
+# TESTING, TO BE REMOVED


The comment # TESTING, TO BE REMOVED indicates this is a temporary change. Please remove the comment before merging this pull request.

laithsakka · 2025-07-09T23:57:30Z

vllm/worker/model_runner.py

-                self.model,
-                fullgraph=envs.VLLM_TEST_DYNAMO_FULLGRAPH_CAPTURE,
-                backend=backend)
+            with torch.fx.experimental._config.patch(backed_size_oblivious=True):


how can we localize this to lora?

mergify · 2025-07-17T02:43:58Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @laithsakka.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

huydhn and others added 6 commits July 8, 2025 16:40

[WIP][RC] Update PyTorch to 2.8.0

d6bf991

Signed-off-by: Huy Do <[email protected]>

Handle xformers

456985c

Signed-off-by: Huy Do <[email protected]>

Some more tweaks

4838d53

Signed-off-by: Huy Do <[email protected]>

Attempt to fix xformers build

ca21216

Signed-off-by: Huy Do <[email protected]>

Silly typo

0c43174

Signed-off-by: Huy Do <[email protected]>

use backed_size_oblivious

1f269f9

laithsakka requested review from hmellor, tlrmchlsmth and LucasWilkinson as code owners July 9, 2025 23:54

laithsakka changed the title ~~[WIP[ backed_Size_oblivious + pytorch2.8~~ [WIP] backed_Size_oblivious + pytorch 2.8 Jul 9, 2025

laithsakka marked this pull request as draft July 9, 2025 23:54

gemini-code-assist bot reviewed Jul 9, 2025

View reviewed changes

mergify bot added documentation Improvements or additions to documentation ci/build rocm Related to AMD ROCm labels Jul 9, 2025

gemini-code-assist bot reviewed Jul 9, 2025

View reviewed changes

laithsakka commented Jul 9, 2025

View reviewed changes

mergify bot added the needs-rebase label Jul 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[WIP] backed_Size_oblivious + pytorch 2.8 #20719

[WIP] backed_Size_oblivious + pytorch 2.8 #20719

Uh oh!

laithsakka commented Jul 9, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jul 9, 2025

Uh oh!

gemini-code-assist bot Jul 9, 2025

Uh oh!

gemini-code-assist bot Jul 9, 2025

Uh oh!

laithsakka Jul 9, 2025

Uh oh!

mergify bot commented Jul 17, 2025

Uh oh!

Uh oh!

Uh oh!

[WIP] backed_Size_oblivious + pytorch 2.8 #20719

Are you sure you want to change the base?

[WIP] backed_Size_oblivious + pytorch 2.8 #20719

Uh oh!

Conversation

laithsakka commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Essential Elements of an Effective PR Description Checklist

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

laithsakka Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Jul 17, 2025

Uh oh!

Uh oh!

laithsakka commented Jul 9, 2025 •

edited

Loading