Inquiry: PLM Support in vLLM

[Hou-Kaiyuan]

I noticed in the latest Transformers pull request that perception_LM has been approved. Are there plans to add PLM support to vLLM as well?

[shuminghu]

I might be wrong but with the latest transformers code, vLLM support should come with it.

[shuminghu]

Unfornately the following doesn't work yet.

pip install vllm
pip install --upgrade git+https://github.com/huggingface/transformers.git
pip install timm
vllm serve facebook/Perception-LM-1B

PE is implemented as implemented in timm, which lacks meta tensor support yet. But I think that's a TODO and I hope that's implemented soon.

  File "/checkpoint/vision_encoder/envs/vllm/lib/python3.12/site-packages/timm/models/eva.py", line 1847, in vit_pe_core_large_patch14_336
    return _create_eva('vit_pe_core_large_patch14_336', pretrained=pretrained, **dict(model_args, **kwargs))
...
RuntimeError: Tensor.item() cannot be called on meta tensors

We wouldn't add support for vLLM through other methods.

[shuminghu]

PE/TIMM aside, native support for transformer VLMs in vllm is pending this PR.
vllm-project/vllm#20543

[Hou-Kaiyuan]

Thank you for digging into this and for sharing the PR link. I’ll keep tracking the progress on those. Really appreciate your help!