[Frontend] Expand tools even if tool_choice="none"

[okdshin]

Add option to include tool definitions even when tool_choice is 'none'

Summary

This PR adds a new command-line option --expand-tools-even-if-tool-choice-none which allows including tool definitions in prompts even when tool_choice='none'.

Motivation

In the current implementation, when tool_choice is set to 'none', all tool definitions are removed from the request, preventing the model from seeing the tool schemas. This change enables a workflow where:

1. The model can be aware of available tools (via their definitions in the prompt)
2. But is not expected to use them automatically (since tool_choice='none')

This is useful for:

• Models that need to "plan" about available tools before using them in subsequent requests
• Cases where tool descriptions provide useful context even if the tools aren't used
• Situations where you want the model to reference tool capabilities in its response without actually calling them

Implementation

• Added a new CLI flag --expand-tools-even-if-tool-choice-none (default: False)
• Modified the request validation in protocol.py to no longer remove tools when tool_choice='none'
• Added the new parameter to OpenAIServingChat and passed it through from the API server

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[okdshin]

Hi @aarnphm, I've fixed the CI issues and updated this PR. The tests are now passing - could you please take another look when you have a chance? Thanks!

[aarnphm]

Do you have an use-case that requires the tool definition even if tool_choice="none"?

From a high level POV, having tool definition when I don't want to use tool seems like a waste of context for small/medium size model

[russellb]

Is the use case that you'd like to allow tool calls to be (potentially) generated, so you want all of the tool call definitions included, but you don't want the API server trying to parse any of it if they are there?

I'm just trying to make sure I understand properly..

[russellb]

I'm not a big fan of the new option, but honestly, the new behavior makes more sense to me. If tools have been defined in a request, it seems like including that data makes sense. Whether it's included or not seems implied by whether they're included in the request.

[okdshin]

Currently we are achieving tool_choice="auto" behavior by sending two requests from reverse proxy server to vLLM's API server:

• tool_choice="none" and guided_json={list_of_one_of_signature_of_tools}
• tool_choice="none"

If the first response was an empty json list, we assume the tool use was unnecessary and just use the second response. This is required because we want to use guided decoding (instead of relying on tool_parser) to enforce that the function call strictly aligns with the expected signature.

[aarnphm]

fyi 'tool_choice="required"' uses structured outputs here.

But I think I understand this case.

[okdshin]

I'm not a big fan of the new option,

Honestly, I agree with @russellb . Instead of adding a new option, could we just change the default behavior to be more consistent by always including tool definitions regardless of tool_choice setting?

fyi 'tool_choice="required"' uses structured outputs here.

Yes. required has already supported generally and has almost same behaviour when response is not empty list and it must be. auto is more complicated to implement properly.

[okdshin]

I found the discussion about why tools are dropped when tool_choice="none": #10000 (comment)
Dropping tools seems like overkill and wrong because it creates inconsistency with other modes (named-function, required and auto).
The modification here should be sufficient to skip tool parsing when the model hallucinates:
https://github.com/vllm-project/vllm/pull/10000/files#diff-190c665c438d34a7190da9a4d9bc1ed24bed8b13ee1b3f20c6da5c8aa52b0f3bR475-R476 .

[russellb]

Currently we are achieving tool_choice="auto" behavior by sending two requests from reverse proxy server to vLLM's API server:

• tool_choice="none" and guided_json={list_of_one_of_signature_of_tools}
• tool_choice="none"

If the first response was an empty json list, we assume the tool use was unnecessary and just use the second response. This is required because we want to use guided decoding (instead of relying on tool_parser) to enforce that the function call strictly aligns with the expected signature.

The newer structural_tag response format is designed to do this. You can express the requirements for all of the tool calls and it will enforce correct formatting whenever they occur in the middle of otherwise freeform text.

There's an example in here:

vllm/examples/online_serving/structured_outputs/structured_outputs.py

Lines 140 to 199 in 154d063

	"structural_tag": {
	"messages": [
	{
	"role": "user",
	"content": """
	You have access to the following function to retrieve the weather in a city:
	{
	"name": "get_weather",
	"parameters": {
	"city": {
	"param_type": "string",
	"description": "The city to get the weather for",
	"required": True
	}
	}
	}
	If a you choose to call a function ONLY reply in the following format:
	<{start_tag}={function_name}>{parameters}{end_tag}
	where
	start_tag => `<function`
	parameters => a JSON dict with the function argument name as key and function
	argument value as value.
	end_tag => `</function>`
	Here is an example,
	<function=example_function_name>{"example_name": "example_value"}</function>
	Reminder:
	- Function calls MUST follow the specified format
	- Required parameters MUST be specified
	- Only call one function at a time
	- Put the entire function call reply on one line
	- Always add your sources when using search results to answer the user query
	You are a helpful assistant.
	Given the previous instructions, what is the weather in New York City, Boston,
	and San Francisco?""",
	},
	],
	"response_format": {
	"type": "structural_tag",
	"structures": [
	{
	"begin": "<function=get_weather>",
	"schema": {
	"type": "object",
	"properties": {"city": {"type": "string"}},
	"required": ["city"],
	},
	"end": "</function>",
	}
	],
	"triggers": ["<function="],
	},
	},
	}

You can see in the example that the tool definition had to be manually included in the prompt. What you're proposing here, that it can (and should) be automatic, makes sense to me.

I think changing the default behavior, plus using the structural_tag response format, would probably give you the behavior you're looking for. Would you agree?

[okdshin]

Yes. I will modify this PR to remove the new option and change the default behaviour. I will check the structural_tag response format.
However, considering combinations with other response formats like <think>...(not json text)...</think> for reasoning feature, I think supporting something like llguidance's Lark format (guided_lark?) would ultimately provide the highest degree of freedom for users. But that's a separate discussion.

[okdshin]

I modified. Could you take another look?

[russellb]

I like this change, though I'll hold off on merging to give others more time to express an opinion

[aarnphm]

I afraid that this is breaking, given that requests relying on the prior behaviour would start breaking if the set of prompts + user message exceed the max context length of the model (unless long context like YaRN are being used).

Intuitively, tool_choice="none" implies that this request won't include any tool definitions. The current behaviour also aligns with OpenAI's logic for tool_choices="none" (see here)

none means the model will not call any tool and instead generates a message.

This doesn't mean that we have to strictly follow what OpenAI does. Just that if the goal is to be compatible, then I don't think this behaviour makes sense.

[okdshin]

What about changing the default as proposed, but adding --exclude-tools-when-none-choice for users who need the old behavior for context optimization?
This would restore the original behavior (before November) while providing an opt-out for users who prefer the current approach. The key asymmetry is that users wanting to avoid prompt tokens can send empty tools, but users wanting guided decoding with tool context currently have no automatic way to include tool definitions (the structural_tag example shows manual inclusion in the user message). This change would resolve that limitation while maintaining flexibility for both use cases.

[aarnphm]

opt-out is breaking right? For any production deployment. I think we should make this opt-in, then follow deprecating policy fwiw then break it in 0.10 at least.

@simon-mo do we have plans to release any patches before 0.10?

I'm fine with making this the default in 0.10, given that 0.10 will be considered breaking regardless.

[okdshin]

@aarnphm You're right about the breaking change concern. Let me revert commit 377f4ac to restore the opt-in approach with the --expand-tools-even-if-tool-choice-none flag.
This will allow users who need the new behavior to opt-in while keeping existing deployments unaffected.

Regarding the transition to 0.10 - could you help me understand the process for:
What's the recommended way to add deprecation warnings for this kind of behavioral change?
Are there any specific documentation or migration guide requirements we should prepare?

[okdshin]

@aarnphm @russellb I've addressed the review feedback by adding the deprecation warning 1ffd7f0 , updating the help text e3733a9 , and documenting the behavior change e282ebb . Could you take another look?

[aarnphm]

Let's not use assert in performance path here, if this is mostly for types the we can gate it in TYPE_CHECKING.

[okdshin]

@aarnphm Thanks for the review! I understand the performance concern, but I'd like to keep the assert here for a specific reason.

This assert serves as a defensive programming guard rather than just type checking. The logic is:

1. First condition: if request.tools is None → tool_dicts = None
2. Second condition: elif (request.tool_choice == "none" and not self.expand_tools_even_if_tool_choice_none)

The assert ensures that if someone modifies the first condition in the future (e.g., adds another OR condition), we'll catch the logic error immediately with a clear AssertionError, rather than getting a confusing AttributeError: 'NoneType' object has no attribute '__len__' when we call len(request.tools) below.

While I understand the performance concern, in the context of vLLM's request processing pipeline, this single assertion check is dwarfed by the actual bottlenecks like model inference, GPU operations, and network I/O. The cost of one conditional check per request is negligible compared to the milliseconds/seconds spent on actual LLM processing.

Given that trade-off, I think the defensive programming benefit outweighs the minimal performance cost. What do you think?

[okdshin]

@aarnphm You're absolutely right. Looking at this again, adding TYPE_CHECKING import just for this assert would be overkill, and the assert itself isn't really necessary here. Let me remove it and keep the code simple. Thanks for the guidance!

[aarnphm]

I'm good with this, can you look at the PR failure? seems like it is relevant to this PR.

[okdshin]

@aarnphm @russellb I've addressed the review feedback. I have no idea why CI failed, but there seems no failure now. I also tested tests/entrypoints/openai/test_completion_with_prompt_embeds.py on my local environment and ensured it passed. Could you take another look?

[aarnphm]

Ok, stamp from me, waiting for @russellb

[okdshin]

Hi @russellb,

Could you please take a look at the latest changes? The PR has been updated based on your previous suggestions about following the deprecation process. I believe this is now ready for your review.
If anything else needs to be fixed, just let me know - happy to make any additional changes.

[russellb]

apologies for the delay. looks good, thank you!