Skip to content

[Frontend] Expand tools even if tool_choice="none" #17177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jul 1, 2025
Merged

[Frontend] Expand tools even if tool_choice="none" #17177

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
ba347cd
add option "--expand-tools-even-if-tool-choice-none"
okdshin Jun 13, 2025
5b1b5e7
fix format
okdshin Jun 13, 2025
38eea42
fix format
okdshin Jun 14, 2025
bc031d9
remove the new option and change the default behaviour
okdshin Jun 17, 2025
b5e4164
Revert "remove the new option and change the default behaviour"
okdshin Jun 19, 2025
e3733a9
update help
okdshin Jun 21, 2025
1ffd7f0
add warn
okdshin Jun 21, 2025
e282ebb
fix docs
okdshin Jun 21, 2025
b51c7b9
fix format
okdshin Jun 21, 2025
0a0920a
fix format
okdshin Jun 21, 2025
90eabb2
Apply suggestions from code review
okdshin Jun 22, 2025
8a45fa8
remove assert
okdshin Jun 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/features/tool_calling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,14 @@ vLLM supports the `tool_choice='required'` option in the chat completion API. Si

When tool_choice='required' is set, the model is guaranteed to generate one or more tool calls based on the specified tool list in the `tools` parameter. The number of tool calls depends on the user's query. The output format strictly follows the schema defined in the `tools` parameter.

## None Function Calling

vLLM supports the `tool_choice='none'` option in the chat completion API. When this option is set, the model will not generate any tool calls and will respond with regular text content only, even if tools are defined in the request.

By default, when `tool_choice='none'` is specified, vLLM excludes tool definitions from the prompt to optimize context usage. To include tool definitions even with `tool_choice='none'`, use the `--expand-tools-even-if-tool-choice-none` option.

Note: This behavior will change in v0.10.0, where tool definitions will be included by default even with `tool_choice='none'`.

## Automatic Function Calling

To enable this feature, you should set the following flags:
Expand Down
2 changes: 2 additions & 0 deletions vllm/entrypoints/openai/api_server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1187,6 +1187,8 @@ async def init_app_state(
chat_template_content_format=args.chat_template_content_format,
return_tokens_as_token_ids=args.return_tokens_as_token_ids,
enable_auto_tools=args.enable_auto_tool_choice,
expand_tools_even_if_tool_choice_none=args.
expand_tools_even_if_tool_choice_none,
tool_parser=args.tool_call_parser,
reasoning_parser=args.reasoning_parser,
enable_prompt_tokens_details=args.enable_prompt_tokens_details,
Expand Down
10 changes: 10 additions & 0 deletions vllm/entrypoints/openai/cli_args.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,16 @@ def make_arg_parser(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
default=False,
help="Enable auto tool choice for supported models. Use "
"``--tool-call-parser`` to specify which parser to use.")
parser.add_argument(
"--expand-tools-even-if-tool-choice-none",
action="store_true",
default=False,
help="[DEPRECATED] Include tool definitions in prompts "
"even when tool_choice='none'. "
"This is a transitional option that will be removed in v0.10.0. "
"In v0.10.0, tool definitions will always be included regardless of "
"tool_choice setting. Use this flag now to test the new behavior "
"before the breaking change.")

valid_tool_parsers = ToolParserManager.tool_parsers.keys()
parser.add_argument(
Expand Down
4 changes: 1 addition & 3 deletions vllm/entrypoints/openai/protocol.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -686,10 +686,8 @@ def check_tool_usage(cls, data):
if "tool_choice" not in data and data.get("tools"):
data["tool_choice"] = "auto"

# if "tool_choice" is "none" -- ignore tools if present
# if "tool_choice" is "none" -- no validation is needed for tools
if "tool_choice" in data and data["tool_choice"] == "none":
# ensure that no tools are present
data.pop("tools", None)
return data

# if "tool_choice" is specified -- validation
Expand Down
25 changes: 22 additions & 3 deletions vllm/entrypoints/openai/serving_chat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ def __init__(
return_tokens_as_token_ids: bool = False,
reasoning_parser: str = "",
enable_auto_tools: bool = False,
expand_tools_even_if_tool_choice_none: bool = False,
tool_parser: Optional[str] = None,
enable_prompt_tokens_details: bool = False,
) -> None:
Expand Down Expand Up @@ -108,6 +109,8 @@ def __init__(
raise TypeError("Error: --enable-auto-tool-choice requires "
f"tool_parser:'{tool_parser}' which has not "
"been registered") from e
self.expand_tools_even_if_tool_choice_none = (
expand_tools_even_if_tool_choice_none)

self.enable_prompt_tokens_details = enable_prompt_tokens_details
self.default_sampling_params = (
Expand Down Expand Up @@ -172,9 +175,25 @@ async def create_chat_completion(
"--enable-auto-tool-choice and --tool-call-parser to be set"
)

tool_dicts = None if request.tools is None else [
tool.model_dump() for tool in request.tools
]
if request.tools is None:
tool_dicts = None
elif (request.tool_choice == "none"
and not self.expand_tools_even_if_tool_choice_none):
assert request.tools is not None
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not use assert in performance path here, if this is mostly for types the we can gate it in TYPE_CHECKING.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aarnphm Thanks for the review! I understand the performance concern, but I'd like to keep the assert here for a specific reason.

This assert serves as a defensive programming guard rather than just type checking. The logic is:

  1. First condition: if request.tools is None → tool_dicts = None
  2. Second condition: elif (request.tool_choice == "none" and not self.expand_tools_even_if_tool_choice_none)

The assert ensures that if someone modifies the first condition in the future (e.g., adds another OR condition), we'll catch the logic error immediately with a clear AssertionError, rather than getting a confusing AttributeError: 'NoneType' object has no attribute '__len__' when we call len(request.tools) below.

While I understand the performance concern, in the context of vLLM's request processing pipeline, this single assertion check is dwarfed by the actual bottlenecks like model inference, GPU operations, and network I/O. The cost of one conditional check per request is negligible compared to the milliseconds/seconds spent on actual LLM processing.

Given that trade-off, I think the defensive programming benefit outweighs the minimal performance cost. What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aarnphm You're absolutely right. Looking at this again, adding TYPE_CHECKING import just for this assert would be overkill, and the assert itself isn't really necessary here. Let me remove it and keep the code simple. Thanks for the guidance!

if len(request.tools) > 0:
logger.warning(
"Tools are specified but tool_choice is set to 'none' "
"and --expand-tools-even-if-tool-choice-none is not "
"enabled. Tool definitions will be excluded from the "
"prompt. This behavior will change in vLLM v0.10 where "
"tool definitions will be included by default even "
"with tool_choice='none'. To adopt the new behavior "
"now, use --expand-tools-even-if-tool-choice-none. "
"To suppress this warning, either remove tools from "
"the request or set tool_choice to a different value.")
tool_dicts = None
else:
tool_dicts = [tool.model_dump() for tool in request.tools]

(
conversation,
Expand Down