-
-
Notifications
You must be signed in to change notification settings - Fork 8.5k
[Misc] Add gemma3 chat template with pythonic-style function calling #17149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
philipchung
wants to merge
1
commit into
vllm-project:main
Choose a base branch
from
philipchung:main
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+123
−0
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,123 @@ | ||
{#- Begin-of-sequence token to start the model prompt -#} | ||
{{ bos_token }} | ||
{#- Extracts the system message. Gemma does not support system messages so it will be prepended to first user message. -#} | ||
{%- if messages[0]['role'] == 'system' -%} | ||
{%- if messages[0]['content'] is string -%} | ||
{%- set first_user_prefix = messages[0]['content'] + '\n\n' -%} | ||
{%- else -%} | ||
{%- set first_user_prefix = messages[0]['content'][0]['text'] + '\n\n' -%} | ||
{%- endif -%} | ||
{%- set loop_messages = messages[1:] -%} | ||
{%- else -%} | ||
{%- set first_user_prefix = "" -%} | ||
{%- set loop_messages = messages -%} | ||
{%- endif -%} | ||
{#- Set tools to none if not defined for this ChatCompletion request (helps avoid errors later) -#} | ||
{%- if not tools is defined %} | ||
{%- set tools = none %} | ||
{%- endif %} | ||
{#- Validate alternating user/assistant messages (excluding 'tool' messages and ones with tool_calls) -#} | ||
{%- for message in loop_messages | rejectattr("role", "equalto", "tool") | selectattr("tool_calls", "undefined") -%} | ||
{%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) %} | ||
{{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }} | ||
{%- endif -%} | ||
{%- endfor -%} | ||
|
||
{#- Main loop over all messages in the conversation history -#} | ||
{%- for message in loop_messages -%} | ||
{#- Normalize roles for model prompt formatting -#} | ||
{%- if (message['role'] == 'assistant') -%} | ||
{%- set role = "model" -%} | ||
{%- elif (message['role'] == 'tool') -%} | ||
{%- set role = "user" -%} | ||
{%- else -%} | ||
{%- set role = message['role'] -%} | ||
{%- endif -%} | ||
{#- Mark the start of a message block with the appropriate role -#} | ||
{{ '<start_of_turn>' + role + '\n' -}} | ||
|
||
{#- Insert system message content (if present) at the beginning of the first message. -#} | ||
{%- if loop.first -%} | ||
{{ first_user_prefix }} | ||
{#- Append system message with tool information if using tools in message request. -#} | ||
{%- if tools is not none -%} | ||
{{- "Tools (functions) are available. If you decide to invoke one or more of the tools, you must respond with a python list of the function calls.\n" -}} | ||
{{- "Example Format: [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)] \n" -}} | ||
{{- "Do not use variables. DO NOT USE MARKDOWN SYNTAX. You SHOULD NOT include any other text in the response if you call a function. If none of the functions can be used, point it out. If you lack the parameters required by the function, also point it out.\n" -}} | ||
{{- "Here is a list of functions in JSON format that you can invoke.\n" -}} | ||
{{- tools | tojson(indent=4) -}} | ||
{{- "\n\n" -}} | ||
{%- endif -%} | ||
{%- endif -%} | ||
|
||
{#- Format model tool calls (turns where model indicates they want to call a tool) -#} | ||
{%- if 'tool_calls' in message -%} | ||
{#- Opening bracket for tool call list. -#} | ||
{{- '[' -}} | ||
{#- For each tool call -#} | ||
{%- for tool_call in message.tool_calls -%} | ||
{#- Get tool call function. -#} | ||
{%- if tool_call.function is defined -%} | ||
{%- set tool_call = tool_call.function -%} | ||
{%- endif -%} | ||
{#- Function name & opening parenthesis. -#} | ||
{{- tool_call.name + '(' -}} | ||
|
||
{#-- Handle arguments as list (positional) or dict (named) --#} | ||
{#-- Named arguments (dict) --#} | ||
{%- if tool_call.arguments is iterable and tool_call.arguments is mapping -%} | ||
{%- set first = true -%} | ||
{%- for key, val in tool_call.arguments.items() -%} | ||
{%- if not first %}, {% endif -%} | ||
{{ key }}={{ val | tojson }} | ||
{%- set first = false -%} | ||
{%- endfor -%} | ||
{#-- Positional arguments (list) --#} | ||
{%- elif tool_call.arguments is iterable -%} | ||
{{- tool_call.arguments | map('tojson') | join(', ') -}} | ||
{#-- Fallback: single positional value --#} | ||
{%- else -%} | ||
{{- tool_call.arguments | tojson -}} | ||
{#-- Closing parenthesis. --#} | ||
{%- endif -%} | ||
{{- ')' -}} | ||
{#-- If more than one tool call, place comma and move to formatting next tool call --#} | ||
{%- if not loop.last -%}, {% endif -%} | ||
{%- endfor -%} | ||
{#- Closing bracket for tool call list. -#} | ||
{{- ']' -}} | ||
{%- endif -%} | ||
|
||
{#- Tool response start tag (for messages from a tool) -#} | ||
{%- if (message['role'] == 'tool') -%} | ||
{{ '<tool_response>\n' -}} | ||
{%- endif -%} | ||
|
||
{#- Render the message content: handle plain string or multimodal content like image/text -#} | ||
{%- if message['content'] is string -%} | ||
{{ message['content'] | trim }} | ||
{%- elif message['content'] is iterable -%} | ||
{%- for item in message['content'] -%} | ||
{%- if item['type'] == 'image' -%} | ||
{{ '<start_of_image>' }} | ||
{%- elif item['type'] == 'text' -%} | ||
{{ item['text'] | trim }} | ||
{%- endif -%} | ||
{%- endfor -%} | ||
{%- else -%} | ||
{{ raise_exception("Invalid content type") }} | ||
{%- endif -%} | ||
|
||
{#- Tool response end tag -#} | ||
{%- if (message['role'] == 'tool') -%} | ||
{{ '</tool_response>' -}} | ||
{%- endif -%} | ||
|
||
{#- Mark end of a single turn -#} | ||
{{ '<end_of_turn>\n' }} | ||
{%- endfor -%} | ||
|
||
{#- If generation is to be triggered, add model prompt prefix -#} | ||
{%- if add_generation_prompt -%} | ||
{{'<start_of_turn>model\n'}} | ||
{%- endif -%} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gemma-3 does not handle tool definitions being added after the system prompt properly. When it is switched around it seems to be ok
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I cannot for the life of me get Gemma3 to consistently place the tool call at the beginning of the response. Despite reiterating that it needs to be there within the system prompt and this template containing the same, more often than not, I see Gemma place the tool call at the end of the response.
Is this a limitation of the template? Can I not have it detect a tool call anywhere in the response? I would really love to continue using Gemma3 but we might have to switch if we can't get more robust with the function calling support.