Question about the function call in chat template
Thank you so much for your hard work in developing these outstanding models and for generously making them available to the community. When investigating the config, we have some questions.
Below is the full chat template from tokenizer_config.json:
"{% if tools -%}\n {{ '<|im_system|>tool_declare<|im_middle|>' -}}\n {{- tools | tojson -}}\n {{ '<|im_end|>' -}}\n{%- endif -%}\n\n{%- for message in messages -%}\n {%- if loop.first and messages[0]['role'] != 'system' -%}\n {{ '<|im_system|>system<|im_middle|>You are a helpful assistant<|im_end|>' }}\n {%- endif -%}\n {%- if message['role'] == 'system' -%}\n {{ '<|im_system|>system<|im_middle|>' }}\n {%- elif message['role'] == 'user' -%}\n {{ '<|im_user|>user<|im_middle|>' }}\n {%- elif message['role'] == 'assistant' -%}\n {{ '<|im_assistant|>assistant<|im_middle|>' }}\n {%- elif message['role'] == 'tool' -%}\n {{ '<|im_system|>tool<|im_middle|>' }}\n {%- endif -%}\n\n {%- if message['content'] is string -%}\n {{- message['content'] + '<|im_end|>' -}}\n {%- else -%}\n {%- for content in message['content'] -%}\n {%- if content['type'] == 'image' or 'image' in content or 'image_url' in content -%}\n {{ '<|media_start|>image<|media_content|><|media_pad|><|media_end|>' }}\n {%- else -%}\n {{ content['text'] }}\n {%- endif -%}\n {%- endfor -%}\n {{ '<|im_end|>' }}\n {%- endif -%}\n{%- endfor -%}\n\n{%- if add_generation_prompt -%}\n {{ '<|im_assistant|>assistant<|im_middle|>' }}\n{%- endif -%}
It appears that this template isn’t correctly handling the assistant’s tool_calls. Am I missing something?
Thanks. The tool calling results from assistant is not handled by chat template, but need to handle by inference engine (e.g. tool parsing module in vLLM and SGLang). If your engine doesn't have tool parser module, you can refer to https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/docs/tool_call_guidance.md#manually-parsing-tool-calls to parse the tool_call related part manually.
Thanks. The tool calling results from assistant is not handled by chat template, but need to handle by inference engine (e.g. tool parsing module in vLLM and SGLang). If your engine doesn't have tool parser module, you can refer to https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/docs/tool_call_guidance.md#manually-parsing-tool-calls to parse the tool_call related part manually.
Thank you very much, will take a look!