Support returning multi-modal content from tools

[DouweM]

Fixes #1497

[hyperlint-ai]

PR Change Summary

Enhanced support for returning multi-modal content from tools, ensuring compatibility with various models.

• Implemented multi-modal content return for OpenAI, Gemini, and Anthropic tools.
• Updated documentation to include examples of function tool outputs.
• Clarified the registration process for function tools via decorators and agent arguments.

Modified Files

• docs/tools.md

---

How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.

[DouweM]

@Kludex Please review!

[Kludex]

What happens with the models that don't support this? I don't see mistral, groq, etc.

[DouweM]

What happens with the models that don't support this?

@Kludex The same thing that would happen if a developer tries to pass unsupported multi-modal content to the LLM as a user part: PydanticAI raises an error. I think that's better than silently serializing the content part as JSON instead, as just passing along the URL would be unlikely to have the desired result.

I don't see mistral, groq, etc.

I've added a test for Groq. I can add one for Mistral as well; is there a particular reason we're not actually calling to the API in test_mistral.py (and using VCR?)? Writing a test that fakes the entire API interaction seems to defeat the point as we specifically want to test that the LLM interprets this correctly :)

[DouweM]

Hmm, looks like Mistral, unlike the other 4 I tested, doesn't like this trick:

Unexpected role 'user' after role 'tool'

Edit: Got it to work!

[DouweM]

@Kludex Ready for review again!

[TheFirstMe]

@DouweM Does this also cover Bedrock (Claude)?

[DouweM]

@DouweM Does this also cover Bedrock (Claude)?

@TheFirstMe The implementation is model-agnostic -- if the model support multi-modal input, PydanticAI will try to send the file after the tool output. I haven't specifically tested Bedrock though; if you have an API token and could give it a try it'd be much appreciated!

[TheFirstMe]

@TheFirstMe The implementation is model-agnostic -- if the model support multi-modal input, PydanticAI will try to send the file after the tool output. I haven't specifically tested Bedrock though; if you have an API token and could give it a try it'd be much appreciated!

@DouweM I have tested and can verify that it works with bedrock. Thanks!

When can we expect this to be released?

[Wh1isper]

LGTM, just one thing I'm concerned about is that claude actually supports tool return image, and I'm not sure there will be a difference between that. I'm guessing the impact is minimal and I'll test it in my case later.