[BUG]: LLM Observability Spans Not Appearing When Payload > 1MB

[lucasiscovici]

Tracer Version(s)

3.2.1

Python Version(s)

3.12

Pip Version(s)

25.0.1

Bug Report

Hello,

Context

It seems that when the payload sent to the LLM Observability endpoint (llmobs-intake.datadoghq.eu) exceeds 1MB, the corresponding LLM Obs spans are not correctly ingested. Specifically, we notice that:

• No LLM Obs spans appear in the platform.
• There are no links between APM spans and LLM Obs spans.

In the codebase:

• The dd-trace library sets the max payload size as EVP_PAYLOAD_SIZE_LIMIT = 5 << 20 (i.e. 5MB, with an actual limit of 5.1MB).
• The Datadog Agent configures the EVP proxy with MaxPayloadSize: 5 * 1024 * 1024.

Question

Does the LLM Observability intake endpoint (llmobs-intake.datadoghq.eu) support and process payloads larger than 1MB? If not, is there a recommended way to handle larger payloads for LLM Obs spans?

Thanks in advance!

Reproduction Code

No response

Error Logs

No response

Libraries in Use

No response

Operating System

No response

[Yun-Kim]

Hi @lucasiscovici, thanks for reaching out!

There should be a bugfix #12890 that resolves an issue where payloads were incorrectly treated as individual events, meaning previously all payloads over 1MB were rejected in our intake. If you upgrade your tracer version to latest or any after ddtrace>=3.3, then you should start seeing traces again. The caveat here is that it will stlil likely truncate the I/O of any individual span events >1MB, which we are still working on resolving in our intake platform.

Please let us know if upgrading doesn't resolve your issue, thanks!