add additional Otel traces #48658

praphi-git · 2025-04-02T07:04:53Z

praphi-git
Apr 2, 2025

Hello.

i am using Airflow 2.10.5 and able to see the traces or my DAGs.

However i would like to add custom attributes to spans for a specific task within the DAG. Can you please help if there is a method to do this ?

Also can you please point me to the docs page to extend the capabilities of traces generated by DAGs

Answered by howardyoo

Apr 10, 2025

Hi Praphi, if you feel like something is not working, please follow the procedure of creating git issues in Airflow. I will take a look at the link whenever I have time, but right now, I'm occupied with some other works that I have to attend to.

View full answer

praphi-git · 2025-04-08T12:41:03Z

praphi-git
Apr 8, 2025
Author

Thank you @howardyoo

I was able to use the airflow_provider_opentelemetry and add a new span, however if if i would like to add an attribute to existing task's span.

Can you help here ?

0 replies

howardyoo · 2025-04-08T15:41:59Z

howardyoo
Apr 8, 2025

Hmm.. I'm not sure whether you can explicitly add an attribute to the task span, unfortunately. This may be a work for the future. Howard

…

On Tue, Apr 8, 2025 at 7:41 AM praphi-git ***@***.***> wrote: Thank you @howardyoo <https://github.com/howardyoo> I was able to use the airflow_provider_opentelemetry and add a new span, however if if i would like to add an attribute to existing task's span. Can you help here ? — Reply to this email directly, view it on GitHub <#48658 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHZNLLRSVRQJV5PZGRVT65T2YO7XLAVCNFSM6AAAAAB2IXLZPWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENZWGQZDMNY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

0 replies

praphi-git · 2025-04-09T10:59:02Z

praphi-git
Apr 9, 2025
Author

ok. Thanks for the reply @howardyoo

I have another question where i need to add custom attributes to tracer to send traces in my Organization.

i think i came across a configuration where we can add custom attributes to Tracer. Can you help me point to the airflow configuration

below is psuedo code to relate my requirement

resource = Resource(attributes={
        "my_key": "value"
    }

trace.set_tracer_provider(TracerProvider(resource=resource))

0 replies

howardyoo · 2025-04-09T12:35:07Z

howardyoo
Apr 9, 2025

No worries. You don't have to configure resource attributes using SDK, but I would suggest setting OTEL_RESOURCE_ATTRIBUTES environment variable when you're running Airflow that will make the tracer attach those when running. More detail is here: https://opentelemetry.io/docs/specs/otel/resource/sdk/#specifying-resource-information-via-an-environment-variable

…

On Wed, Apr 9, 2025 at 5:59 AM praphi-git ***@***.***> wrote: ok. Thanks for the reply @howardyoo <https://github.com/howardyoo> I have another question where i need to add custom attributes to tracer to send traces in my Organization. i think i came across a configuration where we can add custom attributes to Tracer. Can you help me point to the airflow configuration below is psuedo code to relate my requirement resource = Resource(attributes={ "my_key": "value" } trace.set_tracer_provider(TracerProvider(resource=resource)) — Reply to this email directly, view it on GitHub <#48658 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHZNLLTCXGBWUPTTHYVIBMT2YT4QZAVCNFSM6AAAAAB2IXLZPWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENZXGY3TKOI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

3 replies

praphi-git Apr 10, 2025
Author

@howardyoo i've set the environment variables and ran the airflow, it's not picking up the attributes.

I have tried with sample code using the SDK in the same environment and the values from OTEL_RESOURCE_ATTRIBUTES
are being picked up. Looks like an issue with Airflow itself. can you check once ?

For you reference i am sharing the the snippet from sample code used to create TracerProvider

trace.set_tracer_provider(
    TracerProvider(
        # resource=Resource.create({SERVICE_NAME: "my-hello-service"})
        resource=Resource.create()
    )
)

praphi-git Apr 10, 2025
Author

@howardyoo I have posted the sample code and issue in detail as a comment to your reply. Can you please see when you get a chance - link ?

howardyoo Apr 10, 2025

Yes, the OTEL_RESOURCE_ATTRIBUTES, will be picked up by tracer provider, as long as it exists and available.
I believe there is a way to define specific env variable in Airflow so that it can be available globally. Since I'm not a airflow expert, I don't know exactly how to do it, but perhaps you may be able to find out via asking in the community forum / mailing list. I would also check whether the OTEL_RESOURCE_ATTRIBUTES are available during the task's running by printout out the env variables, for example, while running a python operator - to verify whether it exists.

praphi-git · 2025-04-10T06:00:03Z

praphi-git
Apr 10, 2025
Author

ok thank you will use OTEL_RESOURCE_ATTRIBUTES.

i got confused tags which i found it after browsing through the code.

    def get_constant_tags(cls) -> str | None:
        """Get constant tags to add to all traces."""
        tags_in_string = conf.get("traces", "tags", fallback=None)

Haven't found documentation for this in docs page. will this conf(tags) be maintained moving forward ?

0 replies

howardyoo · 2025-04-10T13:20:34Z

howardyoo
Apr 10, 2025

Hi Praphi, Oh yes, I now remember. https://stackoverflow.com/questions/69839049/how-do-we-set-os-environment-variables-in-airflow So, since tasks could be executed on separate processes, the env variable in normal way may not work in Airflow, so that's why I have implemented the code def get_constant_tags(cls) -> str | None: """Get constant tags to add to all traces.""" tags_in_string = conf.get("traces", "tags", fallback=None) So, yes, this works - and you could put the tags inside your airflow configuration under traces to make it work similar to the resource attributes.

…

On Thu, Apr 10, 2025 at 2:54 AM praphi-git ***@***.***> wrote: @howardyoo <https://github.com/howardyoo> i set the environment variables to ran the airflow. it's not picking up the attributes. I have tried using the SDK in the same environment and the values are being picked up. Looks like an issue with Airflow itself. can you check once ? — Reply to this email directly, view it on GitHub <#48658 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHZNLLXX4YK7XMK2DQTN6NT2YYPSTAVCNFSM6AAAAAB2IXLZPWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENZYHAZDKNA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

0 replies

praphi-git · 2025-04-10T14:58:42Z

praphi-git
Apr 10, 2025
Author

@howardyoo

Looks like you are also confused with the attributes for Tracer Vs span

i think tags is used for setting up attributes to span.

OTEL_RESOURCE_ATTRIBUTES variable is used to set resource attributes to tracer. However its not working.

0 replies

howardyoo · 2025-04-10T21:26:58Z

howardyoo
Apr 10, 2025

Hi Praphi, if you feel like something is not working, please follow the procedure of creating git issues in Airflow. I will take a look at the link whenever I have time, but right now, I'm occupied with some other works that I have to attend to.

…

On Thu, Apr 10, 2025 at 10:00 AM praphi-git ***@***.***> wrote: @howardyoo <https://github.com/howardyoo> I have posted the sample code and issue in detail as a comment to your reply. Can you please see when you get a chance - link <#48658 (reply in thread)> ? — Reply to this email directly, view it on GitHub <#48658 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHZNLLQXLE2JYNDJT3GMMS32Y2BRZAVCNFSM6AAAAAB2IXLZPWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENZZGI4TQNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

1 reply

praphi-git Apr 11, 2025
Author

someone else reported it yesterday. - #49053

howardyoo · 2025-04-12T19:05:49Z

howardyoo
Apr 12, 2025

That is nice. So, I did try testing the OTEL_RESOURCE_ATTRIBUTES, and in the breeze testing mode, it seemed to work fine for me. Airflow deployments can vary from user to user, but this may help you to verify whether the env variable is working or not. 1. clone the airflow to your local env. 2. follow the instruction of how to setup breeze in your local env. : https://github.com/apache/airflow/tree/main/dev/breeze 3. Add the env variable OTEL_RESOURCE_ATTRIBUTES for your integration test. see the screenshot. [image: image.png] What this does is that when you're running breeze, this would insert these env variable to the docker container that's running the airflow in dev mode. The integration test is quite helpful when testing otel related functions. 4. run breeze with '--integration otel' which conveniently sets up otel collector, grafana, and jaeger for getting the traces breeze start-airflow --integration otel You'll see the following containers started up in your docker desktop [image: image.png] Now, let's check out the traces. Go to jaeger. It should be running on port 36686. There should be a few traces flowing in. Click one of the traces (circle dot) [image: image.png] Now, check for the resource attributes that we put in. You can see that the 'environment = breeze' and 'purpose = integration-test' are shown here, meaning OTEL_RESOURCE_ATTRIBUTES are being properly used. So, according to a quick test, it looks like the env. variable is working properly, and being used when airflow is running, emitting the resource attributes properly. Now, not every Airflow deployment environment is same, and there may be cases where env variables may not exist when you're running airflow processes. Make sure when Airflow is running, or any executors are running, there exists the necessary env variable for it to read them. Check thoroughly. Hope this helps, Howard

…

On Fri, Apr 11, 2025 at 4:00 AM praphi-git ***@***.***> wrote: someone else reported it yesterday. - #49053 <#49053> — Reply to this email directly, view it on GitHub <#48658 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHZNLLSRBMXTAHPAJMEC77L2Y6ACHAVCNFSM6AAAAAB2IXLZPWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBQGE2DQNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

1 reply

praphi-git Apr 14, 2025
Author

Maybe its working for you since you are using main branch ?

Please see comments on 49053.

The fixes in #41840 are only in the main branch and not in the 2.10.5 tag, the fix in particular is related to

This is due to
resource = Resource(attributes={HOST_NAME: get_hostname(), SERVICE_NAME: self.otel_service})

Vs

        self.resource = Resource.create(
            attributes={HOST_NAME: get_hostname(), SERVICE_NAME: self.otel_service}

howardyoo · 2025-04-14T15:10:05Z

howardyoo
Apr 14, 2025

Yes, I believe that's correct. In that case, you might have to bump your Airflow version to the more recent version in order to have this working.

…

On Mon, Apr 14, 2025 at 2:17 AM praphi-git ***@***.***> wrote: Maybe its working for you since you are using main branch ? Please see comments on 49053 <#49053 (comment)>. The fixes in #41840 <#41840> are only in the main branch and not in the 2.10.5 tag, the fix in particular is related to This is due to resource = Resource(attributes={HOST_NAME: get_hostname(), SERVICE_NAME: self.otel_service}) Vs self.resource = Resource.create( attributes={HOST_NAME: get_hostname(), SERVICE_NAME: self.otel_service} — Reply to this email directly, view it on GitHub <#48658 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHZNLLWVJBB7CYGBMTT3HFL2ZNOJRAVCNFSM6AAAAAB2IXLZPWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBSGU2DIMA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

1 reply

praphi-git Apr 16, 2025
Author

Thanks for your confirmation. I am on the latest version - Airflow 2.10.5

Probably have to wait for it to be pulled it into next release.

howardyoo · 2025-04-16T13:43:36Z

howardyoo
Apr 16, 2025

Yeah, Currently Airflow team is busy on the new Airflow 3.0 release, so I am not sure whether the update on the main branch would be applied to 2.10.5.... You might need to raise an issue or create a discussion on the airflow community mailing list to get a better answer...

…

On Wed, Apr 16, 2025 at 5:12 AM praphi-git ***@***.***> wrote: Thanks for your confirmation. I am on the latest version - Airflow 2.10.5 Probably have to wait for it to be pulled it into next release. — Reply to this email directly, view it on GitHub <#48658 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHZNLLW57PV45AE33CIVC4L2ZYUKLAVCNFSM6AAAAAB2IXLZPWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBVGI4DKNY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add additional Otel traces #48658

{{title}}

Replies: 11 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

add additional Otel traces #48658

Replies: 11 comments · 6 replies

praphi-git Apr 8, 2025 Author

praphi-git Apr 9, 2025 Author

praphi-git Apr 10, 2025 Author

praphi-git Apr 10, 2025 Author

praphi-git Apr 10, 2025 Author

praphi-git Apr 10, 2025 Author

praphi-git Apr 11, 2025 Author

praphi-git Apr 14, 2025 Author

praphi-git Apr 16, 2025 Author

Replies: 11 comments 6 replies

praphi-git
Apr 8, 2025
Author

praphi-git
Apr 9, 2025
Author

praphi-git Apr 10, 2025
Author

praphi-git Apr 10, 2025
Author

praphi-git
Apr 10, 2025
Author

praphi-git
Apr 10, 2025
Author

praphi-git Apr 11, 2025
Author

praphi-git Apr 14, 2025
Author

praphi-git Apr 16, 2025
Author