Scheduler is unable to read logs

[nazarii-koblai]

Apache Airflow version

2.10.5

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Do consistently observe such error in scheduler logs. Were assuming someone closes the DB session explicitly within one of the DAGs, but I didn't found such place.

[2025-03-11T10:06:48.720+0000] {file_task_handler.py:623} ERROR - Could not read served logs Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/log/file_task_handler.py", line 599, in _read_from_logs_server log_type = LogType.TRIGGER if ti.triggerer_job else LogType.WORKER ^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/ext/associationproxy.py", line 193, in __get__ return inst.get(obj) ^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/ext/associationproxy.py", line 575, in get target = getattr(obj, self.target_collection) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/attributes.py", line 487, in __get__ return self.impl.get(state, dict_) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/attributes.py", line 959, in get value = self._fire_loader_callables(state, key, passive) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/attributes.py", line 995, in _fire_loader_callables return self.callable_(state, passive) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/strategies.py", line 863, in _load_for_state raise orm_exc.DetachedInstanceError( sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <TaskInstance at 0x7fe4d65a39b0> is not bound to a Session; lazy load operation of attribute 'trigger' cannot proceed (Background on this error at: https://sqlalche.me/e/14/bhk3)

What you think should happen instead?

Errors should not appear

How to reproduce

Hard to say. Maybe its only related to my particular setup. But there are no something special.

Operating System

Linux

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[karenbraganz]

I have converted this to a discussion since there are no reproduction steps. I have added some questions below that would help gather more context for this discussion.

• Are seeing this in your scheduler logs or task logs in the Airflow UI? I suspect it is from the task logs because the error originates from the FileTaskHandler.
• Are you using callbacks in your task and attempting to retrieve logs within the callback? I am asking this based on the report of a similar issue.
• Are there any logs that appear before this for the task? If so, please add them here.
• Does this always occur on the same DAG/ task?

I recommend posting your DAG code for most context if possible.

[nazarii-koblai]

@karenbraganz
I will send an image as a reply

[yovio-rca]

This issue happened to me as well.

Airflow 2.10.5
CeleryKubernetesExecutor
dagProcessor enable (separate dag processor process)

Error from scheduler service log.
I have callbacks in my operator but not trying to read log in the callback.

It happened on various dags.

[yovio-rca]

I think I may know what causing this.

in my case, my airflow.cfg was like this:

[core]
remote_logging = 'False' -> this was set by default value from helm chart, I did not set this to True because it marked deprecated.

[loging]
remote_logging = 'True' -> This is what I want, I want remote logging to S3

When I set the value to True for both location, the error message disappear.

I think the comment on help chart need to be updated that we need to set the value on both locations? or maybe the code should stop looking at core.remote_logging

[yovio-rca]

I speak too soon. Problem still exists.