Facing intermittent issue in DAG

[AshutoshSrivastavaDev]

I am facing an intermittent issue while I add a new dag file.

File "/usr/local/lib/python3.9/site-packages/pygments/lexers/special.py", line 13, in <module> from pygments.lexer import Lexer, line_re ImportError: cannot import name 'line_re' from 'pygments.lexer' (/usr/local/lib/python3.9/site-packages/pygments/lexer.py)

but if I rewrite the dag file with no changes the error goes away.

im using airflow v2.5.0
I have 3 master node and 3 worker nodes in the cluster.

I checked all three master nodes have pygments ==2.19.1

[potiuk]

Maybe you have some dynamic installation of packages that cause the problem. For example that could have happened if you dynamically installed extra packages after starting the image (which some people do rather than building their own custom images and we strongly advise them to do it instead).

[AshutoshSrivastavaDev]

@potiuk Thanks for the suggestion! However, this doesn’t seem to be the case — we don’t do any dynamic package installations on the master instances, as this is a production setup with all dependencies baked into the base image.

What’s puzzling is that this issue is intermittent — it affects roughly 10 out of 1000 DAGs at random, and often resolves itself after deleting and re-uploading the DAG file.

Could this be related to Airflow's file parsing race conditions or stale .pyc files in the scheduler/webserver process? Happy to dig deeper if there are debugging pointers.

[potiuk]

No - nothing special there. Maybe you have another pygments package on the path somewhere and it gets on the way? Or might be embedded oin your zip code? There is nothing special airflow does, it only runs importlib.import with the DAG, but yeah - if you have time synchronization issues - for example - between the instances, Python's internal handling of .pyc files based on timestamp might be causing some effects (but for that you'd have to have different versions of those files around.

The best thign you can do - modify that line of code that throws that expection and let it raise a more detailed - full stack-trace exception including looking at __path__ property of the module you have problem with to see where it is imported from ?

[AshutoshSrivastavaDev]

I have 3 instances which sync the dags every minute from an s3 location. could that be causing this issue?

[potiuk]

Could be - depend how you do it. If your syncing will DELETE dags to replace them with new ones, and there are periods of time that the dags are missing (even short ones) - yes that might happen. Improve your syncing process in this case so rather than deleting and recreating DAGs you replace them atomically . This is what git-sync does for example - it downloads the dags to folder that it has symbolic link to and after downloading a new copy of DAG folder it swaps the symbolic link to poin to the new directory

[AshutoshSrivastavaDev]

@potiuk Thanks again for the advice! I've made the sync process more atomic as suggested.

I have a very basic (maybe even naive) question:
I'm running the Airflow scheduler using:

nohup airflow scheduler > scheduler.log &

To clear the log as it grows, I occasionally do:
> scheduler.log

After doing this, the log file remains at 0 bytes. Does this mean the scheduler has stopped or is facing issues writing to the log file? Or is this expected behavior?

Also, I observed that on 2 out of our 3 master nodes, the scheduler.log file has been stuck at 0 bytes for the past few days—possibly since this import error started showing up. We restarted the Airflow scheduler on both of those nodes, and since then, the issue seems to have gone away. Could this have been the root cause?