Skip to content

Common airflow.cfg files across all containers in default docker-compose.yaml #49681

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Apr 25, 2025
Merged

Common airflow.cfg files across all containers in default docker-compose.yaml #49681

merged 6 commits into from
Apr 25, 2025

Conversation

dheerajturaga
Copy link
Contributor

When running the default docker-compose.yaml each container auto generates a airflow.cfg in its respective home directory. This causes the jwt_secret to be different in all the containers causing tasks to fail with Invalid auth token: Signature verification failed

We can fix this multiple ways, but I feel the best way is to have a common AIRFLOW_HOME dir for all containers. Can be achived by directly mounting the AIRFLOW_PROJECT_DIR to /opt/airflow

Closes #49646

@dheerajturaga
Copy link
Contributor Author

cc: @amoghrajesh @bugraoz93

@dheerajturaga dheerajturaga changed the title Mount airflow project dir to containers airflow home Mount AIRFLOW_PROJECT_DIR to container's AIRFLOW_HOME Apr 24, 2025
@amoghrajesh amoghrajesh requested review from potiuk, kaxil and ashb April 24, 2025 06:10
@amoghrajesh
Copy link
Contributor

CC @ashb you would be better equipped to review this PR.

ashb
ashb reviewed Apr 24, 2025
View reviewed changes
@ashb ashb mentioned this pull request Apr 24, 2025
Closed
2 tasks
@ashb
Copy link
Member

ashb commented Apr 24, 2025

Ooh here's an idea then:

How about we mount a new config/ folder, and set the AIRFLOW_CONFIG option to point to that config/airflow.cfg -- this will remove the problem of "source doesn't exist, docker create it as a dir"

@dheerajturaga
Remove dags/logs/config/plugin mounts
6874db5 
They are no longer needed as we are directly mounting AIRFLOW_HOME
@dheerajturaga
Copy link
Contributor Author

@ashb, good idea, let me try that out

@dheerajturaga
Copy link
Contributor Author

@potiuk , @ashb
I uncommented the AIRFLOW_CONFIG variable in the docker-compose.yaml. They airlfow-init step should create it correctly in the config area and all other containers will share the common config.

Let me know if this looks good to y'all

@potiuk
Copy link
Member

potiuk commented Apr 24, 2025

@potiuk , @ashb I uncommented the AIRFLOW_CONFIG variable in the docker-compose.yaml. They airlfow-init step should create it correctly in the config area and all other containers will share the common config.

Let me know if this looks good to y'all

As long as the docs for quick-start explain it - sure, that can be ok. The problem is still that we can have potentially several airflow containers trying to wrie this file - so maybe worth updating the docs with instructions on how to initialise it - I think there is a "cli" command that could be run for this.

@dheerajturaga
Copy link
Contributor Author

@potiuk , @ashb I uncommented the AIRFLOW_CONFIG variable in the docker-compose.yaml. They airlfow-init step should create it correctly in the config area and all other containers will share the common config.
Let me know if this looks good to y'all

As long as the docs for quick-start explain it - sure, that can be ok. The problem is still that we can have potentially several airflow containers trying to wrie this file - so maybe worth updating the docs with instructions on how to initialise it - I think there is a "cli" command that could be run for this.

@potiuk , I have updated the docs with instructions on how to initialize the airflow.cfg. I've marked it as optional as airflow-init is a dependency for all other airflow services (condition: service_completed_successfully)

I also cleaned up the airflow-init step to mount the same volumes as the other services, so everything should be aligned

@dheerajturaga dheerajturaga changed the title Mount AIRFLOW_PROJECT_DIR to container's AIRFLOW_HOME Common airflow.cfg files across all containers in default docker-compose.yaml Apr 24, 2025
potiuk
potiuk approved these changes Apr 25, 2025
View reviewed changes
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ it! Thank you!

@potiuk potiuk merged commit ec49402 into apache:main Apr 25, 2025
51 checks passed
@potiuk potiuk added this to the Airflow 3.0.1 milestone Apr 25, 2025
potiuk pushed a commit to potiuk/airflow that referenced this pull request Apr 25, 2025
@dheerajturaga @amoghrajesh @potiuk
[v3-0-test] Common airflow.cfg files across all containers in default…
4e48eef 
… docker-compose.yaml (apache#49681)

* Mount airflow project dir to containers airflow home

* Remove dags/logs/config/plugin mounts

They are no longer needed as we are directly mounting AIRFLOW_HOME

* Set AIRFLOW_CFG in common env

* Update docs and airflow init step to align with common-env

* Fix spell check and static checks

---------
(cherry picked from commit ec49402)

Co-authored-by: Dheeraj Turaga <[email protected]>
Co-authored-by: Amogh Desai <[email protected]>
@potiuk potiuk mentioned this pull request Apr 25, 2025
Merged
potiuk added a commit that referenced this pull request Apr 25, 2025
@potiuk @dheerajturaga @amoghrajesh
[v3-0-test] Common airflow.cfg files across all containers in default…
917674b 
… docker-compose.yaml (#49681) (#49775)

* Mount airflow project dir to containers airflow home

* Remove dags/logs/config/plugin mounts

They are no longer needed as we are directly mounting AIRFLOW_HOME

* Set AIRFLOW_CFG in common env

* Update docs and airflow init step to align with common-env

* Fix spell check and static checks

---------
(cherry picked from commit ec49402)

Co-authored-by: Dheeraj Turaga <[email protected]>
Co-authored-by: Amogh Desai <[email protected]>
@dheerajturaga dheerajturaga deleted the common-cfg-across-containers branch April 25, 2025 11:41
@tirkarthi tirkarthi mentioned this pull request Apr 25, 2025
Closed
2 tasks
potiuk added a commit to potiuk/airflow that referenced this pull request Apr 26, 2025
@potiuk
Fix docker compose tests
965620c 
The docker compose tests on linux have been broken by apache#49681 because
of:

a) perrmission problems with config folder - airflow.cfg created
   eventually was owned by root, not airflow user and it was not
   readable by the airflow user
b) the airflow.cfg has not been initialized in init because airflow
   version command does not create config
c) config directory has not been crated in the docker-compose test
d) /opts/ directory was used to create dirs instead of /opt/ when
   changing permissions
e) chown -R does not work across the volumes

Also it turned out that diagnostic in case of health-check problms
has been broken and did not show the actual errors - because
while handling exception of health check api calls were made that
also raised exception that was subsequently silently swallowed and
did not allow the logs and heealth-check information from the test
to be printed.

This PR fixes those problems:

a) creates config folder during tests
b) runs "airflow config list" in init that actually creates a
   default config when no manual configuration is specified
c) changes ownership for the internal folders after the config
   file is created which allows airflow user to read it, also
   spearately changes ownership for /opt/airflow (volume in image)
   and all the shared volumes mounted from the host.
d) improves diagnostic by switching to rich print and handing the
   health exceptions during exception handling, allowing to print
   detailed logs of what happened
e) the output of `breeze testing docker-compose-tests` is printed
   directly to stdout (with pytest `-s` flag) - so that we see
   the progress of test as it happens - both locally and in CI.
@potiuk potiuk mentioned this pull request Apr 26, 2025
Merged
kaxil pushed a commit that referenced this pull request Apr 26, 2025
@potiuk
Fix docker compose tests (#49814)
67ce622 
The docker compose tests on linux have been broken by #49681 because
of:

a) perrmission problems with config folder - airflow.cfg created
   eventually was owned by root, not airflow user and it was not
   readable by the airflow user
b) the airflow.cfg has not been initialized in init because airflow
   version command does not create config
c) config directory has not been crated in the docker-compose test
d) /opts/ directory was used to create dirs instead of /opt/ when
   changing permissions
e) chown -R does not work across the volumes

Also it turned out that diagnostic in case of health-check problms
has been broken and did not show the actual errors - because
while handling exception of health check api calls were made that
also raised exception that was subsequently silently swallowed and
did not allow the logs and heealth-check information from the test
to be printed.

This PR fixes those problems:

a) creates config folder during tests
b) runs "airflow config list" in init that actually creates a
   default config when no manual configuration is specified
c) changes ownership for the internal folders after the config
   file is created which allows airflow user to read it, also
   spearately changes ownership for /opt/airflow (volume in image)
   and all the shared volumes mounted from the host.
d) improves diagnostic by switching to rich print and handing the
   health exceptions during exception handling, allowing to print
   detailed logs of what happened
e) the output of `breeze testing docker-compose-tests` is printed
   directly to stdout (with pytest `-s` flag) - so that we see
   the progress of test as it happens - both locally and in CI.
github-actions bot pushed a commit that referenced this pull request Apr 26, 2025
@potiuk
[v3-0-test] Fix docker compose tests (#49814)
60daea2 
The docker compose tests on linux have been broken by #49681 because
of:

a) perrmission problems with config folder - airflow.cfg created
   eventually was owned by root, not airflow user and it was not
   readable by the airflow user
b) the airflow.cfg has not been initialized in init because airflow
   version command does not create config
c) config directory has not been crated in the docker-compose test
d) /opts/ directory was used to create dirs instead of /opt/ when
   changing permissions
e) chown -R does not work across the volumes

Also it turned out that diagnostic in case of health-check problms
has been broken and did not show the actual errors - because
while handling exception of health check api calls were made that
also raised exception that was subsequently silently swallowed and
did not allow the logs and heealth-check information from the test
to be printed.

This PR fixes those problems:

a) creates config folder during tests
b) runs "airflow config list" in init that actually creates a
   default config when no manual configuration is specified
c) changes ownership for the internal folders after the config
   file is created which allows airflow user to read it, also
   spearately changes ownership for /opt/airflow (volume in image)
   and all the shared volumes mounted from the host.
d) improves diagnostic by switching to rich print and handing the
   health exceptions during exception handling, allowing to print
   detailed logs of what happened
e) the output of `breeze testing docker-compose-tests` is printed
   directly to stdout (with pytest `-s` flag) - so that we see
   the progress of test as it happens - both locally and in CI.
(cherry picked from commit 67ce622)

Co-authored-by: Jarek Potiuk <[email protected]>
@github-actions github-actions bot mentioned this pull request Apr 26, 2025
Merged
potiuk added a commit that referenced this pull request Apr 26, 2025
@github-actions @potiuk
[v3-0-test] Fix docker compose tests (#49814) (#49817)
884b335 
The docker compose tests on linux have been broken by #49681 because
of:

a) perrmission problems with config folder - airflow.cfg created
   eventually was owned by root, not airflow user and it was not
   readable by the airflow user
b) the airflow.cfg has not been initialized in init because airflow
   version command does not create config
c) config directory has not been crated in the docker-compose test
d) /opts/ directory was used to create dirs instead of /opt/ when
   changing permissions
e) chown -R does not work across the volumes

Also it turned out that diagnostic in case of health-check problms
has been broken and did not show the actual errors - because
while handling exception of health check api calls were made that
also raised exception that was subsequently silently swallowed and
did not allow the logs and heealth-check information from the test
to be printed.

This PR fixes those problems:

a) creates config folder during tests
b) runs "airflow config list" in init that actually creates a
   default config when no manual configuration is specified
c) changes ownership for the internal folders after the config
   file is created which allows airflow user to read it, also
   spearately changes ownership for /opt/airflow (volume in image)
   and all the shared volumes mounted from the host.
d) improves diagnostic by switching to rich print and handing the
   health exceptions during exception handling, allowing to print
   detailed logs of what happened
e) the output of `breeze testing docker-compose-tests` is printed
   directly to stdout (with pytest `-s` flag) - so that we see
   the progress of test as it happens - both locally and in CI.
(cherry picked from commit 67ce622)

Co-authored-by: Jarek Potiuk <[email protected]>
jroachgolf84 pushed a commit to jroachgolf84/airflow that referenced this pull request Apr 30, 2025
@dheerajturaga @amoghrajesh @jroachgolf84
Common airflow.cfg files across all containers in default docker-comp…
b967f2d 
…ose.yaml (apache#49681)

* Mount airflow project dir to containers airflow home

* Remove dags/logs/config/plugin mounts

They are no longer needed as we are directly mounting AIRFLOW_HOME

* Set AIRFLOW_CFG in common env

* Update docs and airflow init step to align with common-env

* Fix spell check and static checks

---------

Co-authored-by: Amogh Desai <[email protected]>
jroachgolf84 pushed a commit to jroachgolf84/airflow that referenced this pull request Apr 30, 2025
@potiuk @jroachgolf84
Fix docker compose tests (apache#49814)
d28a9b7 
The docker compose tests on linux have been broken by apache#49681 because
of:

a) perrmission problems with config folder - airflow.cfg created
   eventually was owned by root, not airflow user and it was not
   readable by the airflow user
b) the airflow.cfg has not been initialized in init because airflow
   version command does not create config
c) config directory has not been crated in the docker-compose test
d) /opts/ directory was used to create dirs instead of /opt/ when
   changing permissions
e) chown -R does not work across the volumes

Also it turned out that diagnostic in case of health-check problms
has been broken and did not show the actual errors - because
while handling exception of health check api calls were made that
also raised exception that was subsequently silently swallowed and
did not allow the logs and heealth-check information from the test
to be printed.

This PR fixes those problems:

a) creates config folder during tests
b) runs "airflow config list" in init that actually creates a
   default config when no manual configuration is specified
c) changes ownership for the internal folders after the config
   file is created which allows airflow user to read it, also
   spearately changes ownership for /opt/airflow (volume in image)
   and all the shared volumes mounted from the host.
d) improves diagnostic by switching to rich print and handing the
   health exceptions during exception handling, allowing to print
   detailed logs of what happened
e) the output of `breeze testing docker-compose-tests` is printed
   directly to stdout (with pytest `-s` flag) - so that we see
   the progress of test as it happens - both locally and in CI.
@dheerajturaga dheerajturaga mentioned this pull request Apr 30, 2025
Closed
@Lee2532 Lee2532 mentioned this pull request Apr 30, 2025
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ashb ashb ashb left review comments

@potiuk potiuk potiuk approved these changes

@kaxil kaxil Awaiting requested review from kaxil

Assignees
No one assigned
Labels
kind:documentation
Projects
None yet
Milestone
Airflow 3.0.1
Development

Successfully merging this pull request may close these issues.

Worker always return Invalid auth token when setting up demo environment using docker compose
4 participants
@dheerajturaga @amoghrajesh @ashb @potiuk