[v3-0-test] Common airflow.cfg files across all containers in default docker-compose.yaml (#49681) (#49775)

* Mount airflow project dir to containers airflow home

* Remove dags/logs/config/plugin mounts

They are no longer needed as we are directly mounting AIRFLOW_HOME

* Set AIRFLOW_CFG in common env

* Update docs and airflow init step to align with common-env

* Fix spell check and static checks

---------
(cherry picked from commit ec49402)

Co-authored-by: Dheeraj Turaga <[email protected]>
Co-authored-by: Amogh Desai <[email protected]>

Author: 3 people

airflow-core/docs/howto/docker-compose/docker-compose.yaml

@@ -71,8 +71,7 @@ x-airflow-common:
     # for other purpose (development, test and especially production usage) build/extend Airflow image.
     _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
     # The following line can be used to set a custom config file, stored in the local config folder
-    # If you want to use it, outcomment it and replace airflow.cfg with the name of your config file
-    # AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg'
+    AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg'
   volumes:
     - ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags
     - ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
@@ -252,8 +251,8 @@ services:
           echo "   https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#before-you-begin"
           echo
         fi
-        mkdir -p /sources/logs /sources/dags /sources/plugins
-        chown -R "${AIRFLOW_UID}:0" /sources/{logs,dags,plugins}
+        mkdir -p /opts/airflow/{logs,dags,plugins,config}
+        chown -R "${AIRFLOW_UID}:0" /opts/airflow/{logs,dags,plugins,config}
         exec /entrypoint airflow version
     # yamllint enable rule:line-length
     environment:
@@ -264,8 +263,6 @@ services:
       _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
       _PIP_ADDITIONAL_REQUIREMENTS: ''
     user: "0:0"
-    volumes:
-      - ${AIRFLOW_PROJ_DIR:-.}:/sources
 
   airflow-cli:
     <<: *airflow-common

airflow-core/docs/howto/docker-compose/index.rst

@@ -125,8 +125,8 @@ Setting the right Airflow user
 ------------------------------
 
 On **Linux**, the quick-start needs to know your host user id and needs to have group id set to ``0``.
-Otherwise the files created in ``dags``, ``logs`` and ``plugins`` will be created with ``root`` user ownership.
-You have to make sure to configure them for the docker-compose:
+Otherwise the files created in ``dags``, ``logs``, ``config`` and ``plugins`` will be created with
+``root`` user ownership. You have to make sure to configure them for the docker-compose:
 
 .. code-block:: bash
 
@@ -143,6 +143,17 @@ safely ignore it. You can also manually create an ``.env`` file in the same fold
 
   AIRFLOW_UID=50000
 
+Initialize airflow.cfg (Optional)
+---------------------------------
+
+If you want to initialize ``airflow.cfg`` with default values before launching the airflow service, run.
+
+.. code-block:: bash
+
+   docker compose run airflow-cli airflow config list
+
+This will seed ``airflow.cfg`` with default values in ``config`` folder.
+
 Initialize the database
 -----------------------
 
@@ -346,12 +357,9 @@ Special case - Adding a custom config file
 
 If you have a custom config file and wish to use it in your Airflow instance, you need to perform the following steps:
 
-1) Remove comment from the ``AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg'`` line
-   in the ``docker-compose.yaml`` file.
-
-2) Place your custom ``airflow.cfg`` file in the local config folder.
+1) Replace the auto-generated ``airflow.cfg`` file in the local config folder with your custom config file.
 
-3) If your config file has a different name than ``airflow.cfg``, adjust the filename in
+2) If your config file has a different name than ``airflow.cfg``, adjust the filename in
    ``AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg'``
 
 Networking