[Doc] Document about using custom ingress controller (#6354)

* [Doc] Document about using custom ingress controller

Signed-off-by: Aylei <[email protected]>

* Fix link

Signed-off-by: Aylei <[email protected]>

---------

Signed-off-by: Aylei <[email protected]>

Author: aylei

charts/skypilot/templates/ingress.yaml

@@ -11,7 +11,7 @@ metadata:
   namespace: {{ .Release.Namespace }}
   annotations:
     {{- if not $useNewIngressClass }}
-    kubernetes.io/ingress.class: nginx
+    kubernetes.io/ingress.class: {{ .Values.ingress.ingressClassName }}
     {{- end }}
     {{- if index .Values.ingress "oauth2-proxy" "enabled" }}
     # OAuth2 Proxy authentication for browser-based access
@@ -44,6 +44,9 @@ metadata:
       # Set X-Skypilot-Auth-Mode header to the value from the auth request.
       proxy_set_header X-Skypilot-Auth-Mode $x_skypilot_auth_mode;
       {{- end }}
+    {{- if .Values.ingress.annotations }}
+    {{- toYaml .Values.ingress.annotations | nindent 4 }}
+    {{- end }}
 
 spec:
   {{- if $useNewIngressClass }}

charts/skypilot/values.yaml

@@ -164,6 +164,8 @@ ingress:
   path: '/'
   # Ingress class name (for newer Kubernetes versions)
   ingressClassName: nginx
+  # Custom annotations for the ingress
+  annotations: null
 
   # Whether enable an additional NodePort service for the ingress controller
   # Default to null for backward compatibility:

docs/source/reference/api-server/api-server-admin-deploy.rst

@@ -66,7 +66,7 @@ Install the SkyPilot Helm chart with the following command:
 
     For more details on the available configuration options, refer to :ref:`SkyPilot API Server Helm Chart Values <helm-values-spec>`.
 
-The above command will install a SkyPilot API server and ingress-nginx controller in the given namespace, which by default conflicts with other installations. To deploy multiple API servers, refer to :ref:`Reusing ingress-nginx controller for API server <sky-api-server-helm-multiple-deploy>`.
+The above command will install a SkyPilot API server and ingress-nginx controller in the given namespace, which by default conflicts with other installations. To deploy multiple API servers, refer to :ref:`Reusing ingress-nginx controller for API server <sky-api-server-helm-multiple-deploy>`. To use a different ingress controller, refer to :ref:`sky-api-server-custom-ingress`
 
 .. tip::
 
@@ -978,6 +978,26 @@ The same approach also applies when you have a ingress-nginx controller deployed
 
 It is a good practice to specify a unique :ref:`ingress.path <helm-values-ingress-path>` too in this case, to avoid conflicts with other backends hosted on the same ingress controller.
 
+.. _sky-api-server-custom-ingress:
+
+Use custom ingress controller
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+By default, the SkyPilot helm chart will deploy a new ingress-nginx controller when installing the API server. However, you can use a custom ingress controller by disabling the creation of nginx ingress controller and setting :ref:`ingress.ingressClassName <helm-values-ingress-ingressclassname>` to the ingress class name of your controller. In addition, most of the ingress controllers support customizing behavior by setting annotations on the ingress resource. You can set :ref:`ingress.annotations <helm-values-ingress-annotations>` in the helm values to pass annotations to the ingress resource. Here is an example of using a custom ingress controller:
+
+.. code-block:: bash
+
+    helm upgrade --install $RELEASE_NAME skypilot/skypilot-nightly --devel \
+        --namespace $NAMESPACE \
+        --reuse-values \
+        --set ingress-nginx.enabled=false \
+        --set ingress.ingressClassName=custom-ingress-class \
+        --set ingress.annotations.custom-ingress-annotation=custom-ingress-annotation-value
+
+.. note::
+
+    :ref:`Basic auth on ingress <helm-values-ingress-authcredentials>` and :ref:`OAuth2 <helm-values-ingress-oauth2-proxy>` are only supported when using ingress-nginx controller. For other ingress controllers, you can refer to :ref:`deploy-api-server-basic-auth` to setup authentication on the API server.
+
 
 .. _sky-api-server-cloud-deploy:
 

docs/source/reference/api-server/api-server-upgrade.rst

@@ -57,7 +57,7 @@ Optionally, you can watch the upgrade progress with:
 
 .. code-block:: console
 
-    $ kubectl get pod -l app=${RELEASE_NAME}-api --watch
+    $ kubectl get pod --namespace $NAMESPACE -l app=${RELEASE_NAME}-api --watch
     NAME                                       READY   STATUS            RESTARTS   AGE
     skypilot-demo-api-server-cf4896bdf-62c96   0/1     Init:0/2          0          7s
     skypilot-demo-api-server-cf4896bdf-62c96   0/1     Init:1/2          0          24s

docs/source/reference/api-server/helm-values-spec.rst

@@ -94,6 +94,7 @@ Below is the available helm value keys and the default value of each key:
     :ref:`nodePortEnabled <helm-values-ingress-nodePortEnabled>`: null
     :ref:`httpNodePort <helm-values-ingress-httpNodePort>`: 30050
     :ref:`httpsNodePort <helm-values-ingress-httpsNodePort>`: 30051
+    :ref:`annotations <helm-values-ingress-annotations>`: null
     :ref:`oauth2-proxy <helm-values-ingress-oauth2-proxy>`:
       :ref:`enabled <helm-values-ingress-oauth2-proxy-enabled>`: false
       # Required when enabled:
@@ -912,6 +913,21 @@ Default: ``30051``
   ingress:
     httpsNodePort: 30051
 
+.. _helm-values-ingress-annotations:
+
+``ingress.annotations``
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Custom annotations for the ingress controller.
+
+Default: ``null``
+
+.. code-block:: yaml
+
+  ingress:
+    annotations:
+      my-annotation: "my-value"
+
 .. _helm-values-ingress-oauth2-proxy:
 
 ``ingress.oauth2-proxy``
@@ -1239,15 +1255,15 @@ Default: see the yaml below.
       - apiGroups: ["apps"]
         resources: ["deployments", "deployments/status"]
         verbs: ["*"]
-              - apiGroups: [ "" ]
-          resources: [ "configmaps" ]
-          verbs: [ "get", "patch" ]
-        - apiGroups: ["apps"]
-          resources: ["deployments", "deployments/status"]
-          verbs: ["*"]
-        - apiGroups: [""]
-          resources: ["persistentvolumeclaims"]
-          verbs: ["*"]
+      - apiGroups: [ "" ]
+        resources: [ "configmaps" ]
+        verbs: [ "get", "patch" ]
+      - apiGroups: ["apps"]
+        resources: ["deployments", "deployments/status"]
+        verbs: ["*"]
+      - apiGroups: [""]
+        resources: ["persistentvolumeclaims"]
+        verbs: ["*"]
 
 .. _helm-values-rbac-clusterRules: