Feature Request: Optimize image export/import, add features for preloaded image volumes

[kornelbicskei]

Problems

Current Clabernetes architecture is unique in using nested containerization (kubernetes runtime containerd starting up launcher pods with dockerd inside them launching NOS containers), this brings in challenges around image pulling, caching, exporting and importing between containerd <> dockerd, especially at scale with ephemeral worker nodes.

Bootup time

The container images for network OS (NOS) that are launched as part of Clabernetes are quite large (e.g., EOS: 870 MB, junos-vsrx: 5898 MB, Clab launcher: 260 MB). For setups with for example 54 devices (54 pods), this results in significant image pulling and caching demands. Even with existing image-caching logic, it takes ~30 sec for launcher image download and 80-120 sec for NOS image download.

Current image caching logic

• Kubelet (containerd) pulls launcher images per EKS worker node with standard containerd caching.
• Containerlab pods (one per NOS) start nested dockerd and initiate ImagePullThrough logic.
• ImagePullThrough logic submits a CRI request (via Kubernetes API) to pull NOS image onto the node (containerd). This is optimized and only happens once per node.
• Each launcher waits for NOS image presence in host dockerd
• All 54 launcher pods use the mounted containerd socket to execute image export (nerdctl save), creating 54 .tar image files (one per pod)
  This is export and file creation cycle is unnecessary and could be reduced from 54 to 1 (or once per worker node)
• Each pod then loads the .tar file into dockerd

Elastic Karpenter nodes

Karpenter-based node scheduling creates just-in-time nodes with empty image caches, exacerbating the above inefficiencies.

Overloading host containerd

Before we added pod resource limits, exporting images 54 times overloaded host containerd, causing kubelet and node unresponsiveness.

Proposed Solution: Shared Host Volume for Image Export/Import

• Use a shared, read-only host volume mount for image .tar export after the once-per-node ImagePull.
• Modify launcher dockerd logic to check the shared folder before performing an export/import. Thus reducing image exports from 54 to 1 (once per node)
• All pods will perform docker image load from the shared volume, greatly reducing redundant operations.
• Further, check the local dockerd image-store (VFS) before loading from .tar; This is important as it sets us up for mounting volumes that already have preloaded image stores.

Benefits

• Improved scalability for large-scale deployments on elastic worker nodes.
• Reduced bootup times (from minutes to seconds) and resource usage.
• Minimizes overload of host containerd and reduces risk of node unresponsiveness.
• Bottlerocket OS already supports using preloaded volumes as you can boot the nodes from a snapshotted EBS data volume that already contains image downloads in the containerd image store, we should mimic this logic with dockerd in the launchers. Preloading the shared volume via a CI pipeline could eliminate image downloads/imports completely for both containerd (launcher image) and dockerd (NOS images), theoretically bringing container start times to 5-10 seconds at any scale.

Next Steps/Questions

• Are there concerns with concurrent access to the shared host volume?
• What is the best approach for preloading images in CI?
• Are there security implications for shared, read-only host volumes in this scenario?

---

Happy to discuss details or provide clarifications. Looking forward to feedback from maintainers and the community!

[carlmontanari]

copying from the discord thread for consistency/searchability -- thread here for those so inclined to be on discord

yeah i think thils is all good ideas but i think we gotta simplify due to all the constraints you have been experiencing haha. (shitty problem to solve for us right??)

i think the simplest path is a totally optional launcher volumes thing like we chatted about last week. then you can put w/e you want into this (obviously in this case the image(s)). and a simple script hook to give you a place to copy from the volume to the docker daemon.

the downside of this is of course that it would expect you to handle creating the volume w/ the image.... which is not ideal. but this would be the easiest/fastest path i can think of

if we must have the image pulled and then put in the volume... we could potentially have a thing similar to puller -- we can call it the copier or whatever -- that runs after puller but before the launchers are deployed. the copier would obviously... copy lol. so maybe this + the script hook/volume mount would work?