Optimize image loading for Podman machines #26660

Honny1 · 2025-07-18T09:34:11Z

Add support for loading images directly from machine paths to avoid unnecessary file transfers when the image archive is already accessible on the running machine through mounted directories.

Changes include:

New /libpod/local/images/load API endpoint for direct machine loading
Machine detection and path mapping functionality
Fallback in tunnel mode to try optimized loading first
Improved error handling for non-JSON API responses

This optimization significantly speeds up image loading operations when working with remote Podman machines by eliminating redundant file transfers for already-accessible image archives.

Fixes: https://issues.redhat.com/browse/RUN-3249
Fixes: #26321

Does this PR introduce a user-facing change?

New API endpoint for MacOS and Windows that significantly speeds up remote image loading by using direct-path detection to avoid unnecessary file transfers, complete with a seamless fallback mechanism.

openshift-ci · 2025-07-18T09:34:18Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Honny1
Once this PR has been reviewed and has the lgtm label, please assign tomsweeneyredhat for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Luap99 · 2025-07-18T10:04:46Z

pkg/api/handlers/libpod/images.go

@@ -31,6 +32,7 @@ import (
 	"github.com/containers/podman/v5/pkg/domain/infra/abi"
 	domainUtils "github.com/containers/podman/v5/pkg/domain/utils"
 	"github.com/containers/podman/v5/pkg/errorhandling"
+	"github.com/containers/podman/v5/pkg/machine"


we should not import anything machine from here I would say that makes the import structure really weird that the server now all of the sudden imports machine code

Luap99 · 2025-07-18T10:06:38Z

pkg/api/handlers/libpod/images.go

+	localMap := machine.LocalAPIMap{}
+	if err := json.NewDecoder(r.Body).Decode(&localMap); err != nil {
+		utils.Error(w, http.StatusBadRequest, fmt.Errorf("failed to decode request body: %w", err))
+		return
+	}


I don't think this is intuitive, you parse struct where one field is just not used at all. My suggestion add a signle query param path where the client must specify the path how it is on the VM.

Luap99 · 2025-07-18T10:07:09Z

pkg/api/handlers/libpod/images.go

+	}
+
+	cleanPath := filepath.Clean(localMap.RemotePath)
+	switch _, err := os.Stat(cleanPath); {


use fileutils.Exists

Luap99 · 2025-07-18T10:08:51Z

pkg/api/handlers/libpod/images.go

+	cleanPath := filepath.Clean(localMap.RemotePath)
+	switch _, err := os.Stat(cleanPath); {
+	case err == nil:
+	case os.IsNotExist(err):


as mention in that function new code should use errors.Is(err, fs.ErrNotExist)

but is there a reason to check for this file here at all? I would assume imageEngine.Load() call would alos fail with ErrNotExist so we could just handle it there?

imageEngine.Load() attempts to open the file several times to determine the correct image format in the tar file. I would prefer to avoid repeatedly attempting to open the tar file.

Luap99 · 2025-07-18T10:10:03Z

pkg/bindings/errors.go

+	if h.Response.Header.Get("Content-Type") == "application/json" {
+		return handleError(data, &errorhandling.ErrorModel{})
+	}
+	return &errorhandling.ErrorModel{
+		Message:      string(data),
+		ResponseCode: h.Response.StatusCode,
+	}


This is nice! But please split this out into a separate commit with its own explanation as it is mostly unrelated to the machine change

Luap99 · 2025-07-18T10:13:38Z

pkg/bindings/images/images.go


 	imageTypes "github.com/containers/image/v5/types"
 	handlersTypes "github.com/containers/podman/v5/pkg/api/handlers/types"
 	"github.com/containers/podman/v5/pkg/auth"
 	"github.com/containers/podman/v5/pkg/bindings"
 	"github.com/containers/podman/v5/pkg/domain/entities/reports"
 	"github.com/containers/podman/v5/pkg/domain/entities/types"
+	"github.com/containers/podman/v5/pkg/machine"


Here as well I don't think we should import any machine code from bindings. The bindings itself are supported by external users so we should not cause machine specifc imports here.

Luap99 · 2025-07-18T10:17:51Z

pkg/domain/infra/tunnel/images.go

@@ -221,6 +223,22 @@ func (ir *ImageEngine) Inspect(ctx context.Context, namesOrIDs []string, opts en
 }

 func (ir *ImageEngine) Load(ctx context.Context, opts entities.ImageLoadOptions) (*entities.ImageLoadReport, error) {
+	if localMap, ok := shim.CheckPathOnRunningMachine(ir.ClientCtx, opts.Input); ok {


AFAIK machine only compiles on amd64/arm64 and not the other arches so this breaks the build, you will need some stub that always returns false.

Also we should not make calls into machine code at all if the connection is not a machine. We actually have the machine info already so the rereading of connection you do int he machine code is redundant.

If you look into NewImageEngine() there is facts.MachineMode you could use. i.e. you could store it in the ImageEngine struct and then just access it here.

packit-as-a-service · 2025-08-04T15:28:25Z

[NON-BLOCKING] Packit jobs failed. @containers/packit-build please check. Everyone else, feel free to ignore.

packit-as-a-service · 2025-08-04T15:28:25Z

[NON-BLOCKING] Packit jobs failed. @containers/packit-build please check. Everyone else, feel free to ignore.

Honny1 · 2025-08-05T10:52:27Z

/packit retest-failed

Honny1 · 2025-08-05T12:07:28Z

@Luap99 PTAL, I have addressed your review.

I don't think the failed machine-wsl test is related to these changes. I think something went wrong with WSL.

Luap99

This is not going to work on windows with WSL since that doesn't do mounts the normal way, all WSL machines have /mnt/c, etc... mounted by default.

There is specgen.ConvertWinMountPath() which must be used to translate the windows paths AFAIK.

Luap99 · 2025-08-07T09:51:31Z

pkg/bindings/images/images.go

@@ -139,6 +140,25 @@ func Load(ctx context.Context, r io.Reader) (*types.ImageLoadReport, error) {
 	return &report, response.Process(&report)
 }

+func LoadLocal(ctx context.Context, m *local_utils.LocalAPIMap) (*types.ImageLoadReport, error) {


local_utils is an internal package which means it will be impossible for any external bindings user to access LocalAPIMap type AFAICT. That is likely not wanted? I would say just accept the path as string argument here.

Luap99 · 2025-08-07T09:52:51Z

pkg/domain/infra/tunnel/images.go

+		errModel, ok := err.(*errorhandling.ErrorModel)
+		if !ok {
+			return nil, err
+		}


use errors.As() instead. Code might wrap the struct in which case your error type check here fails.

Luap99 · 2025-08-07T10:04:47Z

pkg/machine/shim/host.go

+// InspectRunningMachine finds and returns information about the currently running machine.
+// It searches across all VM providers to find a machine in the running state.
+// Returns nil if no running machine is found.
+func InspectRunningMachine(vmstubbers []vmconfigs.VMProvider, _ machine.ListOptions) (*machine.InternalInspectInfo, error) {


I don't think this is right or future proof, wsl machines support multiple running machines at the same time and we might want to have that for other providers as well.

I think the checking the ssh connection info like done in cmd/podman/compose_machine.go is likely the better thing to match the machine vs connection. the bindings client.URI should show the URL

Honny1 · 2025-08-11T16:30:19Z

Benchmark Results

The following benchmarks measure the execution time of the podman load command.

Image: docker.io/tensorflow/tensorflow:latest-gpu
Command: podman load -i tensorflow.tar

Note: All measurements were taken on development machines with background processes running. Performance may be influenced.

MacOS

Chip: Apple M3 Pro
RAM: 36 GB
SSD: 1 TB Apple SSD

AppleHV (Apple Hypervisor)

Before: 74.96 s
After: 25.17 s (66% improvement)

Libkrun

Before: 84.07 s
After: 37.58 s (55% improvement)

Windows

CPU: Intel Core i7-8665U
RAM: 32 GB
SSD: 500 GB Samsung 970 Evo Plus

Hyper-V

Before: 700.74 s
After: 154.58 s (78% improvement)

WSL (16 GB RAM Limit)

Before: 109.63 s
After: 140.33 s

WSL (2 GB RAM Limit)

Before: 147.77 s
After: 142.51 s

Analysis of WSL Results

The Podman machine's performance under WSL is dictated by the global RAM limit of the WSL environment, rather than the specific memory setting configured for the Podman machine. This behavior may be a bug and requires further investigation.

With a 16 GB limit, podman load can copy the large tarball into a RAM-based temporary file system (tmpfs). This explains both the high memory usage and the exceptionally fast "Before" time. The speed advantage is due to two factors: leveraging high-speed RAM over the slower SSD and bypassing the performance overhead of the Windows-to-Linux filesystem translation layer.

When RAM is restricted to 2 GB, the system cannot effectively use tmpfs in RAM for an operation of this size and falls back to SSD. As a result, performance becomes bound by SSD speed, leading to more comparable "Before" and "After" times. The small difference in these results is within the expected margin of error for benchmarks conducted in a non-dedicated development environment.

Honny1 · 2025-08-11T16:33:17Z

@Luap99 I've addressed your comments and fixed the issue with Windows path translation. This is ready for another review.

Honny1 · 2025-08-11T17:17:58Z

/packit retest-failed

Luap99 · 2025-08-12T10:23:35Z

bypassing the performance overhead of the Windows-to-Linux filesystem translation layer.

I think that is the main point, what happens when we copy we read on the windows side send via network, extract to the wsl fs (ext4 I think?) which then has normal fs speed. Now with this change we use the 9p proxy, and I guess the windows 9p mount that it uses has just terrible performance if we can outperform it with the old way. That seems quite concerning for WSL but outside our control.
https://stackoverflow.com/questions/68972448/why-is-wsl-extremely-slow-when-compared-with-native-windows-npm-yarn-processing

Luap99 · 2025-08-12T10:27:08Z

internal/local_utils/local_utils.go

+	dirs, err := env.GetMachineDirs(machineProvider.VMType())
+	if err != nil {
+		return nil, err
+	}
+
+	machineList, err := vmconfigs.LoadMachinesInDir(dirs)
+	if err != nil {
+		return nil, fmt.Errorf("listing machines: %w", err)
+	}
+
+	// Now we know that the connection points to a machine and we
+	// can find the machine by looking for the one with the
+	// matching port.
+	connectionPort, err := strconv.Atoi(parsedConnection.Port())
+	if err != nil {
+		return nil, fmt.Errorf("parsing connection port: %w", err)
+	}
+	for _, mc := range machineList {
+		if connectionPort != mc.SSH.Port {
+			continue
+		}
+
+		state, err := machineProvider.State(mc, false)
+		if err != nil {
+			return nil, err
+		}
+
+		if state != define.Running {
+			return nil, fmt.Errorf("machine %s is not running but in state %s", mc.Name, state)
+		}


This seems to duplicate code from the compose code, I suggest you split this into a function here and then make the compose code use it.

I duplicated the code.

Check Content-Type header before unmarshaling errors to avoid unnecessary JSON parsing overhead for plain text responses. Signed-off-by: Jan Rodák <[email protected]>

Add support for loading images directly from machine paths to avoid unnecessary file transfers when the image archive is already accessible on the running machine through mounted directories. Changes include: - New /libpod/local/images/load API endpoint for direct machine loading - Machine detection and path mapping functionality - Fallback in tunnel mode to try optimized loading first This optimization significantly speeds up image loading operations when working with remote Podman machines by eliminating redundant file transfers for already-accessible image archives. Fixes: https://issues.redhat.com/browse/RUN-3249 Fixes: containers#26321 Signed-off-by: Jan Rodák <[email protected]>

openshift-ci bot added release-note do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jul 18, 2025

github-actions bot added machine kind/api-change Change to remote API; merits scrutiny labels Jul 18, 2025

Honny1 force-pushed the speed-up-load branch from 038c711 to 045806b Compare July 18, 2025 09:46

Luap99 reviewed Jul 18, 2025

View reviewed changes

Honny1 force-pushed the speed-up-load branch 2 times, most recently from 5b32a32 to 49668ff Compare August 4, 2025 15:28

Honny1 force-pushed the speed-up-load branch from 49668ff to faab050 Compare August 4, 2025 16:00

Honny1 marked this pull request as ready for review August 5, 2025 12:03

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 5, 2025

Honny1 requested a review from Luap99 August 5, 2025 12:50

Honny1 force-pushed the speed-up-load branch from faab050 to c4498e2 Compare August 7, 2025 09:13

Luap99 reviewed Aug 7, 2025

View reviewed changes

Honny1 force-pushed the speed-up-load branch 11 times, most recently from 9738a3c to 28e0a9c Compare August 8, 2025 13:07

Honny1 force-pushed the speed-up-load branch 8 times, most recently from 92b20e8 to c104e6f Compare August 11, 2025 15:34

Honny1 requested a review from Luap99 August 11, 2025 17:50

Luap99 reviewed Aug 12, 2025

View reviewed changes

Honny1 force-pushed the speed-up-load branch from c104e6f to 08d3c3c Compare August 12, 2025 13:12

Skip JSON parsing for non-JSON error responses

2266173

Check Content-Type header before unmarshaling errors to avoid unnecessary JSON parsing overhead for plain text responses. Signed-off-by: Jan Rodák <[email protected]>

Honny1 force-pushed the speed-up-load branch from 08d3c3c to 414d784 Compare August 12, 2025 13:28

Honny1 force-pushed the speed-up-load branch from 414d784 to 1659b7a Compare August 12, 2025 14:02

Honny1 requested a review from Luap99 August 12, 2025 16:50

Optimize image loading for Podman machines #26660

Are you sure you want to change the base?

Optimize image loading for Podman machines #26660

Uh oh!

Conversation

Honny1 commented Jul 18, 2025

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci bot commented Jul 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

packit-as-a-service bot commented Aug 4, 2025

Uh oh!

packit-as-a-service bot commented Aug 4, 2025

Uh oh!

Honny1 commented Aug 5, 2025

Uh oh!

Honny1 commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Luap99 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Honny1 commented Aug 11, 2025

Benchmark Results

MacOS

AppleHV (Apple Hypervisor)

Libkrun

Windows

Hyper-V

WSL (16 GB RAM Limit)

WSL (2 GB RAM Limit)

Uh oh!

Honny1 commented Aug 11, 2025

Uh oh!

Honny1 commented Aug 11, 2025

Uh oh!

Luap99 commented Aug 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Honny1 commented Aug 5, 2025 •

edited

Loading