Skip to content

Unpacking huge file contents from zip/tar archives and system memory use #202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jonasbardino opened this issue Feb 19, 2025 · 0 comments
Labels
bug Something isn't working enhancement New feature or request wishlist Tasks or features of lowest priority

Comments

@jonasbardino
Copy link
Contributor

The helper function to unpack zip/tar archives makes an unnecessary copy of the individual archive file contents to memory in
https://github.com/ucphhpc/migrid-sync/blob/edge/mig/shared/archives.py#L132
and therefore may hit an OOM during unpack if any of the files exceed the available system memory in size.

The helper takes that step-wise approach to ease error and out of bounds detection, but should be optimized to stream the reading and writing directly to the destination file without reading the complete file into memory first.

For the time being users will have to upload such huge files directly on SFTP or similar as a workaround.

@jonasbardino jonasbardino added bug Something isn't working enhancement New feature or request wishlist Tasks or features of lowest priority labels Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request wishlist Tasks or features of lowest priority
Projects
None yet
Development

No branches or pull requests

1 participant