Skip to content

zarrinfo can not access files on s3 storage #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Ajaxels opened this issue Apr 1, 2025 · 5 comments
Open

zarrinfo can not access files on s3 storage #11

Ajaxels opened this issue Apr 1, 2025 · 5 comments

Comments

@Ajaxels
Copy link

Ajaxels commented Apr 1, 2025

Hi,
I am not able to get information about the dataset that is stored on s3 storage:

zarrUrl = 'https://s3.embl.de/i2k-2020/platy-raw.ome.zarr';
info = zarrinfo(zarrUrl);

I can access the dataset using MoBIE Fiji Viewer, but not Matlab.

@abhibaruah
Copy link
Member

Hi @Ajaxels,
Do you know if any credentials are needed to access this S3 bucket?
What region is the S3 bucket located at?

@Ajaxels
Copy link
Author

Ajaxels commented Apr 1, 2025

Nothing is needed, it is in Germany

@jm9176
Copy link
Member

jm9176 commented Apr 22, 2025

I tried out the same workflow with s3:// and http:// syntax, and only s3:// syntax is working fine, but http:// syntax throws "folders do not exist" error

>> zarrinfo('s3://<my_test_bucket>/prt_grp/arr_1')

ans = 

  struct with fields:

                 chunks: [2×1 double]
             compressor: []
    dimension_separator: '.'
                  dtype: '<f8'
             fill_value: -9
                filters: []
                  order: 'C'
                  shape: [2×1 double]
            zarr_format: 2
              node_type: 'array'

@Ajaxels
Copy link
Author

Ajaxels commented Apr 22, 2025

@jm9176 have you been able to open 'https://s3.embl.de/i2k-2020/platy-raw.ome.zarr ?

@jm9176
Copy link
Member

jm9176 commented Apr 23, 2025

@Ajaxels No, I get the following error,

Error using zarrinfo (line 12)
 info = zarrinfo(zarrUrl);
                 ^^^^^^^
Invalid argument at position 1. These folders do not exist: 'https://s3.embl.de/i2k-2020/platy-raw.ome.zarr'.

However, I did try a different s3 bucket, and it worked fine as I mentioned in my comment above, but I encountered the same error when accessing buckets with https:// syntax.

>> zarrcreate('s3://<my_test_bucket>/prt_grp/arr_1',[10 10]);
>> zarrwrite('s3://<my_test_bucket>/prt_grp/arr_1',ones([10 10]));
>> zarrwriteatt('s3://<my_test_bucket>/prt_grp/arr_1','myatttr',10);
>> zarrinfo('s3://<my_test_bucket>/prt_grp/arr_1')

ans = 

  struct with fields:

                 chunks: [2×1 double]
             compressor: []
    dimension_separator: '.'
                  dtype: '<f8'
             fill_value: []
                filters: []
                  order: 'C'
                  shape: [2×1 double]
            zarr_format: 2
              node_type: 'array'
                myatttr: 10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants