Unable to Open Files from Cloud Storage

[zhui803]

Describe the bug (mandatory)

Hi! I am not sure if this is a bug or an unsupported feature but I have trouble opening the files in the bucket of my Google Cloud Storage with fitz.open().
I have written a program to loop through the pdf files of my local directory and count the appearances of a list of keywords and it works for local files.
However, I would love to have this feature support the traversal of files from the Cloud Storage given that my computer does not have enough room to download all the pdf files locally. Is there a way to accomplish this goal? Sincerely appreciate anyone's help and suggestions.

To Reproduce (mandatory)

import fitz
import csv
import glob
import os
from google.cloud import storage
from dotenv import load_dotenv
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'ServiceKey_GoogleCloud2.json'
# create storage client instance
storage_client = storage.Client()

def traversal(flattened, target):

        for blob in storage_client.list_blobs('educationpapers', prefix='education'):
            print(blob.name)
            document = fitz.open(str(blob.name))

Screenshots (optional)

Below is a screenshot of the error message I get.
The terminal successfully prints the name of the pdf in my bucket: the "education-10-00313.pdf" from print(blob.name) so I am not sure if the problems occur from fitz.open().

My configuration

• Operating system: macOS Monterey Version 12.6 (I am not sure if this is the correct information about my OS)
• Python version: 3.10.8
• PyMuPDF version: PyMuPDF==1.21.1 (I installed it from PyPI)

[JorjMcKie]

This is a typical "Discussions" item, so let me first transfer it to there.

[JorjMcKie]

When reading from (or writing to) cloud storage, you typically want to avoid using the disk storage of your local machine - as you wrote, too.
Ineraction therefore works using just the memory of your computer as intermediate storage. PyMuPDF supports this by opening documents from memory (and also writing to memory).

The blobs in your loop for blob in storage_client.list_blobs(...): should have a method to access the blob content as binary data - something like blob.download_as_string() or blob.download_as_bytes().
Use this binary data instead using a filename to open the fitz.Document:
doc = fitz.open("pdf", blob.download_as_bytes()).

There is a collection of script for cloud access here which may be useful too.

[zhui803]

Dear Sir,
Thank you so much for your information and timely help. They are really helpful and have successfully solved my questions. Have a nice day!

[JorjMcKie]

Just call me Jorj 😉!
I am glad this helped you!

[Aakif-cloud]

@JorjMcKie i need some help can you
i just want to fitz.open() a pdf file present on gcs bucket

[JorjMcKie]

@Aakif-cloud
The key point is to download the file to local memory (bytes). Then do doc = fitz.open("pdf", pdfdata), where pdfdata is that area.
This script should explain how this works.

[Aakif-cloud]

already did that , thanks need help with one thing

…

On Wed, Jul 26, 2023 at 1:09 PM Jorj X. McKie ***@***.***> wrote: @Aakif-cloud <https://github.com/Aakif-cloud> The key point is to download the file to local memory (bytes). Then do doc = fitz.open("pdf", pdfdata), where pdfdata is that area. This <https://github.com/pymupdf/PyMuPDF-Utilities/blob/master/cloud-interactions/from-google.py> script should explain how this works. — Reply to this email directly, view it on GitHub <#2241 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5C6JBDZCOFHPPYODXIULRTXSDCT3ANCNFSM6AAAAAAVAOYNSA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- DISCLAIMER:The entire content of this email message is confidential. This also applies to any files attached to it. This email is intended for an individual or entity to whom they are addressed. In case you are not the addressee of this email, and you have received it in error, immediately contact the system manager. The information in this email is very sensitive, and it is intended for the specific addressee. This email should not be disseminated, distributed or copied. If you have received this email and it was not for you, make sure to immediately notify the sender by email and afterwards delete this email from your system. Disclosing, copying, distributing, or taking any action in reliance to the email content is strictly prohibited. Consider yourself notified.

[JorjMcKie]

Ok, please tell me.

[Aakif-cloud]

I am deploying sometime on cloud gcs using cloud run In the script I need to pass a folder path which contains all the images extracted from a pdf Like this file_path =“” How can I ?? The script running fine on local as I can make a folder can u help

…

On Wed, 26 Jul 2023 at 6:49 PM, Jorj X. McKie ***@***.***> wrote: Ok, please tell me. — Reply to this email directly, view it on GitHub <#2241 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5C6JBBFCD6THSYQFI766C3XSEKNDANCNFSM6AAAAAAVAOYNSA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- DISCLAIMER:The entire content of this email message is confidential. This also applies to any files attached to it. This email is intended for an individual or entity to whom they are addressed. In case you are not the addressee of this email, and you have received it in error, immediately contact the system manager. The information in this email is very sensitive, and it is intended for the specific addressee. This email should not be disseminated, distributed or copied. If you have received this email and it was not for you, make sure to immediately notify the sender by email and afterwards delete this email from your system. Disclosing, copying, distributing, or taking any action in reliance to the email content is strictly prohibited. Consider yourself notified.

[JorjMcKie]

Ah, I see now. This problem is not related to PyMuPDF.
You must find a way to create a folder inside your cloud storage and put those images into it. Follow the instructions for creating folders inside Google Cloud Storage.

[Aakif-cloud]

Ok, I will try that. Thanks

…

On Wed, 26 Jul 2023 at 10:07 PM, Jorj X. McKie ***@***.***> wrote: Ah, I see now. This problem is not related to PyMuPDF. You must find a way to create a folder *inside your cloud storage* and put those images into it. Follow the instructions for creating folders inside Google Cloud Storage. — Reply to this email directly, view it on GitHub <#2241 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5C6JBF4WR5MGZQCD55RFZLXSFBUJANCNFSM6AAAAAAVAOYNSA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- DISCLAIMER:The entire content of this email message is confidential. This also applies to any files attached to it. This email is intended for an individual or entity to whom they are addressed. In case you are not the addressee of this email, and you have received it in error, immediately contact the system manager. The information in this email is very sensitive, and it is intended for the specific addressee. This email should not be disseminated, distributed or copied. If you have received this email and it was not for you, make sure to immediately notify the sender by email and afterwards delete this email from your system. Disclosing, copying, distributing, or taking any action in reliance to the email content is strictly prohibited. Consider yourself notified.

[Aakif-cloud]

Thanks for being that much responsive On Wed, 26 Jul 2023 at 10:41 PM, Aakif Khan ***@***.***> wrote:

…

Ok, I will try that. Thanks On Wed, 26 Jul 2023 at 10:07 PM, Jorj X. McKie ***@***.***> wrote: > Ah, I see now. This problem is not related to PyMuPDF. > You must find a way to create a folder *inside your cloud storage* and > put those images into it. Follow the instructions for creating folders > inside Google Cloud Storage. > > — > Reply to this email directly, view it on GitHub > <#2241 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/A5C6JBF4WR5MGZQCD55RFZLXSFBUJANCNFSM6AAAAAAVAOYNSA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

-- DISCLAIMER:The entire content of this email message is confidential. This also applies to any files attached to it. This email is intended for an individual or entity to whom they are addressed. In case you are not the addressee of this email, and you have received it in error, immediately contact the system manager. The information in this email is very sensitive, and it is intended for the specific addressee. This email should not be disseminated, distributed or copied. If you have received this email and it was not for you, make sure to immediately notify the sender by email and afterwards delete this email from your system. Disclosing, copying, distributing, or taking any action in reliance to the email content is strictly prohibited. Consider yourself notified.

[JorjMcKie]

You are welcome - and sorry I have no better answer.

[yogeshvjadhav]

How to upload opened gcs document using fitz into the google cloud storage?

[JorjMcKie]

First You must determine how to upload binary data to your cloud storage.
The crate a PDF document's binary image via doc.tobytes() and upload the resulting bytes object.

Not that the .tobytes() method works like .save(), which means you can use all parameters like garbage, deflate etc.

[yogeshvjadhav]

Thanks Jorj. Its working,
Could you please share documentation link where i can explore more about this?

[JorjMcKie]

Here is the documentation of tobytes.
Look in this repo folder to see some examples for cloud interactions.

[yogeshvjadhav]

Thank you very much Jorj