Use case: Frequent write/modification of a file

[bgervan]

How to handle it correctly when one file can have many frequent write? Is there a way to wait for other changes before commit the file to s3?

[martindurant]

The S3 API only allows for a single stream of data in each generation of a stored file. Can you describe in more detail, please, how you anticipate data to arrive and be assembled for writing?

[bgervan]

My use case is:

• Websocket server receives update of a file (db), it updates the db file. (There will be thousands of files)
• The db file is cached locally to update instantly
• Send the updates to s3

But if the updates are frequent enough, I wouldn't send the update to s3, but wait for additional updates.

Is the file close which starts the sending to s3 or there are other ways I can commit? Any idea how to approach this with optimizing the egress? If the db is large (even few mb) a small frequent change to it can create high egress fees.

[martindurant]

Hm, it's not really something that fsspec can control. You can have a write-context within a caching filesystem (e.g., simplecache) over s3, but you still need some way to trigger when writes should happen - so it sounds like you may want your own file-watcher to be able to place a condition on when enough changes have accumulated.

Unfortunately, again, s3 does not allow for partial updates of a file, only replacement.