ZFS running on S3 object storage via NBD on SlateDB: Faster with Less

[john-parton]

SlateDB (See https://slatedb.io/ and https://github.com/slatedb/slatedb) allows you to use object storage such as S3 (or Google Cloud Storage, Azure Blob Storage) in a way that's a lot more like a traditional block device.

I saw another person created a project they called "ZeroFS". It turns out that it uses SlateDB under the hood to provide a file abstraction. There's lots of good ideas in there, such as automatically encrypting and compressing data, however, the fundamental idea is to build a POSIX compatible file API on top of SlateDB and then create a block storage abstraction of the file API. In furtherance of that, there is a lot of code to handle caching and other code paths that don't directly support the "run ZFS on object storage"

I was really curious and wondered: "What if you were to just directly map blocks to object storage using SlateDB and then let ZFS handle all of the details of compression, caching, and other gnarly details?"

The results are significantly better performance numbers with less caching. I was still getting more than twice the throughput on some tests designed to emulate real world usage. The internal WAL and read caches for SlateDB can even be disabled, with no measurable performance hit.

My project is here: https://github.com/john-parton/slatedb-nbd