New ScyllaDB key space partitioning

[ndr-ds]

Motivation

ScyllaDB hashes each partition key into a token. Each token range gets assigned to a shard. Each shard is pinned to a specific CPU by default. This means that having very few partitions is bad for ScyllaDB's performance.
Right now in the current scheme of things we have just one partition key, which is the root_key. For views, which is our mutable data in the DB, root_key will be set, and we'll be in "exclusive mode". For everything else (Certificates, ConfirmedBlocks, Blobs, etc), everything is on the same partition. For example when we're doing 1M TPS, we'll have several thousands of blocks being created every second, as well as certificates. This current scheme won't scale for those numbers.

Proposal

New schema proposal: instead of having a root_key, we'll have a partition_key instead. This partition_key will have two modes: exclusive mode (mutable data) and non exclusive mode (immutable data). The former will work exactly how the previous root_key schema worked.

For exclusive mode, the partition_key's first byte will be 0, indicating the mode. The rest of the bytes will be the root_key that was provided when calling open_exclusive. Everything else should work as it previously did.

For non exclusive mode, the partition_key's first byte will be 1. The rest of the partition key will be a prefix of the key of a predetermined length (through a configuration parameter). This mode we'll have some reservations:

• Batches with queries with keys across different partitions are not allowed (this also applies for exclusive mode by design)
• Batches with prefix deletes are not allowed (would require a full table scan in cases where the prefix is smaller than the partition_key's prefix size)
• On read_multi_values_internal and contains_keys_internal we currently group the keys by partition_key and execute one query per partition_key in parallel. This is done to keep the queries token aware across partitions
• On find_keys_by_prefix_internal and find_key_values_by_prefix_internal, if the prefix is smaller than the partition_key's prefix size, we'll do a full table scan. This happens infrequently enough that we're willing to take the performance hit.

Test Plan

CI + will benchmark this to check performance. Some tests had to be altered as they didn't respect the invariant that if we're using a root_key, we should be in exclusive mode.

Follow ups

There are several follow ups here:

• Bundle Certificate and ConfirmedBlock BaseKeys closer together, as well as Blob and BlobState
• Do the same thing that we did for find_keys_by_prefix_internal for prefix deletes, and take the perf hit of the full table scan, as these seem to be currently infrequent
• Enforce for Views (maybe on load) that they must always be on exclusive mode
• We have a lot of places in the code where we use batches of size 1. These should not be batches. More specifically WritableKeyValueStore should have write_value and write_multi_values methods, analogous to ReadableKeyValueStore. Batches should be used when atomicity is wanted, or when we want to save network requests to the DB

Release Plan

• Nothing to do / These changes follow the usual release cycle.

[ndr-ds]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Scylla test changes #4198
• New ScyllaDB key space partitioning #4049 👈 (View in Graphite)
• Optimize ScyllaDB's batch writes #4047
• Some code cleanups #4066
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[ma2bd]

I don't think we're going to remove the line you added.

[ndr-ds]

Ah, you’re right, I guess the tests lacking these calls are an orthogonal issue. I’ll remove the comment tomorrow

[ma2bd]

back to your queue as we iterate on the design