Fast slow store directions

[chrisstaite]

Description

There are multiple use cases where we don't want a fast-slow store to persist to one of the stores in some direction. For example, worker nodes do not want to store build results on the local filesystem, just with the upstream CAS. Another case would be the re-use of prod action cache in a dev environment, but not vice-versa.

This PR introduces options to the fast-slow store which default to the existing behaviour, but allows customisation of each side of the fast slow store to either persist in the case or get operations, put operations or to make them read only.

Fixes #1577

Type of change

Please delete options that aren't relevant.

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How Has This Been Tested?

Added new tests

Checklist

• Updated documentation if needed
• Tests added/amended
• bazel test //... passes locally
• PR is contained in a single commit, using git amend see some docs

---

This change is 

[CLAassistant]

All committers have signed the CLA.

[MarcusSorealheis]

Hi @chrisstaite, not sure if I screwed anything up here yet, but I did just fix the merge conflicts to make it easier for @aaronmondal to review the PR and get it merged. It's a powerful new feature. I did this from the web browser so I may have screwed something up.

It's clear that the PR needs:

1. Rust format
2. and probably a sanity check against all the new linting that was added.

There are a couple things missing in the PR that I'd like to see that are probably separate from what Aaron's going to ask about:

One, an example of how to use it. A short guide would be awesome. #1577 makes it obvious why someone would want this feature (e.g., flow from Prod to Dev but not vice versa), but for other people to use it, would be great for us to document it. Aaron and I can also help there if time-constraints are an issue. We are simply trying to graduate Nativelink from the era of if users want to use it they need to look at the code, to opening up to more users.

This is really a fantastic feature. Thanks for the contribution!

[MarcusSorealheis]

i'll do the whole amend if my changes fixed the rust formatting issues and I think it wil.