Hermetic GHC bindist on Unix

[aherrmann]

Is your feature request related to a problem? Please describe.
The GHC bindist on Unix requires a ./configure && make && make install step before the toolchain can be used. Currently, haskell_register_ghc_bindists executes this installation in a repository rule. This means that this step is not sandboxed and cannot be cached by Bazel (the repository cache is limited to downloads). Additionally, repository rules don't have access to Bazel's toolchain resolution, so we cannot point GHC to the correct CC toolchain during ./configure and instead it will find whichever toolchain is available in the environment.

Describe the solution you'd like
I would like for the ./configure && make && make install step to happen in a regular sandboxed build action. This would make this installation step cacheable and allow us to control which CC toolchain GHC discovers at this step.

Describe alternatives you've considered
Users can avoid these issues today by using a nixpkgs provided GHC instead.

Another approach would be to pull in a system installed GHC from outside Bazel as described in #1320. I think it would be good if rules_haskell supported both a hermetic haskell_register_ghc_bindists as well as haskell_register_ghc_host.

Additional context
One difficulty is that we will need to predict which files this installation step produces exactly, due to the lack of dynamic dependencies in Bazel. This will depend on the platform and GHC version.

One approach would be to let the installation step run in a mode where it produces only metadata: the list of files, package dependencies, etc. (essentially what pkgdb_to_bzl.py determines). This metadata could then be checked in (similar to a lock file) and be used to predict the outputs of the installation step going forward. We could ship the lock files for the Linux and MacOS GHC bindists that are supported by rules_haskell as part of rules_haskell. This way this approach would not change the current API.

Note, the GHC bindist on Windows does not require any ./configure or make steps and is useable immediately after unpacking.

[mboes]

We could solve that by maintaining maps from GHC (version, platform) -> partial file list. Even host that on a server somewhere if not in rules_haskell itself. After all, we're already doing something similar by hardcoding the SHA's of each bindist we know about. Or we could maintain pre-configured and relocatable bindists, and have the bindist rules use that instead.

[aherrmann]

We could solve that by maintaining maps from GHC (version, platform) -> partial file list. Even host that on a server somewhere if not in rules_haskell itself. After all, we're already doing something similar by hardcoding the SHA's of each bindist we know about.

Yes, hosting these in rules_haskell itself is what I had in mind with

approach would be to let the installation step run in a mode where it produces only metadata: the list of files, package dependencies, etc. (essentially what pkgdb_to_bzl.py determines). This metadata could then be checked in (similar to a lock file) and be used to predict the outputs of the installation step going forward. We could ship the lock files for the Linux and MacOS GHC bindists that are supported by rules_haskell as part of rules_haskell.

Hosting them separately is an interesting idea. That would avoid bloating the repo too much. We could amend the bindist mapping with URLs and hashes of these hosted metadata files.

[mboes]

Hosting them separately is an interesting idea. That would avoid bloating the repo too much. We could amend the bindist mapping with URLs and hashes of these hosted metadata files.

The only downside is that it introduces an extra point of failure. But the bindists are downloaded from the network anyways, so requiring another network download for metadata should be ok.

[Profpatsch]

How much bloat are we talking here? A few Mbs should be totally feasible, since the rule users only fetch the tarball, not the complete git history.

…

On Wed, Aug 5, 2020 at 12:00 PM Mathieu Boespflug ***@***.***> wrote: Hosting them separately is an interesting idea. That would avoid bloating the repo too much. We could amend the bindist mapping with URLs and hashes of these hosted metadata files. The only downside is that it introduces an extra point of failure. But the bindists are downloaded from the network anyways, so requiring another network download for metadata should be ok. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1393 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYB5ZXZ7TN47ASROPVCNQTR7EUVNANCNFSM4OYTLAJA> .

[aherrmann]

How much bloat are we talking here? A few Mbs should be totally feasible, since the rule users only fetch the tarball, not the complete git history.

Listing the files of the installed GHC 8.8.2 bindist requires about 500k, adding metadata like package dependencies shouldn't take much more. rules_haskell currently lists bindists for 15 GHC versions on two Unix OSs (we don't need this for Windows). That adds up to ~15M. The rules_haskell repository currently takes up 3M uncompressed, so that would be a pretty large increase in size.

If we gzip the file list we get down to ~40k lets say 50k to be conservative. Then this adds up to 1.5M. That's still a pretty large increase compared to the current size of rules_haskell, but much less bad. We could also consider pruning the list of GHC bindists to further reduce the size, currently it goes back all the way to 7.10.3.

[mboes]

Interesting numbers. Ideally we wouldn't need to list all files. Just know where they are given the version number and the platform. Perhaps this could be achieved with ctx.actions.declare_directory(). The problem with the latter is that it can't overlap with ctx.actions.declare_file() targets, which we need for each of the individual ghc commands we want to call. But maybe those could be symlinks to somewhere inside the declared directory (as noted in @aherrmann's previous comment, Windows isn't necessary anyways).

Another thought that came to mind - if Windows doesn't need ./configure, why does Linux/macOS? We could pre-configure relocatable bindists and host them on GitHub or elsewhere. i.e. bypass the official bindists entirely.

[aherrmann]

Ideally we wouldn't need to list all files. Just know where they are given the version number and the platform. Perhaps this could be achieved with ctx.actions.declare_directory(). The problem with the latter is that it can't overlap with ctx.actions.declare_file() targets, which we need for each of the individual ghc commands we want to call. But maybe those could be symlinks to somewhere inside the declared directory

We also need libraries for CcInfo and .haddock files for HaddockInfo. These can share directories with .hi or .html files respectively. I'm not sure if it's worth the additional complexity.

Another thought that came to mind - if Windows doesn't need ./configure, why does Linux/macOS? We could pre-configure relocatable bindists and host them on GitHub or elsewhere. i.e. bypass the official bindists entirely.

That's a very good question. If it's possible to pre-configure relocatable bindists for Unix then that sounds like a great solution. If this is possible, it may also be worth asking why the upstream GHC bindists aren't configured that way in the first place.

[mboes]

I asked the question on ghc-dev@: https://mail.haskell.org/pipermail/ghc-devs/2020-August/019126.html.

The thread seems to indicate that having upstream ship relocatable bindists with a much simpler configure step that only writes out the settings file is doable. I think we ought to work with upstream to solve the problem at that level. (See the golden rule of software quality.)

[Profpatsch]

I think we ought to work with upstream to solve the problem at that level. (See the golden rule of software quality.)

I love how this blogpost is already being linked on discussions like these, I had the same thought when I read Andreas’ comment above.

[Profpatsch]

Though https://mail.haskell.org/pipermail/ghc-devs/2020-August/019128.html makes me suspect that until Hadrian hits we can gain a lot by shipping our own pre-extracted binary releases for now, just push to an s3 bucket from CI?

[Profpatsch]

Especially since upstream is worrying about super-edge cases like AIX, and afaik none of the users of rules_haskell uses AIX (I didn’t even know this unix variant still existed tbh).