Build fails on Arch Linux: deps/scratch/OpenBLAS.v0.3.10-0.x86_64-linux-gnu-libgfortran5 missing

[fusion809]

Hi,

On Arch Linux, I tried to build this repository, after disabling the Mathematica and Matlab tests by commenting out lines in the Makefile pertaining to them (as neither are installed on my system) and building the latest Julia commit (55a6dab76329b693f0fab372b1a80289bff01a90) with make and settings its directory as JULIAHOME and I received the errors:

gcc -m64 -std=c99 -O0 perf.c -o bin/perf0  -I/path/to/julia/deps/scratch/dsfmt-2.2.3 /path/to/julia/deps/scratch/OpenBLAS.v0.3.9-4.x86_64-linux-gnu-libgfortran5 bb-uninstaller/libopenblas64_.a -L/path/to/julia/usr/lib/ /path/to/julia/usr/lib/libopenlibm.a  -lpthread
gcc: error: /path/to/julia/deps/scratch/OpenBLAS.v0.3.9-4.x86_64-linux-gnu-libgfortran5: No such file or directory
gcc: error: bb-uninstaller/libopenblas64_.a: No such file or directory
make: *** [Makefile:60: bin/perf0] Error 1

Thanks for your time.

[alkeldi]

It says that you are missing gfortran and OpenBLAS.
You can see the details for why you need them in the build documentation.
https://github.com/JuliaLang/julia/blob/master/doc/build/build.md

[fusion809]

They are installed. which gfortran returns /bin/gfortran while pacman -Qi openblas returns:

Name            : openblas
Version         : 0.3.10-1
Description     : An optimized BLAS library based on GotoBLAS2 1.13 BSD
Architecture    : x86_64
URL             : https://www.openblas.net/
Licenses        : BSD
Groups          : None
Provides        : blas=3.8.0
Depends On      : gcc-libs
Optional Deps   : None
Required By     : cblas  julia  lapack  r
Optional For    : python-numpy  r  sagemath
Conflicts With  : blas
Replaces        : None
Installed Size  : 17.79 MiB
Packager        : Felix Yan <[email protected]>
Build Date      : Tue 16 Jun 2020 23:51:41
Install Date    : Fri 10 Jul 2020 16:05:52
Install Reason  : Installed as a dependency for another package
Install Script  : No
Validated By    : Signature

thus clearly showing they're both installed.

[StefanKarpinski]

Have you got anything in your Make.user file? Are you building with system libraries or with defaults?

[fusion809]

Nothing in Make.user, and just using the defaults.

[StefanKarpinski]

To be clear, is this a failure building Julia or running the makefile in this repo?

[fusion809]

This repo. I define export JULIAHOME="/path/to/julia-built-repo" and run make in this repo and the error I reported earlier is what I get. Julia itself builds fine.

[StefanKarpinski]

Julia 1.1 seems to be the last version of Julia for which the makefile logic for hooking up libraries for other languages works. Should ideally be updated to use artifacts. If we had artifactized versions of all of the open source systems then it would make running these benchmarks much easier. Unfortunately, that's a lot of work and getting this working on a recent Julia version isn't a terribly high priority.

[timholy]

There seems to have been a change in how the deps are unpacked. For example I get

$ JULIAHOME=~/src/julia-master/ make
bin/versions.sh >versions.csv
bin/versions.sh: line 10: go: command not found
bin/versions.sh: line 16: node8: command not found
bin/versions.sh: line 26: math: command not found
bin/versions.sh: line 32: octave-cli: command not found
bin/versions.sh: line 38: R: command not found
bin/versions.sh: line 41: rustc: command not found
echo '#include "/home/tim/src/julia-master//deps/scratch/OpenBLAS.v0.3.10+3.x86_64-linux-gnu-libgfortran5 bb-uninstaller/cblas.h"' > perf.h
echo '#include "/home/tim/src/julia-master//deps/scratch/dsfmt-2.2.4/dSFMT.c"' >> perf.h
gcc -m64 -std=c99 -O0 perf.c -o bin/perf0  -I/home/tim/src/julia-master//deps/scratch/dsfmt-2.2.4 /home/tim/src/julia-master//deps/scratch/OpenBLAS.v0.3.10+3.x86_64-linux-gnu-libgfortran5 bb-uninstaller/libopenblas64_.a -L/home/tim/src/julia-master//usr/lib/ /home/tim/src/julia-master//usr/lib/libopenlibm.a  -lpthread
gcc: error: /home/tim/src/julia-master//deps/scratch/OpenBLAS.v0.3.10+3.x86_64-linux-gnu-libgfortran5: No such file or directory
gcc: error: bb-uninstaller/libopenblas64_.a: No such file or directory
make: *** [Makefile:60: bin/perf0] Error 1

Looking in scratch, the last OpenBLAS version there is OpenBLAS.v0.3.5. In deps/srccache I found a OpenBLAS.v0.3.10+3.x86_64-linux-gnu-libgfortran5.tar.gz. I tried creating a OpenBLAS.v0.3.10+3.x86_64-linux-gnu-libgfortran5 subdirectory of scratch, then unpacked the tar file in it. But now I get

$ JULIAHOME=~/src/julia-master make
gcc -m64 -std=c99 -O0 perf.c -o bin/perf0  -I/home/tim/src/julia-master/deps/scratch/dsfmt-2.2.4 /home/tim/src/julia-master/deps/scratch/OpenBLAS.v0.3.10+3.x86_64-linux-gnu-libgfortran5 bb-uninstaller/libopenblas64_.a -L/home/tim/src/julia-master/usr/lib/ /home/tim/src/julia-master/usr/lib/libopenlibm.a  -lpthread
gcc: error: bb-uninstaller/libopenblas64_.a: No such file or directory
make: *** [Makefile:60: bin/perf0] Error 1

[MilesCranmer]

It doesn't look like libopenblas64_.a is generated by the Julia build anymore:

find ~/julia -name 'libopenblas*'
/home/runner/julia/usr/lib/libopenblas64_.0.3.13.so
/home/runner/julia/usr/lib/libopenblas64_.so
/home/runner/julia/usr/lib/libopenblas64_.so.0

It's also not inside the .tar.gz as @timholy found. Any suggestions appreciated as I try to get this benchmark suite working again (see #51).

[acxz]

@MilesCranmer do you think it would be worth it to replace building our own Julia to instead use a github action like for the other languages?

Is this something feasible/appropriate?
If you would want to create a PR with this change we could see how it fares in the CI.

Maybe I should ask why should we build Julia from source to run these benchmarks now that we have binaries readily available for later versions. (remember that much of the codebase here is around the v0.6 era.

[MilesCranmer]

I'm not sure, what do you think? I think the original reason I made it build Julia from source was so that you could see the updated benchmarks on every commit to the Julia tree. But maybe it's more useful to only study changes from one entire version to the next, and so compiling Julia from scratch every time isn't worth it.

[acxz]

I don't really think this is the proper repo or pipeline, to see changes commit by commit. I think version numbers (particularly stable releases) are sufficient, especially considering it will more accurately reflect results for most users.

Let's try to use the julia binary, instead of building from source.

[MilesCranmer]

Sounds good to me 👍