Building HPL and ATLAS for the Raspberry Pi

Following on from work by Simon Cox and colleagues building the first Raspberry Pi Cluster we are building a new cluster and want to compare the performance of the two.  To do this we are once again going to run Linpack.  Versions of MPI and HPL have moved on since then.  When Wee Archie Blue was being developed issues were found with the off the shelf atlas version so we are building it from scratch. Here’s an updated set of install instructions for running on Raspbian.

  1. Install the dependencies
sudo apt install gfortran automake

2. Download atlas from https://sourceforge.net/projects/math-atlas/. At the time of writing this is version 3.10.3, your version might be different.

tar xjvf atlas3.10.3.tar.bz2

3. Create a directory to build in (it’s recommended to not build in the source hierarchy), and cd into it


mkdir atlas-build
cd atlas-build/

4. Disable CPU throttling on the Pi – the process will not start if it detects throttling.

echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

This will stop throttling, I found it helped to have a fan blowing air over the CPU to make sure it didn’t over heat.

5. Configure and build Atlas. These steps will take a while. Do NOT use the -j flag to parallelize the make process, this will cause inconsistent results. Where possible it will run operations in parallel automatically.

../ATLAS/configure
make

6. Download MPI and install

cd
wget http://www.mpich.org/static/downloads/3.2/mpich-3.2.tar.gz
tar xzvf mpich-3.2.tar.gz
cd mpich-3.2
./configure
make -j 4
sudo make install

6. Download HPL from and extract configure

cd
wget http://www.netlib.org/benchmark/hpl/hpl-2.2.tar.gz
tar xzvf hpl-2.2.tar.gz
cd hpl-2.2
cd setup
sh make_generic
cp Make.UNKNOWN ../Make.rpi
cd ..

7. Then edit Make.rpi to reflect where things are installed.  In our case the following lines are edited from the default. Note line numbers might change with future versions

ARCH = rpi

[…]

TOPdir       = $(HOME)/hpl-2.2

[…]

MPdir        = /usr/local
MPinc        = -I $(MPdir)/include
MPlib        = /usr/local/lib/libmpich.so

[…]

LAdir        = /home/pi/atlas-build
LAinc        =
LAlib        = $(LAdir)/lib/libf77blas.a $(LAdir)/lib/libatlas.a

8. Then compile HPL

make arch=rpi

Congratulations. You should now have a working HPL install. Let’s test it.

9. Change into the working directory and create the configuration needed to test the system. As the pi has 4 cores you need to tell mpi to assign 4 tasks to the host. Depending on the ambient temperature you may need to add a fan to stop the Pi CPU overheating as these tests are very demanding.

cd bin/rpi
cat << EOF > nodes-1pi
localhost
localhost
localhost
localhost
EOF

Customise the HPL.dat input file. The file below is the starting point we use

HPLinpack benchmark input file
Innovative Computing Laboratory, University of Tennessee
HPL.out      output file name (if any)
6            device out (6=stdout,7=stderr,file)
1            # of problems sizes (N)
5120         Ns
1            # of NBs
128          NBs
0            PMAP process mapping (0=Row-,1=Column-major)
1            # of process grids (P x Q)
2            Ps
2            Qs
16.0         threshold
1            # of panel fact
2            PFACTs (0=left, 1=Crout, 2=Right)
1            # of recursive stopping criterium
4            NBMINs (>= 1)
1            # of panels in recursion
2            NDIVs
1            # of recursive panel fact.
1            RFACTs (0=left, 1=Crout, 2=Right)
1            # of broadcast
1            BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)
1            # of lookahead depth
1            DEPTHs (>=0)
2            SWAP (0=bin-exch,1=long,2=mix)
64           swapping threshold
0            L1 in (0=transposed,1=no-transposed) form
0            U  in (0=transposed,1=no-transposed) form
1            Equilibration (0=no,1=yes)
8            memory alignment in double (> 0)

Run the test

mpiexec -f nodes-1pi ./xhpl

If all goes to plan your output should look similar to

================================================================================
HPLinpack 2.2  --  High-Performance Linpack benchmark  --   February 24, 2016
Written by A. Petitet and R. Clint Whaley,  Innovative Computing Laboratory, UTK
Modified by Piotr Luszczek, Innovative Computing Laboratory, UTK
Modified by Julien Langou, University of Colorado Denver
================================================================================

An explanation of the input/output parameters follows:
T/V    : Wall time / encoded variant.
N      : The order of the coefficient matrix A.
NB     : The partitioning blocking factor.
P      : The number of process rows.
Q      : The number of process columns.
Time   : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.

The following parameter values will be used:

N      :    5120
NB     :     128
PMAP   : Row-major process mapping
P      :       2
Q      :       2
PFACT  :   Right
NBMIN  :       4
NDIV   :       2
RFACT  :   Crout
BCAST  :  1ringM
DEPTH  :       1
SWAP   : Mix (threshold = 64)
L1     : transposed form
U      : transposed form
EQUIL  : yes
ALIGN  : 8 double precision words

--------------------------------------------------------------------------------

- The matrix A is randomly generated for each test.
- The following scaled residual check will be computed:
      ||Ax-b||_oo / ( eps * ( || x ||_oo * || A ||_oo + || b ||_oo ) * N )
- The relative machine precision (eps) is taken to be               1.110223e-16
- Computational tests pass if scaled residuals are less than                16.0

================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR11C2R4        5120   128     2     2              25.11              3.565e+00
HPL_pdgesv() start time Wed May 16 10:35:46 2018

HPL_pdgesv() end time   Wed May 16 10:36:11 2018

--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.2389736 ...... PASSED
================================================================================

Finished      1 tests with the following results:
              1 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
              0 tests skipped because of illegal input values.
--------------------------------------------------------------------------------

End of Tests.
================================================================================

The above steps were sufficient to get run atlas on a Pi 3B+ testing has shown that on a model 3B it may crash for N values above about 6000. This appears to be a problem with the hardware of the 3B, as described in a post on the pi forum. Following the step described in the post of adding the following line to /boot/config.txt enabled problem sizes up to and including 10240 to be executed.

over_voltage=2