====== Installing LOFAR Software on a CentOS 6.5 (Final) System - The Newton Cluster of the AIP Potsdam ======

This document is evolving and was created by
 --- //[[fbreitling@aip.de|Frank Breitling]] 2015/06/26 18:47//


A useful starting point were previous installation notes such as: \\

http://www.lofar.org/operations/doku.php?id=public:user_software:lofar \\
http://www.lofar.org/operations/doku.php?id=public:user_software:ubuntu_12_4

However some additional issues occurred at the Newton cluster and had to be solved as described below. \\
Ideas and support by **Arno Schoenmakers**, **Ger van Diepen** and **Marcel Loose** were very helpful and highly appreciated. \\


===== 1. Setup of build environment =====

The following packages are already provided on the cluster:

  * BLAS
  * LAPACK
  * Boost 1.41.0 (but too old for AOFlagger v2.8.0 and later)
  * cfitsio 3.240
  * HDF5
  * OpenMP

The cluster also provides a //module system// to load newer versions of the installed software. \\
Its used below to replace the default version of gcc 4.4.7 with gcc 4.8.3 for C++11 support.


=== 1.1 Paths ===

The following changes are required:

  * A more recent version of GCC with C++11 required by Casacore is activated through the module system. This also requires adding /opt/gcc-4.8.3/lib64 to the LD_LIBRARY_PATH since otherwise Pyrap will not find it and complain with
<file>
File "/home/fkbreitl/local/lib64/python2.6/site-packages/python_casacore-2.1.2-py2.6-linux-x86_64.egg/casacore/tables/table.py", line 43, in <module>
    from ._tables import Table
ImportError: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /home/fkbreitl/local/lib/libboost_python.so.1.63.0)
</file>

  * The packages will be installed in $HOME/local/. \\
  * Pyrap (python-casacore) needs a writable Python path. \\

This adds the following lines to $HOME/.bashrc:
<file>
source /usr/share/Modules/init/sh  # activate module system, needs to be done first
module load gcc/4.8.3  # use gcc 4.8.3 (instead of 4.4.7) for C++11 support
export LD_LIBRARY_PATH=/opt/gcc-4.8.3/lib64:$HOME/local/lib:$LD_LIBRARY_PATH
export PYTHONPATH=$HOME/local/lib64/python2.6/site-packages:$PYTHONPATH
export PATH=$HOME/local:$PATH
</file>


=== 1.2 CMake ===

The 4 Newton head nodes provide CMake 2.6 by default, but the LOFAR Software requires at least 2.6.6. A sufficiently recent version of CMake 3.0.0 is provided via [[https://en.wikipedia.org/wiki/Environment_Modules_%28software%29|Environment Modules]] (see also http://modules.sourceforge.net/).\\
The environment modules also provide more recent HDF5 libraries. Both could be loaded by adding ''module load cmake hdf5'' to $HOME/.bashrc. \\
But unfortunately the CMake 3.0.0 loaded on the head node newl1 is broken. Furthermore loading the HDF5 libs has no effect on the libs found by CMake, which are still the system's default. So we don't load any module and don't add this line to the .bashrc. Instead we build our own CMake as follows:

<file>
mkdir -p $HOME/local/src
cd $HOME/local/src
wget http://www.cmake.org/files/v3.2/cmake-3.2.3.tar.gz
tar xf cmake-3.2.3.tar.gz
cd cmake-3.2.3
.bootstrap --prefix $HOME/local
make -j12
make install
</file>


=== 1.3 Python ===
To use matplotlib 2 one also needs Python 2.7 or later.

<file>
V_PY=2.7.13
cd ~/local/src/
wget https://www.python.org/ftp/python/$V_PY/Python-$V_PY.tgz
tar xf Python-$V_PY.tgz
cd Python-$V_PY
./configure --prefix=$HOME/local --enable-shared                               
time make -j
make install
</file>



=== 1.4 Boost ===
Also a newer version of [[http://www.boost.org/build/doc/html/bbv2/installation.html | Boost]] is required since AOFlagger v2.8.0 and later:

<file>
cd $HOME/local/src
V_BOOST=1_63_0
wget wget https://sourceforge.net/projects/boost/files/boost/$V_BOOST/boost_$V_BOOST.tar.bz2
tar xf boost_$V_BOOST.tar.bz2
cd boost_$V_BOOST
./bootstrap.sh
time ./b2 install --prefix=${HOME}/local
</file>


===== 2. Installation of dependencies =====

Next we need to build and install these other packages as follows:

=== 2.1 WCSLIB ===
The [[http://www.atnf.csiro.au/people/mcalabre/WCS/ | WCSLIB]]:

<file>
cd $HOME/local/src/
wget ftp://ftp.atnf.csiro.au/pub/software/wcslib/wcslib.tar.bz2
tar xf packages/wcslib.tar.bz2
cd wcslib-5.15  # (-5.6)
./configure --prefix=$HOME/local/
time make -j install
</file>


=== 2.2 FFTW ===
The [[http://www.fftw.org/ | FFTW lib]]:

<file>
V_FFTW=3.3.5  # 3.3.4
cd $HOME/local/src/
wget http://www.fftw.org/fftw-$V_FFTW.tar.gz
cd fftw-$V_FFTW
./configure --prefix=$HOME/local --enable-threads --enable-shared --enable-float --enable-sse --enable-sse2 --enable-avx
time make -j install
</file>



===== 3. Installation the LOFAR software =====
Now we can build the LOFAR software. \\

We will need the following CMake switches:

   1. -DCMAKE_INSTALL_PREFIX:PATH=$HOME/local
   2. -DBOOST_ROOT=$HOME/local/src/boost_1_63_0
   3. -DCMAKE_CXX_FLAGS="-lboost_thread -lboost_filesystem -L$HOME/local/lib"
   # 4. -DBoost_NO_BOOST_CMAKE=YES (not necessary with new installed Boost)

1.) Sets the installation path to $HOME/local.

2.) Necessary to find the new Boost installation

3.) Necessary to link against the new Boost installation

# 4.) Was required by some older versions of boost e.g. 1.41 with cmake-2.8.6-rc2, producing this problem: \\
https://stackoverflow.com/questions/9948375/cmake-find-package-succeeds-but-returns-wrong-path resulting in\\
<file>
make[2]: *** No rule to make target `/usr/lib64/lib64/libboost_program_options-mt.so.5', needed by `segment'.  Stop.
make[1]: *** [CMakeFiles/segment.dir/all] Error 2
make: *** [all] Error 2
</file>
It is not necessary any more with the new boost installation.


=== 3.2 Casacore ===
[[https://github.com/casacore/casacore | Casacore]] also requires the CASA data. 
This is for example provided with [[http://casa.nrao.edu/casa_obtaining.shtml | CASA]]. \\
Here we assume CASA is already installed and can be found in the $CASAPATH given below. For convenience this is a link to the CASA folder.
Then the data is in $CASAPATH/data. \\
We will specify this via -DDATA_DIR, otherwise we need a ~/.casarc file with the line: ''measures.directory: $HOME/local/casa/casa/data'' .\\

''-DENABLE_TABLELOCKING=NO'' switches off table locking, since read locks could not be acquired from the AIP Lustre file system leading to errors like:
<file>
msoverview: Version 20110407GvD
2015-06-30 19:32:19     INFO            Process 13592: waiting for read-lock on file
/lustre/fkbreitl/data/L206894/L206894.cal/L206894_SAP000_SB200_uv.MS.dppp_00001~00040/table.lock
Error: Error (Function not implemented) when acquiring lock on
/lustre/fkbreitl/data/L206894/L206894.cal/L206894_SAP000_SB200_uv.MS.dppp_00001~00040/table.lock
</file>
For practical work there is no disadvantage in disabling table locking. \\

<file>
# 2.1, 2.2
cd ~/local/src/
git clone https://github.com/casacore/casacore
mkdir build/casacore
cd build/casacore
CASAPATH=$HOME/local/casa/casa
time cmake ../../casacore -DBUILD_PYTHON=YES -DCMAKE_INSTALL_PREFIX=~/local -DUSE_FFTW3=YES -DENABLE_TABLELOCKING=NO \
  -DUSE_OPENMP=YES -DDATA_DIR=$CASAPATH/data -DUSE_HDF5=YES -DCMAKE_PREFIX_PATH=$HOME/local
time make -j
make install
</file>


=== 3.3 Pyrap ===
<file>
cd ~/local/src/
git clone https://github.com/casacore/python-casacore.git
cd python-casacore
python setup.py build_ext -I/usr/include/cfitsio:$HOME/local/include -L$HOME/local/lib
python setup.py install --prefix=$HOME/local
</file>


== 3.3.1 Alternative Pyrap installation ==
Alternatively we can install [[https://github.com/casacore/python-casacore | Pyrap]] with //pip// if we install pip before:
<file>
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py --user
~/.local/bin/pip install python-casacore --global-option=build_ext --global-option=-I/usr/include/cfitsio:$HOME/local/include \
   --global-option=-L$HOME/local/lib --install-option=--prefix=$HOME/local
</file>


=== 3.4 Casarest ===
Now we continue with the rest:
<file>
cd ~/local/src/
svn co https://svn.astron.nl/casarest/trunk/casarest
mkdir build/casarest
cd build/casarest
cmake ../../casarest -DCMAKE_INSTALL_PREFIX:PATH=$HOME/local -DCASACORE_ROOT_DIR=$HOME/local -DCMAKE_PREFIX_PATH=/usr/include/cfitsio -DBUILD_ALL=1 \
  -DBOOST_ROOT=$HOME/local/src/boost_1_63_0 -DCMAKE_CXX_FLAGS="-lboost_thread -lboost_filesystem -L$HOME/local/lib"
# not necessary any more -DBoost_NO_BOOST_CMAKE=YES
time make -j
make install
</file>


=== 3.5 AOFlagger ===
The [[https://sourceforge.net/projects/aoflagger/ | AOFlagger]] is also required since the 
[[http://www.lofar.org/wiki/doku.php?id=public:offline_release_2_15_0 | LOFAR Offline release 2.15.0]]. \\
For further details see the LOFAR release link or the 
[[https://sourceforge.net/p/aoflagger/wiki/installation_instructions/ | installation instructions]]. \\
The AOFlagger v2.8.0 and later requires a version of Boost later than 1.41.0.

<file>
V_AOF=2.9.0  # 2.7.1, 2.8.0
cd ~/local/src/
wget http://downloads.sourceforge.net/project/aoflagger/aoflagger-$AOVER/aoflagger-$V_AOF.tar.bz2
tar xf aoflagger-$V_AOF.tar.bz2
mkdir aoflagger-$V_AOF/build/
cd aoflagger-$V_AOF/build/
time cmake ../ -DCMAKE_INSTALL_PREFIX:PATH=$HOME/local -DBUILD_SHARED_LIBS=YES -DBOOST_ROOT=$HOME/local/src/boost_1_63_0 \
  -DCMAKE_CXX_FLAGS="-lboost_thread -lboost_filesystem -L$HOME/local/lib"
time make -j
make install
</file>


=== 3.6 The LOFAR packages ===
Finally we can build the LOFAR packages. We can chose between [[http://www.lofar.org/operations/doku.php?id=maintenance:lofar_release_notes | a release]], 
[[http://www.lofar.org/operations/doku.php?id=public:lofar_offline_releases | an offline release]] 
or the latest version in [[https://svn.astron.nl/LOFAR/ | trunk/]] by commenting in and out the corresponding RELEASE lines. \\

''-DUSE_BACKTRACE=NO'' is necessary for building shared libraries (the default), but libiberty.so is not installed.\\
(See https://support.astron.nl/lofar_issuetracker/issues/8046).

<file>
cd ~/local/src/
RELEASE=LOFAR-Release-2_19 && svn checkout --ignore-externals -N https://svn.astron.nl/LOFAR/branches/$RELEASE
# 2_16_4, 2_16, 2_12
#RELEASE=LOFAR && svn checkout --ignore-externals -N https://svn.astron.nl/LOFAR/trunk $RELEASE
svn update $RELEASE/CMake
mkdir -p build/gnu_opt
cd build/gnu_opt
time cmake ../../$RELEASE -DCASACORE_ROOT_DIR=$HOME/local -DBUILD_PACKAGES="DP3 Calibration" -DCMAKE_INSTALL_PREFIX:PATH=$HOME/local -DUSE_BACKTRACE=NO
# -DUSE_LOG4CPLUS=NO was required in absence of the Log4cplus lib
time make -j
make install
</file>

Done!

Now the commands ''NDPPP'' and ''calibrate-stand-alone'' should be available on this system.

Enjoy!