====== Intallation of LOFAR in Red Hat Enterprise 5.8 ======

This is the description of the installation of LOFAR on the cluster of the [[http://www.iaa.es/|IAA]] in [[http://es.wikipedia.org/wiki/Granada|Granada]]. It bundles an old version of Red Hat Enterprise Linux that does not contain many of the new libraries required by LOFAR.

Description of the system:
  * Operating system: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
  * PBS queue system: [[http://www.adaptivecomputing.com/products/open-source/torque/|TORQUE]]
  * Libraries available: HDF5 and old versions of boost and fftw3.

After some tests we determined that we needed to install, at least, the following software and libraries:
  * scons
  * WCS
  * lapack
  * atlas
  * fftw3
  * Python
    * numpy
    * virtualenv (optional)
    * h5py (optional)
    * scipy
    * matplotlib
    * pandas (optional)
    * pyfits
    * pywcs
    * astropy (optional)
    * cython
    * IPython (optional)
  * Boost
  * LOFAR related:
    * casacore
    * pyrap
    * casarest
    * LOFAR (LofIm)

===== Installation steps =====

The software was installed in the home of the user ''jsm'' (''/home/users/dae/jsm''). You should change the scripts accordingly.

We will install Python in ''/home/users/dae/jsm/python'' and the rest of libraries and software in ''/home/users/dae/jsm/local''. The HDF5 library is already in the system in ''/usr/local/hdf5-1.8.11-linux-x86_64-shared''. A standalone version of casapy is also present in the system.

Previous steps:
<code bash>
## Create the installation directory
mkdir -p /home/users/dae/jsm/local/src
cd /home/users/dae/jsm/local/src
</code>

Download and copy the data:
<code bash>
# scons
wget http://prdownloads.sourceforge.net/scons/scons-2.3.0.tar.gz
# WCS
wget ftp://ftp.atnf.csiro.au/pub/software/wcslib/wcslib.tar.bz2
# lapack
wget http://www.netlib.org/lapack/lapack-3.4.2.tgz
# atlas
wget http://prdownloads.sourceforge.net/math-atlas/Developer%20%28unstable%29/3.11.11/atlas3.11.11.tar.bz2
# FFTW3
wget http://www.fftw.org/fftw-3.3.3.tar.gz
# Boost
wget http://prdownloads.sourceforge.net/boost/boost/1.54.0/boost_1_54_0.tar.gz
# LOFAR related software
# casacore
svn co http://casacore.googlecode.com/svn/tags/casacore-1.5.0
# pyrap
svn co http://pyrap.googlecode.com/svn/tags/pyrap-1.1.0
# casarest
svn co https://svn.astron.nl/casarest/trunk/casarest/
</code>

We had already got a copy of the LOFAR software with SVN in ''/home/users/dae/jsm/LOFAR'':
<code bash>
# LOFAR
cp -r /home/users/dae/jsm/LOFAR .
</code>

We used the following paths:
<code bash>
# PATH
export PATH=/home/users/dae/jsm/local/bin:/home/users/dae/jsm/python/bin:/usr/local/casapy-stable-42.0.25430-001-64b:/usr/local/iraf/bin:/usr/local/mpich-install/bin:/usr/local/bin:/bin:/usr/bin
# LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/home/users/dae/jsm/local/lib:/home/users/dae/jsm/local/atlas/lib:/home/users/dae/jsm/python/lib:/usr/local/hdf5-1.8.11-linux-x86_64-shared/lib:/usr/local/mpich-install/lib:/usr/local/lib
</code>

Install Python and some Python libraries:
<code bash>
# Compile
mkdir -p /home/users/dae/jsm/python/src
cd /home/users/dae/jsm/python/src
# Download Python **** CHANGE
# Extract Python **** CHANGE
cd Python-2.7.5 
./configure --prefix=/home/users/dae/jsm/python --enable-shared
make 
make install

# Install zeromq (used by IPython parallel)
cd /home/users/dae/jsm/python/src
wget http://download.zeromq.org/zeromq-3.2.4.tar.gz
tar xzf zeromq-3.2.4.tar.gz
cd zeromq-3.2.4
./configure --prefix=/home/users/dae/jsm/local
make 
make install

# Install easy install and pip
cd /home/users/dae/jsm/python/src
wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
python ez_setup.py
wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
# rm -r /tmp/pip-build-jsm/pip
python get-pip.py

# Install packages
pip install virtualenv
pip install numpy
pip install h5py
pip install scipy
pip install matplotlib
pip install pandas
pip install pyfits
pip install pywcs
pip install astropy
pip install cython

# Install pyzmq
cd /home/users/dae/jsm/python/src
wget https://pypi.python.org/packages/source/p/pyzmq/pyzmq-13.0.2.tar.gz --no-check-certificate
tar xzf pyzmq-13.0.2.tar.gz
cd pyzmq-13.0.2
python setup.py configure --zmq=/home/users/dae/jsm/local/
python setup.py install

# Install IPython
cd /home/users/dae/jsm/python/src
pip install ipython[all]
</code>

From now on we need to enter an interactive PBS job to avoid our jobs to be killed after certain amount of time (the maximum allowed length for a job without using PBS is 10 minutes). The nodes do not have direct access to the Internet (that is why the packages were previously downloaded):

<code>
### Enter interactive qsub session
qsub -I -X -N LOFAR_soft

## Update the PATH and the LD_LIBRARY_PATH
# PATH
export PATH=/home/users/dae/jsm/local/bin:/home/users/dae/jsm/python/bin:/usr/local/hdf5-1.8.11-linux-x86_64-shared/bin:/usr/local/casapy-stable-42.0.25430-001-64b:/usr/local/iraf/bin:/usr/local/mpich-install/bin:/usr/local/bin:/bin:/usr/bin
# LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/home/users/dae/jsm/local/lib:/home/users/dae/jsm/local/atlas/lib:/home/users/dae/jsm/python/lib:/usr/local/hdf5-1.8.11-linux-x86_64-shared/lib:/usr/local/mpich-install/lib:/usr/local/lib
</code>
We needed to manually specify the paths to avoid cmake finding old or incorrect versions of the libraries.

Compile the libraries and tools:
<code bash>
## scons
cd /home/users/dae/jsm/local/src
tar xfz scons-2.3.0.tar.gz
cd scons-2.3.0
python setup.py install --prefix=/home/users/dae/jsm/local

## ATLAS + LAPACK + BLAS
cd /home/users/dae/jsm/local/src
tar xfj atlas3.11.11.tar.bz2
mkdir -p ATLAS/build
cd ATLAS/build
../configure -b 64 -D c -DPentiumCPS=2932 --shared \
  --prefix=/home/users/dae/jsm/local/atlas \
  --with-netlib-lapack-tarfile=/home/users/dae/jsm/local/src/lapack-3.4.2.tgz
make build
make check
make ptcheck
make time
make install
ln -s /home/users/dae/jsm/local/atlas/lib/libcblas.a \
  /home/users/dae/jsm/local/atlas/lib/libblas.a

## WCS
cd /home/users/dae/jsm/local/src
tar xfj wcslib.tar.bz2
cd wcslib-4.19
./configure --prefix=/home/users/dae/jsm/local/
make
make install

## FFTW3
cd /home/users/dae/jsm/local/src
tar xfz fftw-3.3.3.tar.gz
cd fftw-3.3.3
./configure --enable-threads --enable-shared \
  --prefix=/home/users/dae/jsm/local/
make
make install
make clean
./configure --enable-threads --enable-shared  --enable-float \
  --prefix=/home/users/dae/jsm/local/
make 
make install

## Boost 
cd /home/users/dae/jsm/local/src
tar xfz boost_1_54_0.tar.gz
cd boost_1_54_0
./bootstrap.sh --prefix=/home/users/dae/jsm/local \
  --with-python=/home/users/dae/jsm/python/bin/python
./b2 
./b2 install
</code>

Some notes:
  * You should adapt the parameter "DPentiumCPS=2932" to your system. 
  * 


Compile casacore
<code bash>
## casacore
cd /home/users/dae/jsm/local/src
cd casacore-1.5.0
mkdir build; cd build
cmake .. -DWCSLIB_LIBRARY=/home/users/dae/jsm/local/lib/libwcs.so \
         -DWCSLIB_INCLUDE_DIR=/home/users/dae/jsm/local/include/ \
         -DCMAKE_INSTALL_PREFIX=/home/users/dae/jsm/local/casacore \
         -DUSE_HDF5=ON -DUSE_FFTW3=ON \
         -DHDF5_ROOT_DIR=/usr/local/hdf5-1.8.11-linux-x86_64-shared \
         -DLAPACK_LIBRARIES=/home/users/dae/jsm/local/atlas/lib/liblapack.a \
         -DBLAS_LIBRARIES=/home/users/dae/jsm/local/atlas/lib/libf77blas.a\;/home/users/dae/jsm/local/atlas/lib/libcblas.a\;/home/users/dae/jsm/local/atlas/lib/libatlas.a \
         -DBLA_VENDOR=ATLAS \
         -DFFTW3_LIBRARY=/home/users/dae/jsm/local/lib/libfftw3.so \
         -DFFTW3_THREADS_LIBRARY=/home/users/dae/jsm/local/lib/libfftw3_threads.so \
         -DFFTW3F_LIBRARY=/home/users/dae/jsm/local/lib/libfftw3f.so \
         -DFFTW3F_THREADS_LIBRARY=/home/users/dae/jsm/local/lib/libfftw3f_threads.so \
         -DFFTW3_INCLUDE=/home/users/dae/jsm/local/include/ \
         -DCMAKE_Fortran_COMPILER:FILEPATH=/usr/bin/gfortran
make
make install
</code>

Install pyrap (scons needed):
<code bash>
## pyrap

# pyrap looks for libblas
cd /home/users/dae/jsm/local/atlas/lib
ln -s libcblas.a libblas.a

cd /home/users/dae/jsm/local/src
cd pyrap-1.1.0
## PATCH pyrap ******* (see next section)
python batchbuild.py --boost-root=/home/users/dae/jsm/local \
  --casacore-root=/home/users/dae/jsm/local/casacore \
  --enable-hdf5 --hdf5-root=/usr/local/hdf5-1.8.11-linux-x86_64-shared \
  --lapack-root=/home/users/dae/jsm/local/atlas \
  --wcs-root=/home/users/dae/jsm/local \
  --prefix=/home/users/dae/jsm/local 
</code>

Install casarest
<code bash>
## casarest
cd /home/users/dae/jsm/local/src
cd casarest
mkdir build; cd build
cmake .. -DBOOST_ROOT=/home/users/dae/jsm/local \
         -DLAPACK_LIBRARIES=/home/users/dae/jsm/local/atlas/lib/liblapack.a \
         -DLAPACK_INCLUDE=/home/users/dae/jsm/local/atlas/include/ \
         -DBLAS_LIBRARIES=/home/users/dae/jsm/local/atlas/lib/libf77blas.a\;/home/users/dae/jsm/local/atlas/lib/libcblas.a\;/home/users/dae/jsm/local/atlas/lib/libatlas.a \
         -DBLAS_INCLUDE=/home/users/dae/jsm/local/atlas/include/ \
         -DWCSLIB_LIBRARY=/home/users/dae/jsm/local/lib/libwcs.so \
         -DWCSLIB_INCLUDE_DIR=/home/users/dae/jsm/local/include/ \
         -DCASACORE_ROOT_DIR=/home/users/dae/jsm/local/casacore \
         -DCMAKE_INSTALL_PREFIX=/home/users/dae/jsm/local/casarest \
         -DHDF5_ROOT_DIR=/usr/local/hdf5-1.8.11-linux-x86_64-shared \
         -DLIB_EXTRA_SYNTHESIS=/home/users/dae/jsm/local/atlas/lib/libcblas.a\;/home/users/dae/jsm/local/atlas/lib/libatlas.a \
         -DBUILD_ALL=1
make
make install
</code>

Install LOFAR:
<code bash>
cd /home/users/dae/jsm/local/src
#cp -r ~/LOFAR .
cd LOFAR
### PATCH LOFAR ******** (see next section)
mkdir -p build/gnu_opt; cd build/gnu_opt
mkdir /home/users/dae/jsm/local/LofIm
cmake ../.. -DBUILD_SHARED_LIBS=ON \
  -DCMAKE_INSTALL_PREFIX=/home/users/dae/jsm/local/LofIm \
  -DUSE_LOG4CPLUS=OFF \
  -DBOOST_ROOT=/home/users/dae/jsm/local \
  -DHDF5_ROOT_DIR=/usr/local/hdf5-1.8.11-linux-x86_64-shared \
  -DWCSLIB_LIBRARY=/home/users/dae/jsm/local/lib/libwcs.so \
  -DWCSLIB_INCLUDE_DIR=/home/users/dae/jsm/local/include/ \
  -DLAPACK_LIBRARIES=/home/users/dae/jsm/local/atlas/lib/libf77blas.a\;/home/users/dae/jsm/local/atlas/lib/libcblas.a\;/home/users/dae/jsm/local/atlas/lib/libatlas.a\;/home/users/dae/jsm/local/atlas/lib/liblapack.a \
  -DBLAS_LIBRARIES=/home/users/dae/jsm/local/atlas/lib/libf77blas.a\;/home/users/dae/jsm/local/atlas/lib/libcblas.a\;/home/users/dae/jsm/local/atlas/lib/libatlas.a\;/home/users/dae/jsm/local/atlas/lib/liblapack.a  \
  -DFFTW3_LIBRARY=/home/users/dae/jsm/local/lib/libfftw3.so \
  -DFFTW3_THREADS_LIBRARY=/home/users/dae/jsm/local/lib/libfftw3_threads.so \
  -DFFTW3F_LIBRARY=/home/users/dae/jsm/local/lib/libfftw3f.so \
  -DFFTW3F_THREADS_LIBRARY=/home/users/dae/jsm/local/lib/libfftw3f_threads.so \
  -DFFTW3_INCLUDE=/home/users/dae/jsm/local/include/ \
  -DCASACORE_ROOT_DIR=/home/users/dae/jsm/local/casacore \
  -DCASA_CASA_LIBRARY=/home/users/dae/jsm/local/casacore/lib/libcasa_casa.so\;/home/users/dae/jsm/local/atlas/lib/libf77blas.a\;/home/users/dae/jsm/local/atlas/lib/libcblas.a\;/home/users/dae/jsm/local/atlas/lib/libatlas.a\;/home/users/dae/jsm/local/atlas/lib/liblapack.a \
  -DUSE_OPENMP=ON \
  -DBUILD_PACKAGES=Offline\;LofarFT\;StaticMetaData\;SPW_Combine \
  -DCASAREST_ROOT_DIR=/home/users/dae/jsm/local/casarest/ \
  -DPYTHON_LIBRARY=/home/users/dae/jsm/python/lib/libpython2.7.so \
  -DPYTHON_EXECUTABLE:FILEPATH=/home/users/dae/jsm/python/bin/python \
  -DPYTHON_INCLUDE_DIR:PATH=/home/users/dae/jsm/python/include \
  -DPYRAP_INCLUDE_DIR=/home/users/dae/jsm/local/include/pyrap \
  -DPYRAP_LIBRARY=/home/users/dae/jsm/local/lib/libpyrap.so \
  -DCMAKE_Fortran_COMPILER:FILEPATH=/usr/bin/gfortran \
  -DF2PY_FCOMPILER=gnu95 2>&1 | tee log.txt
make
make install
</code>

Finish the installation (FIXME):
  * Configure .bashrc
  * Configure .casarc

==== Patches and hacks needed ====

Pyrap and LOFAR need to be patched to allow them to compile in this old system.

**Pyrap** tries to find the libraries in lib64 in a Red Hat linux. This happens even if the libraries are manually compiled and are located in a ''lib'' directory. If the user cannot create a symbolic link from ''lib64'' to ''lib'', pyrap needs to be patched. The ''lib64'' entries linked to ''redhat'' found in the setupext.py files and utils.py need to be edited to ''lib''.
Edit:
  * pyrap_fitting/current/setupext.py
  * pyrap_functionals/current/setupext.py
  * pyrap_images/current/setupext.py
  * pyrap_measures/current/setupext.py
  * pyrap_quanta/current/setupext.py
  * pyrap_tables/current/setupext.py
  * pyrap_util/current/setupext.py
  * libpyrap/current/scons-tools/utils.py 

**Pyrap** cannot determine the location of some shared libraries in this old system. //Ugly hack// - The file ''libpyrap/current/SConstruct'' can be edited to ad some linker flags:
After
<code Python>
    conf = Configure(env)
</code>
insert:
<code Python>
    conf.env.Append(LINKFLAGS = ["-Wl,-rpath,/home/users/dae/jsm/local/casacore/lib:/usr/local/hdf5-1.8.11-linux-x86_64-shared/lib:/home/users/dae/jsm/local/lib"])
</code>

**Pyrap** looks for libblas instead of libcblas. A symbolic link can be made:
<code bash>
cd /home/users/dae/jsm/local/atlas/lib
ln -s libcblas.a libblas.a
</code>

**LOFAR** uses the flag ''-Wno-ignored-qualifiers'' that is not supported by this old system. It has to be removed from the following files:
  * CEP/Imager/LofarFT/src/CMakeLists.txt
  * CEP/Calibration/BBSTools/src/CMakeLists.txt
  * LCS/AMC/AMCImpl/src/CMakeLists.txt

The version of cmake bundled in the system contains buggy versions of the recipes to find BLAS and LAPACK (FindBLAS.cmake and FIindLAPACK.cmake). Ii is possible to see in the CMakeCache.txt files that these libraries are not detected by CMAKE in this system (for example in the case of **casacore**). The manual setting of the cache variables //LAPACK_LIBRARIES// and //BLAS_LIBRARIES// made the compilation and linking work. The variables //LAPACK_INCLUDE// and //BLAS_INCLUDE// were ignored. But... take into account that these cache variables may not be present in modern compilations of cmake.

//The ugliest hack// is this line in the cmake options of **LOFAR**: 
<code>
-DCASA_CASA_LIBRARY=/home/users/dae/jsm/local/casacore/lib/libcasa_casa.so\;/home/users/dae/jsm/local/atlas/lib/libf77blas.a\;/home/users/dae/jsm/local/atlas/lib/libcblas.a\;/home/users/dae/jsm/local/atlas/lib/libatlas.a\;/home/users/dae/jsm/local/atlas/lib/liblapack.a
</code>
The library librfiengine.so is linked with dependencies to LAPACK (and this can present dependencies with BLAS). These dependencies are not linked in some of the remaining packages because they are not a direct dependency but the linker will fail. We overrided the cache variable pointing to one of the casacore libraries. Wherever librfiengine.so is linked, the libcasa_casa.so is linked as well. This ugly thing worked.

==== Warnings and advice ====

  * Compile the libraries with the ''shared'' option (-fPIC in the compiler).
  * The tools ''ldd'' and ''nm'' could help you to debug problems with the linking of libraries.
  * Have a look to CMakeCache.txt before running cmake to spot possible problems with the libraries.
  * If you use the ifort FORTRAN compiler you might find problems with the linking of ifort shared libraries. 
  * Cmake could find some of the casapy libraries and link them as dependencies. If you want to avoid that you should specify the cache variables by hand (as it is done in this example).
  * To link to precompiled external libraries (like the ones provided by casapy or Anaconda) might be a bad idea.
  * The option "--blas" of pyrap did not work and made the build script fail.
  * Final advice: Try to work on modern systems (e. g. Ubuntu 12.04 LTS) if you can :).