Differences

This shows you the differences between two versions of the page.

--- public:grid_srm_software_installation [2012-11-29 15:45] – Adriaan Renting
+++ public:grid_srm_software_installation [2024-05-10 12:14] (current) – Hanno Holties
@@ Line 1: / Line 1: @@
 ====== GRID storage access tools installation ======
-The following documentation was developed for the installation of a GRID tools and certificates on an Ubuntu 12.04 system using a bash shell (using .bashrc as a configuration script). Please note that especially some of the srm scripts only work under bash, not csh derivatives. Thanks goes to Martin van den Akker for providing most of this information.
+**//:!: Several sections in this page are deprecated or obsolete. In particular, users are advised to avoid SRM clients and globus client software and instead look into using the [[https://dmc-docs.web.cern.ch/dmc-docs/gfal2/gfal2.html|gfal]] client library and command line tools. SURF provides good [[https://doc.grid.surfsara.nl/en/latest/Pages/Advanced/grid_storage.html|documentation on usage of grid storage client software]].// **
-There is also an instruction for installing on an [[public:grid:ubuntu_install|old version of Ubuntu]] that doesn't have the apt-get packages yet.
+//This page describes the procedure to set up the tools for working directly on the srm storage. You may want to consider using the staging and download services provided by Astron if you just want to retrieve data from the archive: [[http://www.lofar.org/wiki/doku.php?id=public:lta_howto|http://www.lofar.org/wiki/doku.php?id=public:lta_howto]].//The following documentation was developed for the installation of a GRID tools and certificates on an Ubuntu 12.04 system using a bash shell (using .bashrc as a configuration script). We have not tested the installation using csh derivatives but it is to be expected that not all scripts will work. Thanks to Martin van den Akker for providing notes of his installation procedure.
-There is also an instruction for installing on [[http://glite.web.cern.ch/glite/packages/R3.2/sl5_x86_64/deployment/glite-UI/glite-UI.asp|Scientific Linux 5 using gLite version 3.2]] or [[http://glite.web.cern.ch/glite/packages/R3.1/deployment/glite-UI/glite-UI.asp|Scientific Linux 4 using gLite version 3.1]]. These use RPM and might also work on Redhat or SuSE or derivatives.
-===== 1 Installation of software packages =====
+Sites that provide packages and further information on installation of grid middleware on linux based systems:
+  * [[http://repository.egi.eu|Software repository of the European Grid Infrastructure]]
+NB We have tested the installation as described below only on Ubuntu 12 and CentOS 7. Please let us know if you have feedback or if you can contribute instructions on installations in different environments. You might also first need to obtain a [[:public:grid_certificate|GRID certificate]].
+===== Installation of software packages =====
 Note: all installations require root permissions.
-==== 1.1 globus ====
+==== globus client software ====
-You will need to install the file transfer tools from the Globus package, most importantly globus-url-copy:
+The file transfer tools from the Globus package are needed, most importantly globus-url-copy:
-  sudo apt-get install globus-gass-copy-progs globus-common-progs globus-core
+Ubuntu
+<code>
+sudo apt-get install globus-gass-copy-progs
-==== 1.2 voms ====
+</code>
-You will need to install the VOMS tools for logging in and user account management:
+CentOS
-  sudo apt-get install voms-clients
+<code>
+sudo yum install globus-gass-copy-progs
-==== 1.3 srmtools ====
+</code>
-You will need to install the SRM tools to be able to communicate with the storage management system.
+==== voms client software ====
+The VOMS tools for logging in and user account management:
+Ubuntu
+<code>
+sudo apt-get install voms-clients
+</code>
+CentOS
+<code>
+sudo yum install voms-clients-cpp
+</code>
+==== Certificates for the Grid Certificate Authorities (CA) ====
+Execute the following commands to install the certificates from the site of the European Grid Infrastructure (EGI) (root permissions are required).
+Ubuntu
+Note 2021-05-19: newer versions of Ubuntu require to first add the appropriate GPG key for the EGI repository:
+<code>
+wget -q -O - https://dist.eugridpma.info/distribution/igtf/current/GPG-KEY-EUGridPMA-RPM-3 | sudo apt-key add -
+</code>
+<code>
+sudo add-apt-repository 'deb http://repository.egi.eu/sw/production/cas/1/current egi-igtf core'
+sudo apt-get update
+sudo apt-get install ca-policy-egi-core
+</code>
+CentOS
+<code>
+wget http://repository.egi.eu/sw/production/cas/1/current/repo-files/EGI-trustanchors.repo
+sudo mv EGI-trustanchors.repo /etc/yum.repos.d/
+sudo yum install ca-policy-egi-core
+</code>
+==== srmtools ====
+The SRM tools are needed to communicate with the storage management system.
   - Download srmtools
-    *  {{:public:srm.tar.gz|srm.tar.gz}}
+      * {{:public:srmclient-2.6.28.tar.gz|srmclient-2.6.28.tar.gz}}  (Java7, Java8)
-    *  http://www.astro.ru.nl/~martinva/software/srm.tar.gz
+      * {{:public:srmclient-2.2.25.tar.gz|srmclient-2.2.25.tar.gz}}  (Java6)
-  - Extract and install the srmtools in ''/opt/''\\ this will create a subdirectory ''srm'' under ''/opt/''
+  - Extract and install the srmtools, e.g. in ''/opt/'' \\ This will create a subdirectory, e.g. ''srmclient-2.6.28'', containing the required files. \\ Note that the srm package may be installed anywhere (e.g. in your home directory).
-  - Add the path to the srmtools to .bashrc:\\ ''export SRM_PATH=/opt/srm''\\ ''export PATH=$SRM_PATH/bin:$PATH''
+  - Set the relevant environment path variables, e.g. in .bashrc (modify version number if applicable): \\ ''export SRM_PATH=<Install Directory>/srmclient-2.6.28/usr/share/srm'' \\ ''export PATH=<Install Directory>/srmclient-2.6.28/usr/bin:$PATH''
-===== 2 Installation of certificates =====
+NB The srm client tools depend on JAVA. There is a known issue with openjava version 7. If you have this version of JAVA installed, or otherwise get JAVA exceptions when running an srm command, please install another JAVA VM. Java-7-oracle is known to work. It is possible to have multiple JAVA VM installations and it is thus not required to replace an existing installation: if the default VM installation does not work with srm, another VM may be configured to be used by setting the following environment parameter:
+<code>
+export JAVA_HOME=/usr/lib/jvm/java-7-oracle/jre
-====2.1 Personal certificate ====
+</code>
-  * Request a user certificate using jGridstart\\ http://ca.dutchgrid.nl/start/jgridstart.jnlp\\ and follow the instructions
+The JAVA VM used by default in Ubuntu and CentOS can be selected using the following command:
-  * Install the certificate as described in the certificate e-mail
-  - Save this entire mail as ''$HOME/.globus/usercert.pem'' and make sure that the ''userkey.pem'' file in that directory matches this certificate. As these ''*.pem'' files are strictly personal their permissions should be set properly by the following command:\\ ''> chmod 600 *.pem''
-  - Load the certificate into your web browser on your own system. See http://ca.dutchgrid.nl/info/browser for instructions. Installation can also be done using the jGridstart tool (see above) for supported browsers.
-  - [optional] Apply for authorization to use resources or services. Please consult your service provider or help desk for more information, e.g.:
-    * https://voms.grid.sara.nl:8443/voms/lofar for access to the LOFAR LTA storage
-    * deisa-support@sara.nl for access to DEISA HPC Grid resources
-    * https://voms.grid.sara.nl:8443/vomses for other GRID communities hosted in the Netherlands
-    * https://cic.gridops.org/index.php?section=home&page=volist for all Grid communities registered in Europe by discipline
-==== 2.2 Additional certificates from the site of the European Grid Infrastructure (EGI). ====
+<code>
+sudo update-alternatives --config java
-Execute the following commands to install the certificates from the
+</code>
-site of the European Grid Infrastructure (EGI) (root permissions are
-required).
-  sudo add-repository 'deb http://repository.egi.eu/sw/production/cas/1/current egi-igtf core'
+==== Certificate Revocation List retrieval (optional) ====
-  sudo apt-get update
-  sudo apt-get install ca-policy-egi-core
+The fetch-crl tool retrieves Certificate Revocation Lists.
-===== 3 Additional configuration =====
+Ubuntu
-==== 3.1 Add VOMSES string ====
+<code>
+sudo apt-get install fetch-crl
-Add the vomses string for the LOFAR Virtual Organization (VO) to the vomses file. You can find this string on the following website https://voms.grid.sara.nl:8443/voms/lofar/configuration/configuration.action in the text block under //VOMSES string for this VO//\\ The string should be copied to the following file: ''$HOME/.glite/vomses''
+</code>
+CentOS
-==== 3.2 List of certificates (voms.grid.sara.nl.lsc) ====
+<code>
+sudo yum install fetch-crl
-Put the following string (without the quotes):
+</code>
-  "/O=dutchgrid/O=hosts/OU=sara.nl/CN=voms.grid.sara.nl"
+NB This is not required unless you intend to allow others to access your system by providing their grid certificate.
+===== Additional configuration =====
+==== VOMSES file for LOFAR ====
+Add the following string for the LOFAR Virtual Organization (VO) to the vomses file (any filename is fine).
+<code>
+"lofar" "voms.grid.sara.nl" "30019" "/O=dutchgrid/O=hosts/OU=sara.nl/CN=voms.grid.sara.nl" "lofar"
+</code>
+You can find this string also on the following website [[https://voms.grid.sara.nl:8443/voms/lofar/configuration/configuration.action|https://voms.grid.sara.nl:8443/voms/lofar/configuration/configuration.action]] in the text block under //VOMSES string for this VO// \\ The vomses file should be placed in one of the following default locations: ''/etc/vomses'', ''$HOME/.voms/vomses'', ''$HOME/.glite/vomses''
+==== List of certificates for voms.grid.sara.nl.lsc ====
+Put the following strings:
+<code>
+/O=dutchgrid/O=hosts/OU=sara.nl/CN=voms.grid.sara.nl
+/C=NL/O=NIKHEF/CN=NIKHEF medium-security certification auth
+</code>
 in the file (root permissions required):
-  /etc/grid-security/vomsdir/lofar/voms.grid.sara.nl.lsc
+<code>
+/etc/grid-security/vomsdir/lofar/voms.grid.sara.nl.lsc
+</code>
+NB If this step is skipped or nor configured correctly ''voms-proxy-init''  will work but finish with warnings.
+==== Environment (optional) ====
+You may want to provide the following settings in ''.bashrc''  or another initialisation/startup script if the relevant files are not in the default locations (defaults provided below).
+<code>
+export X509_USER_CERT=$HOME/.globus/usercert.pem
+export X509_USER_KEY=$HOME/.globus/userkey.pem
+export X509_CERT_DIR=/etc/grid-security/certificates
+export X509_VOMS_DIR=/etc/grid-security/vomsdir
+export X509_USER_PROXY=$HOME/.proxy
+export VOMS_USERCONF=$HOME/.glite
+</code>
+**Note:**  For (t)csh, use *.csh init scripts and 'setenv <key> <value>' instead of 'export <key>=<value>'.
+==== CRL cron job (optional) ====
+You may want to create a cron job to automatically retrieve certificate revocation lists (CRLs) by invoking the fetch-crl tool at regular intervals (at least once a year).
+===== Usage =====
+This creates a proxy (valid for 48 hours, increase if needed) in your home directory:
+<file>
+voms-proxy-init -valid 48:00 -voms lofar:/lofar/user -out ~/.proxy
+</file>
+You can test that everything works by copying this file from surfsara to your working directory:
+<file>
+srmcp -server_mode=passive srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lofar/ops/fifotest/file1M file://`pwd`/file1M
+</file>
-==== 3.3 Environment ====
+If your firewall allows incoming connections to non-standard ports, you can try this command without the server_mode option which will enable utilization of multiple streams to increase performance.
-Add the following settings to ''.bashrc'' or another initialisation/startup script.
+If you have the [[:public:grid_srm_software_installation#globus_client_software|gridftp client software]] installed and in your path, it provides superior performance as compared to the native JAVA gridftp client that is provided by srmcp. In order to utilize this, download {{:public:lta-url-copy.sh.gz|lta-url-copy.sh.gz}}, unzip it and use the command:
+<file>
-  export X509_VOMS_CERT=$HOME/.globus/usercert.pem
+srmcp -use_urlcopy_script=true -urlcopy=./lta-url-copy.sh -server_mode=passive srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lofar/ops/fifotest/file1M file://`pwd`/file1M
-  export X509_VOMS_KEY=$HOME/.globus/userkey.pem
-  export X509_VOMS_DIR=/etc/grid-security/vomsdir
-  export VOMS_USERCONF=$HOME/.glite
+</file>
+**Note:**  You may have to force the use of TLS, (export GLOBUS_GSSAPI_FORCE_TLS=1 or in /etc/grid-security/gsi.conf set FORCE_TLS=true) to make this work.