public:srmclientinstallation

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
public:srmclientinstallation [2014-07-15 09:19] – [The 'vomses' file] Joern Kuensemoellerpublic:srmclientinstallation [2017-03-08 15:27] (current) – external edit 127.0.0.1
Line 39: Line 39:
 **Note** that the previouly provided ''proxy-init.sh'' script from [[http://code.google.com/p/jlite/|jLite]] is now discouraged because its encryption strength of 512 bit is not considered sufficient any more. SRM sites require a minimum strength of 1024 bit.  **Note** that the previouly provided ''proxy-init.sh'' script from [[http://code.google.com/p/jlite/|jLite]] is now discouraged because its encryption strength of 512 bit is not considered sufficient any more. SRM sites require a minimum strength of 1024 bit. 
  
-To allow usage of the LOFAR VO (Virtual Observatory), there are three additional steps to take:+To allow usage of the LOFAR VO (Virtual Organization), there are three additional steps to take:
  
   * Create a ''vomses'' file to allow ''voms-proxy-init'' to contact the relevant VOMS server   * Create a ''vomses'' file to allow ''voms-proxy-init'' to contact the relevant VOMS server
Line 50: Line 50:
 <code>"lofar" "voms.grid.sara.nl" "30019" "/O=dutchgrid/O=hosts/OU=sara.nl/CN=voms.grid.sara.nl" "lofar"</code> <code>"lofar" "voms.grid.sara.nl" "30019" "/O=dutchgrid/O=hosts/OU=sara.nl/CN=voms.grid.sara.nl" "lofar"</code>
  
-Place the file in a location where the proxy generator (voms-proxy-init) can find it (e.g. ~/.glite/vomses).+Place the file in a location where the proxy generator (voms-proxy-init) can find it (e.g. ~/.glite/vomses or export custom location to $X509_VOMSES variable).
 Alternatively, you may point voms-proxy-init to any custom location by the '-vomses' option. Alternatively, you may point voms-proxy-init to any custom location by the '-vomses' option.
 ==== VOMS server certificate chain ==== ==== VOMS server certificate chain ====
Line 64: Line 64:
 ==== Trusted grid CA certificates ==== ==== Trusted grid CA certificates ====
  
-If your system has been installed with grid software already, it is likely that the trusted CA certificates can be found in the default directory ''/etc/grid-security/certificates''. If this is not the case, or you receive error messages related to not being able to determine the authenticity of a certificate, you can retrieve the latest certificates, e.g. from [[https://dist.eugridpma.info/distribution/igtf/current/accredited/igtf-preinstalled-bundle-classic.tar.gz|EUGridPMA]], and place them in a directory of your choosing.+If your system has been installed with grid software already, it is likely that the trusted CA certificates can be found in the default directory ''/etc/grid-security/certificates''. If this is not the case, or you receive error messages related to not being able to determine the authenticity of a certificate, you can retrieve the latest certificates, e.g. from [[https://dist.eugridpma.info/distribution/igtf/current/accredited/igtf-preinstalled-bundle-classic.tar.gz|EUGridPMA]], and place them in a directory of your choosing (set environment variable X509_CERT_DIR accordingly). If you use our prepackaged tarball, you can use the provided script 'update_certificates.sh' (for bash; use the .csh version for c-shell) to deploy the latest certificates (Note: The script requires the lynx text browser to be installed on your system).
  
 ===== SRM client tools ===== ===== SRM client tools =====
Line 72: Line 72:
 **NB If the client srm copy call returns timeout messages, the most likely cause is that a firewall is blocking outward connections. The following ports are typically needed by the srm client tools: 8443/8444, 2811, and any port in the gridftp port range (typically in the range 20000 - 25000). Note that these ports are configured on the server side so this list may not be complete for all situations. If at all possible, it is advisable to configure the firewall to allow all outward connections. The next best option is to allow all outward connections to the domains that provide LOFAR LTA services (currently grid.sara.nl, fz-juelich.de, and target.rug.nl)** **NB If the client srm copy call returns timeout messages, the most likely cause is that a firewall is blocking outward connections. The following ports are typically needed by the srm client tools: 8443/8444, 2811, and any port in the gridftp port range (typically in the range 20000 - 25000). Note that these ports are configured on the server side so this list may not be complete for all situations. If at all possible, it is advisable to configure the firewall to allow all outward connections. The next best option is to allow all outward connections to the domains that provide LOFAR LTA services (currently grid.sara.nl, fz-juelich.de, and target.rug.nl)**
  
-**NB2 If you want to enable 'active' transfers, firewalls should allow incoming access to the ports configured as the globus port range (look in client documentation for more details). This type of transfer can improve performance of a single transfer as it will use multiple parallel connections for retrieving a file. For the FNAL/dCache client, 'active' transfers are initiated if the ''-server_mode=passive'' setting is omitted. For the Berkely client, parallel transfers will be initiated when the ''-parallelism'' parameter is set to a value larger than 1. Since in most cases LOFAR datasets consist of a large number of files, a similar performance improvement can be achieved by splitting the set of files over multiple srm copy processes.** 
  
 ==== FNAL/dCache client tools ==== ==== FNAL/dCache client tools ====
Line 78: Line 77:
 We provide a slightly modified package in two different versions: {{srmclient-2.6.28.tar.gz|srm client tools (Java 7)}} (or alternatively {{srmclient-2.2.25.tar.gz|srm client tools (Java 6)}}). (Note that both are also part of the prepackaged tarball together with an init script to set up the environment.) These contain updated scripts that allow installation in any directory. The client can be unpacked anywhere after which the ''$SRM_PATH'' environment variable should be set to the root directory of the unpacked tarball. Additionally, the ''$SRM_PATH/bin'' directory should be added to ''$PATH'' and [[#trusted_ca_certificates|trusted CA certificates]] must be installed. Finally, a valid proxy must be available in the default grid location (''/tmp/x509up_u<UID>'') or in the location configured as ''$X509_USER_PROXY''. You should now be able to retrieve a file from the LOFAR LTA: We provide a slightly modified package in two different versions: {{srmclient-2.6.28.tar.gz|srm client tools (Java 7)}} (or alternatively {{srmclient-2.2.25.tar.gz|srm client tools (Java 6)}}). (Note that both are also part of the prepackaged tarball together with an init script to set up the environment.) These contain updated scripts that allow installation in any directory. The client can be unpacked anywhere after which the ''$SRM_PATH'' environment variable should be set to the root directory of the unpacked tarball. Additionally, the ''$SRM_PATH/bin'' directory should be added to ''$PATH'' and [[#trusted_ca_certificates|trusted CA certificates]] must be installed. Finally, a valid proxy must be available in the default grid location (''/tmp/x509up_u<UID>'') or in the location configured as ''$X509_USER_PROXY''. You should now be able to retrieve a file from the LOFAR LTA:
 <code> <code>
-srmcp -server_mode=passive srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lofar/ops/fifotest/file1M file:///`pwd`/file1M+srmcp -server_mode=passive srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lofar/ops/fifotest/file1M file:///file1M
 </code> </code>
 +
 +If your firewall allows incoming connections to non-standard ports, you can try [[#active_gridftp|active gridftp]], which will enable utilization of multiple streams to increase performance.
 +
 +If you have the [[public:grid_srm_software_installation#globus_client_software|gridftp client software]] installed (requires installation with root privileges) and in your path, it provides superior performance as compared to the native JAVA gridftp client that is provided by srmcp. In order to utilize this, download {{:public:lta-url-copy.sh.gz|lta-url-copy.sh.gz}}, unzip it and use the command:
 +<file>
 +srmcp -use_urlcopy_script=true -urlcopy=./lta-url-copy.sh -server_mode=passive srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lofar/ops/fifotest/file1M file:///`pwd`/file1M
 +</file>
 +
 +**Note:** Usually, the url-copy script requires an absolute destination path. Make sure that you don't use the one from other sources, but the modified script from this page (named //lta-url-copy.sh//, now also included in the tarball below) to retrieve data with the //srm.txt// files from your notification mails. 
 +
 +
 +=== Active gridftp ===
 +
 +In the examples above, srmcp is run with the option -server_mode=passive, which limits the transfer to a single stream. If you want to enable 'active' transfers, your firewall has to allow **incoming** access to the ports configured as the globus port range (typically ports 20000-25000, also open 8443, 8444, 2811). 
 +The IP ranges for remote gridftp servers that need to be able to connect to your machine are:
 +
 +  * 145.100.32.0/22, i.e. 145.100.32.0 to 145.100.35.255, for SURFsara 
 +  * 134.94.32.0/22, i.e. 134.94.32.0 to 134.94.32.255, for FZJuelich. 
 +
 +Active gridftp can improve performance of a single transfer as it will use multiple parallel connections for retrieving a file. For the FNAL/dCache client, 'active' transfers are initiated if the ''-server_mode=passive'' setting is omitted. For the Berkely client, parallel transfers will be initiated when the ''-parallelism'' parameter is set to a value larger than 1. Since most cases LOFAR datasets consist of a large number of files, a similar performance improvement can be achieved by splitting the set of files over multiple srm copy processes. This is usually easier to set up than the firewall requirements. Note that e.g. the dCache client does not have a default setting for the gridftp port range. Further, srmcp ignores the GLOBUS_TCP_PORT_RANGE environment variable. You have to specify the port range (that you opened in your firewall) via the ''globus_tcp_prt_range'' option of srmcp, e.g.:
 +
 +
 +  srmcp -globus_tcp_port_range=20000,25000 srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lofar/ops/fifotest/file1M file:///file1M 
  
  
Line 85: Line 107:
  
 We provide the {{lofar_grid_clients.tar.gz|Lofar Grid Clients}} tarball that comes prepackaged with the above (except your personal grid certificate of course) to allow easy access to the LOFAR VO. We provide the {{lofar_grid_clients.tar.gz|Lofar Grid Clients}} tarball that comes prepackaged with the above (except your personal grid certificate of course) to allow easy access to the LOFAR VO.
-Extract the tarball to a directory of your liking and source 'init.sh' if you use Java 7 or 'init_java6.sh' if you use Java 6 to set up the environment. +Extract the tarball to a directory of your liking and source 'init.sh' if you use Java 7 (or newer) or source 'init_java6.sh' if you still use Java 6. This sets up the environment for you
-You can ''voms-proxy-init'' for generating a proxy and ''voms-proxy-info'' for inspecting the generated proxy. +You can ''voms-proxy-init'' for generating a proxy and ''voms-proxy-info'' for inspecting the generated proxy. \\ 
 +It needs (semi-)regular updates of the certificates, with one of the supplied scripts.
  
 ===== Walkthrough ===== ===== Walkthrough =====
Line 98: Line 121:
   * Untar package in directory of your choosing:\\ <code>tar -xvzf lofar_grid_clients.tar.gz</code>   * Untar package in directory of your choosing:\\ <code>tar -xvzf lofar_grid_clients.tar.gz</code>
   * Determine your java version:\\ <code>java -version</code>   * Determine your java version:\\ <code>java -version</code>
-  * Source init.sh (Java 7) or init_java6 (Java 6) in lofar_grid/, e.g. :\\ <code>. lofar_grid/init.sh</code>+  * Source init.sh (Java 7 or 8) or init_java6 (Java 6) in lofar_grid/, e.g. :\\ <code>. lofar_grid/init.sh</code> 
 +  * Update the certificates with one of the provided scripts, e.g.: \\ <code>. update_certificates_eugridpma.sh</code>
   * Optional: Set proxy environment variable to custom location:\\ <code>export X509_USER_PROXY=<proxy_location></code>   * Optional: Set proxy environment variable to custom location:\\ <code>export X509_USER_PROXY=<proxy_location></code>
   * Generate a proxy:\\ <code>voms-proxy-init -voms lofar:/lofar/user</code>   * Generate a proxy:\\ <code>voms-proxy-init -voms lofar:/lofar/user</code>
-  * Test data retrieval:\\ <code>srmcp -server_mode=passive srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lofar/ops/fifotest/file1M file:///file1M</code> +  * Test data retrieval:\\ <code>srmcp -server_mode=passive srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lofar/ops/fifotest/file1M file://`pwd`/file1M</code> 
-  * Done!// NB If you modified any default location by the ''export'' command, you have to put it in a shell start-up script like '.bashrc' to make your changes permanent, of course (with full paths where appropriate).//+  * Done!\\ // NB If you modified any default location by the ''export'' command, you have to put it in a shell start-up script like '.bashrc' to make your changes permanent, of course (with full paths where appropriate).// 
 +  * If you get any errors <del>related to CA certificates</del>, retry after running one of the provided scripts to update your certificates, e.g.\\ <code>. update_certificates_eugridpma.sh</code>  The certificates change every now and then, and then you need to update them.
  
-**Note:** For (t)csh, use *.csh init scripts and 'setenv <key> <value>' instead of 'export <key>=<value>'+**Note:** For (t)csh, use *.csh init scripts and 'setenv <key> <value>' instead of 'export <key>=<value>'.
  
 ====== Troubleshoot ===== ====== Troubleshoot =====
  
-  * OpenJDK 7 seems not to be capable of dealing with the certificate, make sure to run Java provided by Oracle +  * There is a [[public:lta_faq|LTA FAQ page]] that should help with the common difficulties. 
-  * Maybe your private key uses an unsupported algorithm. You might want to try the following command: +
-<code> +
-openssl rsa -des3 -in .globus/userkey.pem -out .globus/userkey.pem +
-</code> +
-  *  Error: //org.glite.voms.contact.VOMSException: AC validation failed!//  +
-    * Have you registered at the Lofar VO? --> https://voms.grid.sara.nl:8443/voms/lofar +
-    * You need to have the certificate installed in your browser for this ([[http://ca.dutchgrid.nl/info/browser]])+
  • Last modified: 2014-07-15 09:19
  • by Joern Kuensemoeller