public:lta_howto

This is an old revision of the document!


Long Term Archive Howto

This is a short manual on how to search for and retrieve data from the Long Term Archive.

To access the LTA you need to have an account in MoM that is enabled for the archive.

  1. This automatically happens if you were a member of the original project proposal in MoM.
  2. Otherwise Science Support needs to add you to the project to which you need access.
  3. For public data you can use an anonymous account.

You might get an email asking you to set a new password in ASTRON Web Applications Password Self Service. This will set not just a new password for the LTA but for MoM (LOFAR/WSRT) and Northstar as well.

Once your account is set up, you can navigate to LTA Catalog site Please note that this address might change before full operations in December 2012.

Currently you can only search the LTA catalog per project. This means you need so select a project first.

After this you can either use

  1. The Search screen which allows you to search by RA/Dec, ObservationId, Frequency, etc.
  2. The Show Latest screen which shows you the most recently added data for this project ordered in several ways.

The result of either query will be a list of data products similar to this:

Once you have a list of dataproducts, you can use the check boxes to select which files you want to download. The first check box can be used to select or deselect all files on the page.

When you have made your selection of files, you click on stage. This show you the following message. It means that a request has been sent to the LTA staging service to start retrieving the requested files from tape storage and make them available. You will get an e-mail when this tape retrieval is complete.

The e-mail that you get when the tape retrieval is complete gives you a list of files and has two attachments, html.txt and srm.txt:

There are two ways you can use this list to retrieve the files: http and srm

If you open http.txt this file contains a list of http links that you can feed to a unix commandline tool like wget or curl or even use in a browser. For wget you can use the following command line:

wget -i http.txt

This will download the files in http.txt to the current directory. Do not set the username and password on the wget command line because this allows other users on the system to view them in the process list. Instead you should create a file ~/.wgetrc with two lines according to the following example:

user=lofaruser
password=secret

If you open the file srm.txt this file contains a list of srm locations which you would feed to srmcp. SRM is a GRID specific protocol that is currently supported for data at the SARA and Jülich locations. It is faster, especially if you have significantly more than 1 GB/s bandwidth. It requires a fully fledged GRID account and access and installation of the GRID srm software. Contact Science Support if you think you might need a GRID account but it can not be provided by your own institute. An example command line would be:

srmcp --server_mode=PASSIVE srm://lofar-srm.juelich.de:8443/pnfs/fz-jeulich.de/data/lofar/ops/projects/commissioning2012/file.tar file://///data/files/file.tar

You only need –server_mode=PASSIVE if you are behind a firewall or on an internal network. This will result in a slower transfer speed as it will only use one instead of multiple streams.

  • Last modified: 2012-12-18 15:23
  • by Adriaan Renting