public:lta_howto

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
public:lta_howto [2020-10-18 19:39] – Marco Iacobellipublic:lta_howto [2025-01-17 11:00] (current) – [Change of account registration method] Hanno Holties
Line 13: Line 13:
  
 ====== User Access ====== ====== User Access ======
 +
 +==== Change of account registration method ====
 +
 +:!: We are in the process of upgrading the LOFAR applications. The services that have been offered through [[https://lofar.astron.nl|https://lofar.astron.nl]] are no longer available for public use. Please contact the [[https://support.astron.nl/sdchelpdesk|SDC Helpdesk]] in case you have a question or need help. Specifically: new accounts and account changes, for the time being, need to be requested through the helpdesk. The password self-service documented below is still active. :!:
  
 === I forgot my password === === I forgot my password ===
  
-Please visit [[https://webportal.astron.nl/pwm/public/ForgottenPassword|https://webportal.astron.nl/pwm/public/ForgottenPassword]] to request a new one.\\+Please visit [[https://webportal.astron.nl/pwm/public/ForgottenPassword|https://webportal.astron.nl/pwm/public/ForgottenPassword]] to request a new one. 
 === Searching / retrieving data === === Searching / retrieving data ===
  
 The LTA catalogue can be searched directly without needing any account. Access to all projects and search queries will return results of the entire catalogue because metadata are public for all LTA content. The LTA catalogue can be searched directly without needing any account. Access to all projects and search queries will return results of the entire catalogue because metadata are public for all LTA content.
  
-Staging and subsequent downloading of __**public**__  data always requires an account with LTA user access privileges. This automatically happens if you were a member of the original project proposal in Northstar/MoM. If you do not have an account yet, you can register with "[[https://lofar.astron.nl/useradministration/public/setUpUserAccount.do|Create account]]". LTA access privileges will need to be granted by Science Data  Center Operations (SDCO). Please, __once you have created your account, submit a support request to the [[https://support.astron.nl/rohelpdesk|Radio Observatory helpdesk]] asking for LTA privileges stating clearly your username__.+Staging and subsequent downloading of __**public**__  data always requires an account with LTA user access privileges. This automatically happens if you were a member of the original project proposal in Northstar/MoM. If you do not have an account yet, you should contact the [[https://support.astron.nl/sdchelpdesk|SDC Helpdesk]]. LTA access privileges will need to be granted by Science Data Centre Operations (SDCO). Please, __submit a support request to the [[https://support.astron.nl/rohelpdesk|ASTRON helpdesk]] asking for LTA privileges__.
  
 To stage and retrieve project-related data in the LTA which are __**proprietary**__  you need to have an account in [[https://lofar.astron.nl/mom3|MoM]] that is enabled for the archive and coupled with the projects of interest. To this aim you can request SDCO to be added to the list of co-authors of the project. When you send such a request, you must add the project's PI in cc. After SDCO adds you to the project, you might get an email asking you to set a new password in [[https://webportal.astron.nl/pwm/private/Login|ASTRON Web Applications Password Self Service]]. Please note that this will set a new password not just for the LTA //but for MoM (LOFAR/WSRT) and Northstar as well//. To stage and retrieve project-related data in the LTA which are __**proprietary**__  you need to have an account in [[https://lofar.astron.nl/mom3|MoM]] that is enabled for the archive and coupled with the projects of interest. To this aim you can request SDCO to be added to the list of co-authors of the project. When you send such a request, you must add the project's PI in cc. After SDCO adds you to the project, you might get an email asking you to set a new password in [[https://webportal.astron.nl/pwm/private/Login|ASTRON Web Applications Password Self Service]]. Please note that this will set a new password not just for the LTA //but for MoM (LOFAR/WSRT) and Northstar as well//.
  
-Please read the [[https://old.astron.nl/radio-observatory/observing-proposals/lofar-data-policy/lofar-data-policy|LOFAR Data Policy]] for more information about //proprietary// vs //public// data.\\+Please read the [[https://old.astron.nl/radio-observatory/observing-proposals/lofar-data-policy/lofar-data-policy|LOFAR Data Policy]] for more information about //proprietary// vs //public// data.
 === Step-by-step guide to search and retrieve data === === Step-by-step guide to search and retrieve data ===
  
Line 55: Line 60:
       * use the 'show data' button to select the project and to show all data in it       * use the 'show data' button to select the project and to show all data in it
   * from the screen that follows, you should be able to either search / select / stage the data products   * from the screen that follows, you should be able to either search / select / stage the data products
 +
  
 ====== How to find data in the archive ====== ====== How to find data in the archive ======
Line 123: Line 129:
   - On a 1 Gbit/s connection as a general rule of thumb, you should be able to retrieve data at about 100-500 GB/hour, especially if you try to retrieve 4-8 files concurrently. If you see speeds much lower than this, you might have some kind of network problem and should in general contact your IT staff.   - On a 1 Gbit/s connection as a general rule of thumb, you should be able to retrieve data at about 100-500 GB/hour, especially if you try to retrieve 4-8 files concurrently. If you see speeds much lower than this, you might have some kind of network problem and should in general contact your IT staff.
   - Staging the data from tape to disk might take quite a bit of time. In the large data centres that the LTA uses, the tape drives are shared with all users and requests are queued. This is not just users of LOFAR but large data other projects like the LHC. This might mean that it takes anywhere from a few hours to a day or more to stage a copy of your data from tape to disk.   - Staging the data from tape to disk might take quite a bit of time. In the large data centres that the LTA uses, the tape drives are shared with all users and requests are queued. This is not just users of LOFAR but large data other projects like the LHC. This might mean that it takes anywhere from a few hours to a day or more to stage a copy of your data from tape to disk.
-  - The amount of space available for staging data is limited although quite large. This space is however shared between all LOFAR LTA users. This includes LTA operations for buffering data from CEP to the LTA before it gets moved to tape. If many users are staging data at the same time, and/or SDCO operations is transferring large amounts of data, the system might temporarily run low on disk space. You might then get a message that your request was only partially successful. In general the request will still finish 1-2 days later and we do monitor if requests don't get stuck and restart if needed.+  - The amount of space available for staging data is limited although quite large. This space is however shared between all LOFAR LTA users. This includes LTA operations for buffering data from CEP to the LTA before it gets moved to tape. If many users are staging data at the same time, and/or SDC operations is transferring large amounts of data, the system might temporarily run low on disk space. You might then get a message that your request was only partially successful. In general the request will still finish 1-2 days later and we do monitor if requests don't get stuck and restart if needed.
   - We strive to keep a copy of data that was staged on disk for 1-2 weeks so you have some time to download it. After that it might get removed to make space for more recent requests. The copy of the data on tape is only read and will still be available if you need to access the data again at a later stage but you might need to stage a copy to disk again.   - We strive to keep a copy of data that was staged on disk for 1-2 weeks so you have some time to download it. After that it might get removed to make space for more recent requests. The copy of the data on tape is only read and will still be available if you need to access the data again at a later stage but you might need to stage a copy to disk again.
   - We are continuously trying to improve the reliability and speed of the available services. Please contact SDCO if you have any problems or suggestions for improvement.   - We are continuously trying to improve the reliability and speed of the available services. Please contact SDCO if you have any problems or suggestions for improvement.
Line 130: Line 136:
 ==== Staging Transient Buffer Board (TBB) data ==== ==== Staging Transient Buffer Board (TBB) data ====
  
-TBB data needs to be staged by hand. Please send a request at support.astron.nl/rohelpdesk to stage the data for you, specifying the filenames to be staged. To download the data, please follow the instruction under Download Data for proper authentication. Data will then be available for download using+TBB data needs to be staged by hand. Please send a request at [[https://support.astron.nl/rohelpdesk|https://support.astron.nl/rohelpdesk]] to stage the data for you, specifying the filenames to be staged. To download the data, please follow the instruction under Download Data for proper authentication. Data will then be available for download using
 <file> <file>
  
 wget --no-check-certificate https://lofar-download.grid.surfsara.nl/lofigrid/SRMFifoGet.py?surl=<filename> . wget --no-check-certificate https://lofar-download.grid.surfsara.nl/lofigrid/SRMFifoGet.py?surl=<filename> .
 +
 +The filename should start or be prepended with srm://
  
 </file> </file>
  
-You will need a valid LTA account to access this data.+You will need a valid LTA account to access this data. If the filename is very short, you can view (e.g. cat) it to view errors that have occured.
  
 ===== Download data ===== ===== Download data =====
Line 277: Line 285:
 ==== SRM download ==== ==== SRM download ====
  
-If you open the file ''srm.txt''  this file contains a list of srm locations which you would feed to ''srmcp''. SRM is a GRID specific protocol that is currently supported for data at the SARA and Jülich locations. It is faster, especially if you have significantly more than 1 Gbit/s bandwidth. It requires a valid [[:public:grid_certificate|GRID certificate]] and installation of the [[:public:grid_srm_software_installation|GRID srm software]]. NB There is an [[:public:srmclientinstallation|alternative installation that does not require root privileges]]. Contact Science Operations and Support if you think you might need a GRID account but it can not be provided by your own institute. An example command line would be:+If you open the file ''srm.txt''  this file contains a list of srm locations which you would feed to ''srmcp''. SRM is a GRID specific protocol that is currently supported for data at the SARA and Jülich locations. It is faster, especially if you have significantly more than 1 Gbit/s bandwidth. It requires a valid [[:public:grid_certificate|GRID certificate]] and installation of the [[:public:grid_srm_software_installation|GRID srm software]]. NB There is an [[:public:srmclientinstallation|alternative installation that does not require root privileges]]. Contact SDC Operations via __[[https://support.astron.nl/rohelpdesk|ASTRON helpdesk ]]__if you think you might need a GRID account but it can not be provided by your own institute. An example command line would be:
 <code> <code>
  
Line 298: Line 306:
  
   * There is a [[:public:lta_faq|LTA FAQ page]] that should help with the common difficulties.   * There is a [[:public:lta_faq|LTA FAQ page]] that should help with the common difficulties.
- 
-\\ 
  
  
  • Last modified: 2020-10-18 19:39
  • by Marco Iacobelli