| Both sides previous revision Previous revision Next revision | Previous revision |
| public:lta_faq [2020-11-04 15:10] – Bernard Asabere | public:lta_faq [2023-02-17 10:48] (current) – Hanno Holties |
|---|
| === What is all this SRM / 'staging' stuff about? === | === What is all this SRM / 'staging' stuff about? === |
| |
| These are technical terms that refer to the storage backend of the LTA. Each of the three LTA sites (in Amsterdam, Juelich and Groningen) operates an SRM (Storage Resource Management) system. Each SRM system consists of magnetic tape storage and hard disk storage. Both are addressed by a common file system, where each file has a specific locality: it can be either on disk ('online') or on tape ('nearline') or both. The usual case for LTA data is, that it is on tape only. Since the tape is not directly accessible but placed in a library shelf, the data on it first has to be copied from tape to disk, in order to retrieve it. This process is called 'staging'. Only while the data is (also) on disk, you will be able to download it. (In physics terms, think of it as an excited state.) To save cost, the disk pool is of limited capacity and only meant for temporary caching data that a user wants to access right now. After 7 days, all data is automatically 'released', which means that it may be deleted from the disk storage, as soon as the space is required for other data. It then has to be staged again in order to become accessible again. | These are technical terms that refer to the storage backend of the LTA. Each of the three LTA sites (in Amsterdam, Juelich, and Poznan) operates an SRM (Storage Resource Management) system. Each SRM system consists of magnetic tape storage and hard disk storage. Both are addressed by a common file system, where each file has a specific locality: it can be either on disk ('online') or on tape ('nearline') or both. The usual case for LTA data is, that it is on tape only. Since the tape is not directly accessible but placed in a library shelf, the data on it first has to be copied from tape to disk, in order to retrieve it. This process is called 'staging'. Only while the data is (also) on disk, you will be able to download it. (In physics terms, think of it as an excited state.) To save cost, the disk pool is of limited capacity and only meant for temporary caching data that a user wants to access right now. After 7 days, all data is automatically 'released', which means that it may be deleted from the disk storage, as soon as the space is required for other data. It then has to be staged again in order to become accessible again. |
| |
| Usually, you don't have to worry about the details. But be aware, that data retrieval is a two-step procedure: 1) preparation for download ('staging') and 2) the download itself. Also, take care not to request [[:public:lta_faq#what_is_an_appropriate_amount_of_data_to_retrieve|too much data at the same time.]] | Usually, you don't have to worry about the details. But be aware, that data retrieval is a two-step procedure: 1) preparation for download ('staging') and 2) the download itself. Also, take care not to request [[:public:lta_faq#what_is_an_appropriate_amount_of_data_to_retrieve|too much data at the same time.]] |
| === I did not receive a mail notification that my data is ready for retrieval! Has my request gone lost? === | === I did not receive a mail notification that my data is ready for retrieval! Has my request gone lost? === |
| |
| After you got a notification that your requests was scheduled, it is in our database and there's hardly a possibility that it got lost. Staging requests can take up to a day or two, but will finish a lot sooner in most cases. This depends on your request's size but also on how busy the storage systems are by other user's requests at the moment. Sometimes, the LTA storage systems are down for maintenance and this can delay the whole procedure. You can [[http://web.grid.sara.nl/cgi-bin/lofar.py|check for downtimes here]]. | After you got a notification that your requests was scheduled, it is in our database and there's hardly a possibility that it got lost. Staging requests can take up to a day or two, but will finish a lot sooner in most cases. This depends on your request's size but also on how busy the storage systems are by other user's requests at the moment. Sometimes, the LTA storage systems are down for maintenance and this can delay the whole procedure. You can [[https://ganglia.grid.surfsara.nl/cgi-bin/lofar.py|check for downtimes here]]. |
| |
| It is not alarming when your request did not finish in 24 hours, even when your last request finished within 10 minutes. In urgent cases or if you did not receive a notification after 48 hours, please contact the [[https://support.astron.nl/rohelpdesk|ASTRON helpdesk]]. | It is not alarming when your request did not finish in 24 hours, even when your last request finished within 10 minutes. In urgent cases or if you did not receive a notification after 48 hours, please contact the [[https://support.astron.nl/rohelpdesk|ASTRON helpdesk]]. |
| === My downloads don't start / time out === | === My downloads don't start / time out === |
| |
| Maybe the SRM system is down for maintenance, please check [[http://web.grid.sara.nl/cgi-bin/lofar.py|http://web.grid.sara.nl/cgi-bin/lofar.py]]. If there is nothing going on, there is probably something wrong with the download service. Please try again a bit later and submit a support request to the [[https://support.astron.nl/rohelpdesk|ASTRON helpdesk]], if the issue persists. | Maybe the SRM system is down for maintenance, please check [[https://ganglia.grid.surfsara.nl/cgi-bin/lofar.py|https://ganglia.grid.surfsara.nl/cgi-bin/lofar.py]]. If there is nothing going on, there is probably something wrong with the download service. Please try again a bit later and submit a support request to the [[https://support.astron.nl/rohelpdesk|ASTRON helpdesk]], if the issue persists. |
| |
| === Http downloads randomly fail with "503 Service Temporarily Unavailable" === | === Http downloads randomly fail with "503 Service Temporarily Unavailable" === |