CS1 Data storage and archiving policy

16/07/2007, H.A. Holties

As for the future LOFAR system, the CS1 storage capacity is too small to store all data in a central location indefinitely. Moreover, the central storage location needs to provide storage not only for observation datasets, but also for (temporary) commissioning and analysis data. The policy below has been defined to allow efficient use of the offline storage cluster while providing as much support to people involved in the commissioning of LOAR CS1 as possible.

NB: The policy as provided here applies principally to the data in the “offline” domain. In this domain, the main central storage facility consists of twelve storage systems, each with a storage capacity of 1.2 Tbyte. A more complete description of the CS1 offline systems can be found here.

The datasets created with the LOFAR CS1 system are copied to the offline storage cluster after observations have finished. The available datasets and their locations can be found in the observation catalog. Data is kept on the central storage system for a period of at least a week. After the retention period, the data is either deleted or archived, depending on its status in the “Archive” column in the observation catalog. The oldest datasets will be (re) moved first. The exact time at which deletion or archiving is carried out will, amongst other things, depend on the storage capacity required for near future observations and commissioning activities.

Archived data will be moved to GRID based storage servers. Commissioning data will be owned by the /lofar/ops group and stored under the lfc directory /grid/lofar/ops/cs1_data as a tarball. The current policy for this data is that it will be archived on tape in two copies after an unknown period. Please beware that GRID based storage is experimental. At present, no guarantees can be given about data lifetime and retrieval although these issues are being addressed and data security is likely orders of magnitude better than on the central CS1 storage facilities.

To allow commissioning analysis of datasets and testing of offline processing software, users may request usage of central strorage capacity. Requests should be directed to the observer on duty (observer at aston dot nl). The request should be accompanied by the capacity needed. Further information may be requested (such as motivation, expected period for which storage capacity is needed, etc).

Operations personnel will confirm the allocated storage capacity and the period for which it can be used. If demanded so for operational or commissioning purposes, operations personnel can ask users to (re) move their data. An overview of data capacity allocated to users is given in the following table:

NB1: The CS1 offline storage servers provide storage on “best effort” basis. Data is stored on disks in a RAID striping configuration (for optimal performance) and is not backupped. Any data that is important enough to keep for any period of time should be copied to other systems that provide better data security.

NB2: There is no software used for accounting of storage capacity. Please check the used capacity yourself. Manual checks will be carried out by operations personnel regularly, in particular in situations where the available storage capacity becomes too small to allow planned operations and commissioning. Offenders will be notified and prompt action is expected in that case. In case there is no response, corrective actions may be taken by operations personnel including, ultimately, removal of data. Offending users may be “named and shamed”.