Differences

This shows you the differences between two versions of the page.

--- public:cs1_storage [2007-07-16 08:43] – Hanno Holties
+++ public:cs1_storage [2010-03-09 14:02] (current) – removed Arno Schoenmakers
@@ Line 1: / Line 1: @@
-====== Retrieving CS1 data ======
-So where does all the data go?
-The diagram below shows the, for users, relevant systems for the CS1 data streams.
-{{public:cs1_user_data_systems.gif?800|}}
-First, it is important to realize that there is no such thing as THE data. There is observation data, monitoring data, configuration data, etc. Next, it is also important to be aware that, just as for the future "normal" LOFAR operations, the bulk of the data will not be stored in a single central location indefinitely. Most data will be stored centrally for a limitted period after which it (or at least a selection) will be archived elsewhere.
-===== Observation datasets =====
-To find the most interesting datasets, check the [[public:dataproduct:catalog_of_observation|Observation catalog]]. The "MS-name" column provides the name of the dataset. The "stored" column provides information on where the associated datasets can be found.
-For example: "lifs001 - lifs003, lifs005" in the location columnn indicates that the measurement is to be found in parts divided over four of the offline storage nodes. On each node you will find the datasets in the ''/data/<MS-name>'' directory. Note that the /data directories are mounted on all offline processing servers, as well as on the listfen frontend system. It is therefore in practice not necessary (and even not recommended) to actually log in to one of the storage nodes!
-An entry "archived" indicates that the dataset has moved to long term storage on the GRID where it is located at <TBD>. Check the [[engineering:software:grid_ui|CS1 GRID UI page]] for information on how to retrieve data from the GRID. NB: in general, datasets should not be transfered from the GRID based archive back to the LOFAR cluster. Rather, transfer data to your own system or analyse it using GRID based processing resources.
-===== Station (auto) correlation data =====
-Station data is copied (manually) from the station SCU's to a machine (lofar15) in Dwingeloo. The oldest datasets are archived, as a temporary solution, on adam. Contact Rob Millenaar or Hanno Holties for details. The intention for the future is to create and store station data (at least as long as is deemed necessary for commissioning purposes) together with the observation datasets.
-===== Configuration data =====
-The intention is to store all system configuration (a.k.a. "parset") files together with the observations (TBD). These files are used throughout the system to configure observation processes. The files are generated manually or, in the near future, automatically bu the SAS/MAC system and distributed to the appropriate processes.
-===== CS1 Data storage and archiving policy =====
-The CS1 data storage and archiving policy is provided [[CS1_data_polic"|here]].