Daily Image
28-10-2015
LOBOS mini survey - challenging the LOFAR system
| Submitter: | Emanuela Orru' |
| Description: | LOBOS is a LOFAR commissioning project (PI N. Jackson) meant to search for calibrator sources for the HBA observations using the International stations. It is a mini survey that plans to cover in a relatively short amount of time (96 hours) approximately 40000 pointings. The amount of data products archived in the LTA will reach about 1.5 million. This project has challenged the LOFAR system in various aspects. An ad-hoc script was developed to automatically generate the settings for observations and processing in order to optimise the telescope time. This resulted in a single file containing all the settings for 24 hours of observation and proved too big to be handled by the specification system. Therefore it was split into separate blocks of 6 hours each. The observations were scheduled and processed without particular problems, but the process of ingesting the data into the LTA was more difficult. The top left corner of the image shows a screenshot of the one LOBOS 24 hours run as seen by the Control Room Scheduling system. The LOBOS data products are made of many small files, which adds substantial overheads to the ingest process. The ingest job could not be initiated once per block due to a safety limitation of about 50.000 files per ingest. This meant that each 6-hour block had to be further split into at least 4 blocks. Since the LTA-LOFAR system often experiences instability over long periods of time due to maintenance, failing network connections or other spurious problems, the ingest process required way more than 4 attempts to finish the ingest. The right part of the image shows a screenshot of Mom with the ingest tentatives. The data of only 48 hours represent a significant fraction of the small size LTA data products if we consider that this a single project (96 hours) while the rest includes the total amount of data products present in the LTA since the beginning of LOFAR Cycle0 (from December 2012 until now). In the bottom left corner of the image, two istograms present the total number of the LTA data products before (left panel) and after LOBOS (right panel). |
| Copyright: | LOFAR Radio Observatory |
| Tweet |  |