Daily Image
14-11-2022
LDV gets busy
| Submitter: | Carla Baldovin |
| Description: | The LOFAR long-term archive (LTA) hosts about 60 PB of data. This is the largest astronomical data collection to date and an invaluable resource for science. Most of these data are in the form of either raw or preprocessed data, therefore far from being science-ready and useful to the largest astronomical community of non-experts. The main objectives of the LOFAR Data Valorization (LDV) project are: * to give added value to the LOFAR data in the LTA, generating higher-level data products and to make these available to the community; * to reduce the volume of the data in the archive, which will directly reduce the operational costs LDV paves the way for ASTRON's SDC by building community services and developing experience for running and offering these to the community. The LDV strategy consists of several phases; for imaging data these include compression and reprocessing with LINC, Rapthor and LOFAR VLBI pipelines. For beamformed data the processing consists of resampling, correcting and reconstructing missing metadata in the data collection catalogue. In order to allow for the re-processing of this massive archive, a new task execution workflow was developed: ATDB-LDV which can perform task management, data management, pipeline execution, and prepare the ingest to the LTA. ATDB-LDV set the basis for the SDC ATDB processing service. In the past weeks, LDV entered a pilot operational phase, in which 2967 BF datasets distributed in more than 20 science projects will be edited, amounting to an overall size of 0.96 PB. This is the first step to bring LDV operations to full-scale. On November 3rd the LDV team met at ASTRON for a busy day, in which we discussed different issues related to the operational phase and to the preparations for a request to renew the computing resources at SURF. The project has support from SURF through an NWO grant that ensures the availability of processing, storage resources and technical support. |
| Copyright: | ASTRON |
| Tweet |  |