ASTRON/JIVE Daily Image

© LDV team

The LOFAR Data Valorisation (LDV) project aims to give added value to LOFAR archived data in the Long Term Archive (LTA) by offering users well curated, science-ready data. It also paves the way for LOFAR2.0 operations because it streamlines data processing operations at the LTA sites (currently Surf, Juelich and PSNC) and ensure sustainability of science operations by reducing data volume at the LTA, thereby also lowering the operational costs.

LDV is reaching the conclusion of its first phase of operations at Surf. The storage occupancy was reduced by 7.5 PB through resampling of beam formed data, compression of interferometric data and a large data placement action. This gain in storage translates in a saving of about 56 kEuro/yr. Curation of data enabled the removal of duplicates and the addition of quality flags. Currently, preparations are ongoing to perform a major retirement action (~9.5 PB distributed over the three LTA sites) of the early cycles (0-6) archived data with no legacy value.

The next operational aim at Surf is the generation of higher level data products by running the LOFAR Initial Calibration (LINC) pipeline on a significant portion of the archived data. For the first time, calibrated visibilities through the observatory processing framework with latest workflows will be offered to the astronomical community.

Furthermore efforts are ongoing to extend the processing framework infrastructure at the Juelich LTA site, for enabling the next round of data editing operations with an expected duration of 1 year. As a result more than 4 PB of tape storage space will be gained. Looking ahead, the LDV processing infrastructure will be expanded to support the LOFAR2.0 MPS operations, marking a key step toward future-ready science operations.