Thanks to the careful preparation, hard work, and a bit of luck with the nice weather, the upgrade of stations from LOFAR1 to LOFAR2.0 is steadily progressing. This may sound simple, but it certainly isn’t! A lot of preparation has gone into this, and all sorts of practical issues are solved on the fly. I helped dismantling two LOFAR1 stations myself and was impressed by the positive spirit and smoothness of the process. We’re already well over halfway through, so we’re now counting down how many LOFAR1 stations are left. Many thanks to all people who are contributing to this success. I can’t wait to see the LOFAR2.0 stations being commissioned.
Happy reading!
Stress test LOFAR VLBI data reduction pipeline
The LOFAR Very Long Baseline Interferometry (VLBI) data reduction pipeline is getting a stress test! A small team is using it to post-process data from the full international array to produce sub-arcsecond resolution postage-stamp images of all sources brighter than 10 mJy across ~150 square degrees in the H-ATLAS area of the sky.
This will form the first data release of the LOFAR Two-metre Sky Survey (LoTSS) High Resolution extension (LoTSS-HR). It is an excellent way to test the production readiness of the LOFAR-VLBI pipeline, while producing a valuable legacy dataset for the community, all using data from the LOFAR Long Term Archive. Initial results are excellent, showing we can reach noise levels of 50 uJy/beam, with minimal human intervention. This work will also be used to inform the optimisation of directional dependent calibration for widefield high-resolution imaging with LOFAR.
DANTE
On December 2024, DANTE had the critical design review (CDR). The result was a ‘conditional pass’. The main concern from the panel was that at the time of the review, the team had not completed all the tests in the plan. In particular, the ones involving multiple tiles. The reason was a delay in the production of the final prototypes that led to the decision of having a CDR based on the tests available at that point, with a single tile. At the beginning of March, the team could finally install the remaining boards, station CS001 (the LOFAR2.0 prototype station) has now 4 tiles with DANTE frontends. The installation happened during the period of renovation work at CIT Groningen. This meant that there was no network for several weeks. The final tests are now ongoing, and a delta CDR will be held on May 21st. After this date, the team expects to close the development phase and deliver the design documentation for the production of the DANTE frontends for the new stations in Italy and Bulgaria.
LOFAR2.0 AIV
Rollout work in the Dutch array
LOFAR1.0 dismantling: started in January, progress is heavily influenced by weather conditions, so the pace was very low during the winter months. As of April 18, 21 out of 36 stations to be upgraded, have been stripped out the old LOFAR1.0 material. Stations CS032 and RS307 are the production test stations (PTS) and were upgraded in 2024.
LOFAR2.0 integration: electronic boards are delivered weekly to ASTRON where the antenna processing subracks are integrated by a dedicated team. Currently ~35% of the subracks have been integrated.
LOFAR2.0 installation: one of the first modifications to the stations is the installation of new network and data switches, now installed in 18 stations.
The new timing distribution system based on White Rabbit technology began to be rolled out in the Dutch array in 2024, and will not continue. The antenna processing subracks are placed in the cabinets of 14 stations so far. Important to note that at the moment, the stations are not yet started up. This will happen only in summer, when have available the Local Control Units (LCU2), only then station verification will start.
PTS handover review in March: The purpose was to review the installation and verification procedures and formally handover the stations to Operations and the commissioning team. This review helped at strengthening our confidence for the rollout.
The work in the fields of the Dutch array will continue in the months to come, while we also start making plans and preparations for the rollout of the international stations to take place mostly in 2026.
See the daily images for more impressions:
- https://www.astron.nl/dailyimage/main.php?date=20250203
- https://www.astron.nl/dailyimage/main.php?date=20250401
LOFAR Software Development
Over the past few months, the LOFAR Software Development team has focused on identifying the steps toward the Minimum Production System (MPS) and planning the work along with its expected dependencies. This will enable efficient and focused efforts toward the delivery of the MPS. Additionally, the following activities are worth mentioning:
- Implementing the ability to control the instrument through the Grafana interface, which has allowed us to demonstrate the switching of antennas on and off through the interface.
- Automatically starting QA statistics (Cobalt -> TMSS feedback).
- Providing support for various important activities such as CIT reracking, AIV, and Commissioning.
With the addition of our new team member, Danita, starting May 1st, we hope to make great progress on the MPS toward the summer.
LIFT
The implementation of the firmware and OPC-UA interface for the Transient Buffer functionality for LIFT is ongoing. A test setup has been made in the digital lab that enables us to test the implementation on a single Uniboard, which suffices at this stage of the development. We expect to have the firmware ready for testing in a LOFAR2.0 station at the end of June.
After the firmware delivery, LIFT requires additional effort in station software for control and monitoring via the Telescope Manager system. Considering the current planning and priorities, that will have to wait until early 2026, at least. Until then, we can control the system only through the OPC-UA interface, built on top of the firmware. Though this is good enough for early testing, it will definitely not be good enough for the intended production usage of the transient buffer system.
CEP
In the last six months the focus for the CEntral Processing (CEP) project has been on designing and procuring the Cobalt3 and the first installment of CEP6 hardware, as well as planning the development of the Cobalt3 software.
Cobalt3 will be the new and improved LOFAR2.0 correlator, consisting of new hardware and software. During the commissioning phase the correlation will happen on a mixture of old and new hardware, as well as a combination of an upgraded version of the Cobalt2 software (Cobalt2.1) and first prototypes of the Cobalt3 software. The Cobalt3 software development will use a Cobalt3 hardware system in-house in ASTRON, which enables the developers to test and run the software without affecting the LOFAR2.0 systems in Groningen.
In December 2024 the first discussions with vendors started on the Cobalt3 hardware, which requires high-end GPUs to meet the performance requirements. The team provided a very well-founded comparison document to support the design and final decision. The Cobalt3 correlator will consist of four nodes with three GPUs each. For redundancy an identical fifth node will be purchased to allow a quick fix in case one of the operational nodes breaks.
The infamous GraceHopper systems turned out to be sub-optimal for Cobalt3, and the choice of GPUs fell on the NVIDIA L40S. This was preferred over more recent GPU models to ensure reliability. The hardware allows for upgrades to a newer GPU model should the need arise. The purchase process has been completed, and the team is waiting for the hardware to arrive by late June.
The CEP6 cluster consists of a compute and a storage cluster. The compute cluster will run the pre-processing and data quality control pipelines, to ensure the data is ready for further processing with the LOFAR2.0 post-processing pipelines on the LTA. The storage system hosts the data from the correlator, the final data products, and intermediary data products as needed.
The CEP6 system is currently going through the procurement process, and the choice will be finalized late April. As this hardware is more standard, the delivery time is expected to be shorter, and it should also arrive around the end of June at the latest.
The philosophy of the CEP6 cluster is that it can grow as LOFAR2.0 evolves. The initial purchase consists of four compute nodes with increased memory capacity compared to CEP4, and three storage nodes. On paper this system is already equivalent to the current CEP4 system, but the demands for LOFAR2.0 will be larger. This initial system will be used to understand how the hardware performs in the LOFAR2.0 operational ecosystem. As LOFAR2.0 grows, CEP6 can also grow, and an additional purchase to double to CEP6 capacity, both compute and storage, is foreseen in the summer or shortly after.
The architecture of the CEP6 systems includes an operational and an experimental component (CEP6-OP and CEP6-EX). The operational part is not accessible for anyone except software and hardware maintenance and will be dedicated to processing supported LOFAR2.0 observations. The experimental part is available to observers from non-standard science cases that are not (yet) supported or when a user needs to run custom-made scripts. The initial CEP6 cluster this summer will be an experimental design to optimally support the commissioning.
LOFAR2.0 Commissioning
On the 28th of February, the first two actual production stations (CS032 and RS307) have been transferred to the telescope operations group. Most of the experiments the commissioning team currently does involve single station properties, and verification that all specified observation settings are properly propagating to the stations and the correlator. Emma van der Wateren e.g. demonstrated that the so-called “Fly’s Eye” mode, in which the central correlator/beamformer stores complex voltage data from all individual antenna fields simultaneously, works again, although we need a tad more integration time in the LBA. David McKenna has proven that the LOFAR2.0 station calibration procedure, based on full sky diffuse models of the Milky Way, combined with full electromagnetic models of every single dipole involved, doubles our sensitivity in the LBA compared to the old point-source-based method used for LOFAR1. Therefore, over the next few months, detailed EM simulations of every LBA field will be conducted by Michel Arts of ASTRON’s Smart Frontend group to ensure we are ready to quickly calibrate stations when the superterp is expected to be ready somewhere in July.
New LOFAR stations
The preparations for two new international LOFAR2.0 stations are in full swing. The stations will be deployed in Italy, at the Medicina Radio Astronomical Station and in Bulgaria, at the Rozhen National Astronomical Observatory.
The current focus is on the station layout design, to determine the location and rotation of the LBA and HBA antennas, cable routing and trenches, as well as the location of the processing container.
MCAP assemblies BV is working on the production of coaxial cables, which include the container cables, ground cables and HBA inter-tile cables.
Comtest Engineering delivered (picture attached) the RF-shielded containers to the Westerbork Synthesis Radio Telescope site, where all processing and network electronics will be integrated and tested.
AARTFAAC2.0
First steps are being taken to start the development of AARTFAAC2.0, the new system to replace AARTFAAC in LOFAR1. AARTFAAC2.0 will correlate all antenna signals for 12 stations, with 48 antennas, 2 polarizations per station to search for transient phenomena in LOFAR2.0 observations.
The project allows for the purchase of the additional network equipment, a dedicated GPU-based correlator and storage facilities to store 24 hours of visibility data from this system. Changes to the system include firmware changes and software changes as well, to be able to run this system in the background while LOFAR2.0 is operational. It is a project led by Antonia Rowlinson (ASTRON, UvA). At this moment we are starting up this project at ASTRON, with Arno Schoenmakers as project lead for the technical development and implementation. Early planning estimates hope to see the AARTFAAC2.0 system installed and operational early in 2027.
SEE ALSO: