LOFAR Development Newsletter October 2024

Jurjen de Jong

Producing ultra-deep, high angular resolution wide-field images with LOFAR presents significant computational challenges. In particular, the imaging process is so resource-intensive that handling datasets accumulated over hundreds of hours on a single pointing is infeasible with current imagers.

We are working on a solution that has been considered in the past but is not implemented in most imaging pipelines: sidereal visibility averaging. This technique merges multiple individual observations into a much smaller dataset by averaging visibilities at similar baseline coordinates.

Our findings show that for creating ultra-deep sub-arcsecond resolution images with LOFAR, this method offers substantial imaging speed improvements. For example, the proposed LUDO project, which involves imaging 3000 hours of observing time on a single field with LOFAR, could be created at least 75 times faster using this technique.

Carla Baldovin and DANTE team

DANTE is now in its final phase: the design qualification, which goal is to provide proof that the current design can be manufactured on a large scale.

A series of 80 HBAFE are being produced by an external manufacturer. Production started during the summer and the first batch of 20 prototypes were received at ASTRON at the end of August. The boards were tested in the laboratory, and as a result some modifications were implemented in-house. The next step was to install 16 of them in one HBA tile in CS001 on September 12 and test them for a period of 1 week. After this, the production of the remaining 60 boards will follow. The graphs show the first results from the measurements in the files: the sensitivity of the DANTE frontend (black line) is compared to the one of LOFAR1. The DANTE HBAFEs have a sensitivity comparable LOFAR1 up to ~140 MHz, and higher to LOFAR1 beyond ~150 MHz. These encouraging results show already the improvement achieved with the new HBA frontends.

Carla Baldovin

The current phase in AIV is PTS (Production Test Stations); consisting of the upgrade of 2 stations with hardware manufactured in a first production run. The goal is to test the production and the procedures for upgrading the stations. In addition, the PTS stations are equipped with White Rabbit switches for the timing distribution and new network switches to handle the higher data rate from observing with HBA and LBA in parallel.

The key accomplishments during the past period are listed:

• Dismantling of the LOFAR1 hardware in the stations CS032 and RS307.
• Assembly, integration, and testing of the Antenna processing subracks in Westerbork.
• Installation of LOFAR2.0 hardware in CS032 (July) and first all-sky image (see picture).
• Kickoff meeting for the PTS verification phase to align the goals and objectives of the verification, discuss the process and responsibilities during this phase.
• During a retrospective session we reviewed the process of the upgrade in CS032, identifying the points that worked well, and the ones that can be improved for the next stations.
• Installation of RS307 (August).
• CS032 verification completed, RS307 verification started but has been interrupted due to the network upgrade work ongoing during September.
• The design of the LOFAR2.0 products is released for production. During testing at the station, some of the issues found can be directly linked to a specific product, e.g., UniBoard2, RCU, etc. For each of these products a review meeting is organized where issues are assessed, considering their severity and impact. Only when the team is confident that for a specific product the issues are minor, not related to hardware, or a mitigation measure is identified, the design is released for production. This means that the team will communicate to the manufacturer that the production (at large scale) of that component can start. Up to now the following products have been released:
  • UniBoards 2 (production started in August)
  • Receiver Unit (high and low)
  • Mid-plane
  • Antenna processing subrack (APS) Fan-box

The design of the cabinet clock distribution box (CDD), the APS clock and translator (APSCT) and the ASP mechanical housing are not yet released for production, but we expect them to be released in the coming weeks.

PTS is a crucial step towards the completion of the LOFAR2.0 upgrade and it is only possible thanks to the collaborative work of a multidisciplinary team of dedicated professionals, all of them committed to deliver a great instrument to our science community. A big thanks to everyone involved in this project and congratulations for making it a success!

Arno Schoenmakers

TMSS

Development on the new proposal tool, codenamed LUDWIG, has started. To facilitate a seamless integration between Ludwig and TMSS, we are contributing effort to develop the interface between Ludwig and TMSS and contribute to the actual frontend development of Ludwig by allowing our frontend developer to work for the majority of his time in the team developing Ludwig. As a result, frontend work in TMSS has slowed down quite a bit.

Despite that we have delivered the improved QA workflow that is used by the Operators to communicate quality of the acquired data to the PIs of LOFAR projects. Also, we improved the reporting of LOFAR cycles, primarily for stakeholders such as the (former) ILT board. We added some plots to the dynamic scheduler to give more insight in the outcome of the scheduler.

Meanwhile we also did some package updates, both in the frontend and in the backend. For many reasons it is important to keep updated so that we use the most recent libraries. TMSS, as a web-enabled application, has many dependencies on 3^rd party packages, so this is a continuing task for the team.

With LOFAR1 Cycle 20 having come to an end on June 1st, we can now fully focus on the needs and changes for LOFAR2.0. For example, specifically for LOFAR2.0, we added the automatic creation and administration of station statistics data products (sub-band statistics SST, cross-correlation statistics XST and beamlet statistics BST) to TMSS, which allows our commissioners to specify and handle these data products in TMSS. Also, we added the ability to specify dithering by the RCU2s to TMSS. In the coming months, we will focus on the support of LOFAR2.0 commissioning needs.

LOFAR2.0 Monitor and Control

Two new LOFAR2.0 stations (CS032, RS307) have now been installed, and the software team has been working hard to provide the necessary software-based support for that.

Furthermore, the software now creates statistics files routinely, in hdf5-format with full self-describing metadata in the files. The statistics files are written to a minio-based object store running on a self-deployed central storage system that consists of a few former CEP4 node, converted to storage systems. We added alarms on station temperatures and several other vital parameters, which (for now) report to a Slack channel when an alarm is triggered. This must be extended to a proper alarm handling workflow system, for which we are looking at Grafana’s OnCall system.

We also fixed an issue that caused slow beam pointing at the start of a TMSS-controlled observation, and an issue that caused data loss between stations and Cobalt issues when doing LOFAR1 + LOFAR2.0 observations.

A major achievement is that we can now correlate data from (one of the) the PTS LOFAR2.0 stations with the existing LOFAR1 stations. This indicates that we have full control over the relevant parameters (beams, in-station and between-stations delays, absolute timing, data processing at COBALT) which is an important milestone for the LOFAR2.0 development project. So far, we have seen correlated data with CS032 only, as RS307 still must be delivered.

The Operators are developing new Grafana based monitoring screens for the stations and the system as a whole. This helps the whole development team to understand current issues at the new stations, as well as to find out what really is needed in terms of underlying functionality. We are supporting their efforts with infrastructure and knowledge transfer, and with providing requested functionality and libraries.

For the coming months, our priorities will be on preparations for the rollout of all LOFAR2.0 stations. This will create an increase in our control and monitoring data throughput by a factor of 15, at least We need to prepare for that order of magnitude increase, based on what we learn for the current 3 stations. For that, new computing hardware for the central services of LOFAR2.0 must be ordered and prepared. Also, we need to decide on the type of systems to use for the LCU2 and order and prepare those.

Arno Schoenmakers

There are no new developments for the LIFT Transient Buffer implementation. We hope that the FPGA firmware implementation can start after coming October. Once that part is done, we still need to implement the software side of LIFT to fully deliver the Transient Buffer functionality of LOFAR2.0. That can best be done on the longer term, as it requires several remote stations to be ready to properly test all functionality for LIFT.

Ilse van Bemmel

On the 1^st of September 2024 the operations of LOFAR formally came to a close, allowing the CEP Project members to undertake a major step in the project: downsizing the current CEP4 and COBALT2 to ~25% of their capacity. As the hardware is getting old, the risk of failure increases. By reducing the number of nodes in use, we will have spares when something breaks beyond repair. The downsized CEP system is designed to enable support of the commissioning efforts for LOFAR2.0 until the new COBALT3 and CEP6 systems become available.

The downsizing is progressing as planned. Tons of cabling has been disconnected and reconnected and the nodes have physically been moved to another rack space. Everything will be connected to the new LOFAR2.0 network switch, allowing for higher bandwidths and laying the basis for a massive step in performance once all new LOFAR2.0 hardware is operational.

The new hardware for CEP6 and COBALT3 will be ordered using a broker. In June, a tender has been published to hire this broker, and the contract will be final in early October. We then start the process of ordering, first the COBALT3 hardware, which has the longest lead time, and then the CEP6 storage and compute hardware. We expect this to be in house in spring 2025, and to start provisioning the new systems over the following summer.