Welcome to the May 2021 edition of the LOFAR2.0 Newsletter in which we share recent highlights of the LOFAR2.0 development. My highlight for this edition is that the ILT Board approved the outcome of the LOFAR2.0 down scope, bringing our ambitions and our resources back in balance. Of course we preferred to deliver "the full thing", but we better be realistic, make decisions early, and communicate the consequences. The main conclusions approved by the ILT Board are:
- The transient buffer functionality is down scoped. The firmware delivering this functionality will not be implemented now, but can be added later by a firmware update. A scientist already anticipated on this situation and submitted a funding proposal from which the transient buffer can be reinstated. Fingers crossed!
- After May 2022, procurement for the LOFAR2.0 rollout will start and final hardware quantities must be known. Some station owners need more time to secure the funds for their station upgrade. Consequently, LOFAR will temporarily become a hybrid system from 2023, with both LOFAR1 and LOFAR2.0 stations in operation. The ILT board has agreed on the intention to stop the compatibility with LOFAR1 stations by the end of 2026. The higher operational costs to operate LOFAR1 stations in parallel to LOFAR2.0 stations are accepted.
- The updated high-level planning and some changes to the LOFAR2.0 budget have been approved.
Please read on to learn about the other highlights of the last two months. I hope you enjoy reading!
Overview of interfaces, reducing failures during integration
The LOFAR2.0 program is currently in the development phase. Many parts are being designed, prototyped, tested, updated, retested, etcetera. At some point all these parts will be delivered and need to be integrated into a larger system, for example in the Dwingeloo Test Station, and tested. A failing integration test can have a large impact on planning and budget. When a hardware board fails, updating the design and again producing the part can take many weeks to even months. As the planning and budgets are tight already, such failures would lead to either a budget overrun, or deciding to drop key functionality. Such changes so late in the program can have dramatic consequences for the operational use, and as such the scientific capabilities of the instrument.
To get a clear overview of the system and the design decisions, we made a number of figures like the one shown here. It shows products and the interfaces between them, in this case for the power distribution. All these overviews together provide clarity, for example: If an interface for monitoring and control exists, then obviously there also needs to be a power connection and a connection through the network to the local control unit. These simple checks with the correct group of people will verify that all interfaces are taken into account and that the implementation is agreed upon.
So what does this bring us?
Having such a model gives confident in our design. Using these kinds of overviews, we step-by-step verify the interfaces in the design. If there is a unclarity, people from a multi-disciplinary team are asked to resolve that. Over time a more complete picture of what actually is being built arises. With this picture in place, we are confident that we will serve the science community with the best possible LOFAR2.0 instrument for at least another 10 years.
Some of Station project highlights
Last week the DDR of the RCU2 high-band board took place. In a productive meeting with several ASTRON and INAF colleagues, the highlights and compromises of the current design have been discussed. Overall, the design meets most of the requirement, with a few trade-offs to be decided. It is quite an achievement for the team to get to this point, given Covid and all the difficulties because of that. We are looking forward to some real life tests of the first prototype production boards in the Dwingeloo Test Station, DTS.
The preparation for the production of the various DTS prototype boards is progressing well. We are definitely feeling the pain of increased global component delivery times and even PCB substrate availability. The result of this is that production of the DTS prototype boards in some cases will take a few weeks longer than planned. On the positive side, the most expensive and complicated component of LOFAR2.0, the new Uniboard2c, is being produced at this moment at Neways and is on schedule for delivery.
The DTS cabinet is coming to completion. It has been put on a concrete base platform, the preparations for the sunroof have been finished, and the power and grounding have been connected. An additional HBA tile has been kindly provided by Telescope Operations and constructed on site. See the Figure for a picture of the current situation. Next in line is finalising the network connection and getting a White Rabbit clock system in the cabinet. As the receiver chain boards will not be arriving before the summer period, we will set off with extensive cooling tests. This will inform us about the real cooling capacity of the current design. We will use remote controlled heaters and by using dummy PCB boards we will study if the airflow through the subrack and inside the cabinet is sufficient to provide the cooling capacity that is needed, even on hot summer days. Combined with thermal modelling we can identify possible issues and find solutions for these in this early stage of development, which will create a much more robust and thought-out design for the cabinet cooling. Together with the installation of a sunroof this should provide much more resilience against the extreme temperatures of (future) summers. All DTS activities and news can be followed in a blog in Confluence at this URL: https://support.astron.nl/confluence/display/L2M/DTS+NEWS
Meanwhile a simple configuration management system has been setup with the help of ICT, which will serve for the purposes of DTS. Telescope Operations is leading the effort to select a much more mature system for future rollout and operations. Furthermore, the work on ICDs and software monitor and control is progressing steadily as well. The software team has developed a uniformized interface device in Tango, which they hope to present to the Tango community at the end of April. They also have provided the team with a remote access console, in the form of an interactive web-based Jupyter notebook interface and a logging analyzer to the lab test system, LTS.
One of the goals of DUPLLO is to upgrade the clock system in all Dutch stations. The upgrade, in simple words, consists in establishing one single clock for all stations: “one clock to rule them all”.
To achieve this, the White Rabbit technology (developed by CERN) was identified as the best candidate. In theory, the upgrade is straightforward and can be done before the rest of the LOFAR2.0 system is rolled out. But of course, reality is not always simple.
The next months will be critical for the Timing Distributor team; after we perform the last set of measurements on the currently operational LOFAR system and complete the documentation, our design will be subjected to the critical design review. This milestone will mark the end of the development phase; after that the new central clock solution can be finally installed in the LOFAR1.0 system. A detailed procurement plan was established and a request for quotations will be released in a couple of weeks. We are now assessing what will be the best timing to proceed with the rollout.
The LOFAR development is gradually progressing towards the first outdoor tests with the Dwingeloo Test Station (DTS), which is sited at the outdoor test facility next to the laboratory. A major step has been made in the design of the electronic boards for the station subrack for DTS. In parallel, procurement activities started for the production of the prototype boards. At this moment, an order has been placed for the production of ten RCU2L boards (LOFAR2.0 Low-band receiver board) and two clock-boards, APSCT. Orders for the remaining boards for DTS will follow as soon as the design and request for quotations are finalised. Due to extreme long lead-time for components and (PCB) materials, (we have to deal with an overheated electronic components market), delivery times of the assembled boards are longer than expected. Therefore, the procurement process must run in parallel with the design period as much as possible to shorten the overall time. Phase-1 of the Uniboard2 order has just been delivered and will be tested in the lab.
For this newsletter, we present an exciting science result. On the 16th July 2020 a rare opportunity arose to further demonstrate what LOFAR can do for space weather science. The strong compact radio source 3C196 passed behind the tail of comet C/2020 F3 (Neowise), close to the comet itself, enabling us to test whether scintillation from a 100,000km diameter plasma tail could induce enough intensity scintillation to be detectable above that of the solar wind along a 300 million km line of sight. The answer was an emphatic yes! The scintillation was strong and the dynamic spectrum showed features that we do not usually see in scintillation from the solar wind. Interpretation of the data suggests that we are looking at strong turbulence along the boundary between the surrounding fast solar wind and the much slower material enclosed within the tail itself. The observation is completely unique, with the results now being written up for possible submission to Nature.
The COBALT 2.0 project is nearing completion. We are currently finalising the implementation of a spectral leakage issue and the ability to re-point the beamformer during a long-running observation. The final sprint will run until March 30th, and we will host a short demonstration on May 6th showcasing the newly developed abilities of the COBALT correlator and beamformer.
ASTRON is developing TMSS (Telescope Manager Specification System), which is a brand-new software application for the specification, administration, and scheduling of LOFAR observations. It will enable the required support for LOFAR2.0 use cases, while also streamlining LOFAR operations and improving the adaptability and maintainability of software for future extensions.
Over the past two months, several improvements were applied to the user interface to streamline both the monitoring of the observing program and templates preparations. An important milestone was reached in March through the successful execution of various dynamically-scheduled observations and pipelines. A flexible engine that is able to generate reports tailored to the users’ needs is now available. A federated solution is now in place to authenticate users. Eventually, an important achievement was also made in the area of stakeholder engagement. From early April, users are able to track the project progress through the TMSS infographics, which provide a quick and insightful summary of the overall progress of the project and clarify the focus for the current and future developments sprints. A few screenshots taken of the various TMSS interfaces are presented in the image.
The next sprints will see the implementation of additional priority classes in the dynamic scheduler, the support for additional use cases (including the BF modes and responsive telescope), and the implementation of project roles in the federated AAI solution now in place. TMSS phase 1 will end in June 2021. A second phase of TMSS development will start after that to support more science use cases and realise further streamlining of the operational processes.
In recent years an implicit question has been raised regarding on what to do direction dependent (DD) ionospheric calibration: should one model the ionosphere on visibilities, or on gains?
Recently, we have discovered an unusual property related to fitting phase models on gains, which shows that sometimes simply adding more data leads to worse accuracy.
Recall that systematics, e.g. beam model errors, often manifest as very slowly changing phase offsets over the course of an observation. Naturally, one might apply the prior that these systematics are fixed over time windows, to increase the signal of the ionospheric components.
We discover the non-intuitive fact that the posterior accuracy of the ionospheric components peaks at a certain window width and then actually gets worse with larger window sizes (Figure 1). This unusual property is partly explained by the humped behaviour of the Fisher information (Figure 2).