I want to start by giving a warm welcome to team Ruby. From January 2023, the LOFAR software development teams Red and Gold merged into the new sparkling team Ruby. In team Ruby, software developers from ASTRON, INAF, GLOW, and soon also FLOW, collaborate on new software for LOFAR.
The biggest highlight of the past few weeks was undoubtedly that the LOFAR2.0 hardware has been ordered. The global market is still unpredictable: Lead times remain long and prices continue to rise. That is a big risk and that is why it is so important that those purchase orders have been placed. We expect that the hardware will be delivered from early 2024. Many thanks to everyone who contributed!
You can read about all the other highlights of the past two months in this newsletter. Well done everyone, and happy reading!
Introducing CMMS: a better tool for maintaining our world-class telescope!
Imagine yourself being the maintainer of the LOFAR telescope. You want to keep the telescope in the best shape to provide scientific data. Suppose you are in the field repairing a Low Band Antenna. In that case, you may want to quickly get access to the history and technical drawings of that particular piece of hardware. With the current tooling available that turns out to be quite a challenge: You probably have to send some emails and ask around to get the correct technical drawing. In the case that no one is answering immediately, that could even take multiple days and you may still not have the most recent document. It should be possible to make that process easier.
Moreover, as part of the LOFAR2.0 upgrade, many new hardware units will be installed throughout Europe. In that case, you want to be able to keep track of every hardware unit: you want to know whether it is in production at the manufacturer, already delivered to the warehouse, integrated into a larger sub-system, or already transported and installed at a station.
Therefore, we combine the needs and introduce the Configuration Maintenance Management System (CMMS): one tool for tracking the status and history of the hardware units and also providing access to detailed technical documents. The LOFAR2.0 rollout benefits from having an overview during the rollout process. The maintainer in the field will be able to quickly retrieve the technical and historic data. This will save a lot of time in repairing and in turn, improving the telescope in delivering more and better data to our scientists.
After an intensive testing period we have almost concluded the LOFAR2.0 Test Station (L2TS) phase 1 tests. These tests, aimed at validating the board designs in a real LOFAR environment, have been extremely successful. We have not only demonstrated that we comply to nearly all highly stringent signal processing requirements, but also that we are capable to detect issues with the LOFAR antennas than we could not detect in LOFAR1. And, very importantly, that we can run and monitor the entire system for two weeks continuously, without issues and without losing any data.
We are now in the process of preparing for L2TS phase 2 testing. For that, additional hardware has to be delivered so we can expand the phase 1 test setup to that of a fully equipped LOFAR2.0 station. This will be the first full-scale setup of a LOFAR2.0 station, with all antennas connected again. The goal of this setup is to test the signal processing at the full scale of a single station and to find out if we indeed have the cooling capacity to run the entire station at full scale. Furthermore, we intend to combine beamlet data from this LOFAR2.0 station with that of existing LOFAR1 stations at the central processing facility COBALT so we can look in detail at the L2TS signal quality in astronomical observations. We will also enable the combined observation specification of LOFAR1 stations and L2TS in TMSS, the new observation specification and scheduling system, so we can run L2TS as part of LOFAR1 observations.
The hardware for L2TS phase 2 is delivered board by board. At the moment of writing, we have received six new Uniboards, four new APSPU power boards and three APSCT clock distribution boards. A large number of RCU2s are planned for delivery by the end of February. When these and all other boards have arrived and been accepted, we can assemble and verify the new subracks at ASTRON, and ultimately transport them to the CS001 station. By early April we should then be ready for the first test runs of L2TS phase 2.
Meanwhile we are focusing on the procurement of the final hardware for rollout, and building the manufacturing test setups that will be used by the board manufacturers to test the assembled boards before shipping these to ASTRON.
The LOFAR2.0 Station Control software team has been strengthened with the software engineers that worked on TMSS in a new team called “Team Ruby”. This team has all the experience and knowledge to deliver the required LOFAR2.0 telescope system monitor, control, specification and scheduling systems.
The Station development project management position has changed on February 1st as well, with Arno Schoenmakers accepting the role of “Product Owner” for the newly formed team Ruby and Ágnes Mika as the new Station development project manager.
After hard work for many of us and a long period of preparation, we finally got the orders out for the production of all the LOFAR2.0 electronic modules. As mentioned earlier, we will upgrade all current LOFAR stations and two additional new stations. Production will be carried out in two phases. Phase-1 has a limited number of hardware production suitable for the upgrade of two LOFAR stations including some spares. This phase is a so called “null-series” production phase in which we can verify the design and production quality. This allows us to make (small) adjustments in the design, production and process if necessary. In this way, we reduce the risk and have a better control over the production process.
In order to limit the financial and lead time (and maybe availability) risk, we have ordered the components for both phase-1 and phase-2 at the start of the orders. PCBs will be ordered in two phases, bare boards for phase-2 will be ordered after the verification of the null-series.
At this moment, the electronic modules for L2TS arriving and are currently in a lab-test phase. As expected, small issues need to be fixed before these modules are ready for L2TS verification. The mechanical hardware such as the subracks and module front-panels are on order.
We are planning to complete L2TS hardware delivery at the coming spring period. At that time, the weather conditions will be hopefully fine to carry on the work outdoors at CS001.
At the end of December, we had the final sprint review (demo) for TMSS for the institute and show the work over the past 3 years to our colleagues. The demo was not only done by the team, but also by the users who showed major improvements in their workflow and will now have to schedule their own coffee breaks instead of having them be initiated by the system. With the two extra sprints added to the project we managed to deliver a complete system for running the LOFAR observations.
Unfortunately, the server we installed TMSS on initially is not up for the job. We are still working on installing TMSS on a new webserver. This work is delayed by the precarious situation of sysadmin availability. Once this is done, and approved, and some final bugs on a few observing modes are resolved, LOFAR will run fully on TMSS.
TMSS support and development will continue with Team Ruby, the merger of Team Red (TMSS) and Team Gold (LOFAR2 station). We have built a new “operating system” for LOFAR where with small investments we can make significant improvements to make the work of the operators, support and users easier and more efficient and where we can integrate new functionality. High on the list: scheduling observations for the LOFAR2 test station.
Timing Distributor update
White Rabbit switches have been purchased and they will be delivered in June this year. We are preparing now for the rollout activities that will start in Spring. Preparations include the definition of the order in which stations will be upgraded. The plan is to upgrade remote stations during summer 2023, while core stations will be upgraded later, as part of the LOFAR2.0 Station upgrade. This will reduce disturbance to regular LOFAR operations.
The upper panel of the image shows the timeline of the rollout activities; the planning of the core stations will be better defined in the coming months, as the planning of the LOFAR2.0 upgrade takes a more concrete shape. The bottom panel shows a map of the LOFAR stations. As the Timing Distributor activities focus on the Dutch stations, the map is centred in the Netherlands. To show the full array in a single comprehensive figure, distances to the array centre are in a logarithmic scale. The path of the connections to each station are shown in pink, their configuration is relevant to decide the order in which stations will be upgraded.
The goal of the current phase in DANTE is to prototype a full HBA tile with the new AHBAFE. This prototype will be thoroughly tested; first in laboratory conditions and later in the test field. We expect that by May we will have the tile completely tested in the laboratory and will be ready to move to the test field.
After PDR in October, a series of design decisions were identified. The team is working on them prioritizing according to a timeline based on how far in the development process a formal decision is needed.
During the months of January-February the work focuses on improving the HBAFE design with a series of reviews taking place:
- Schematics review covers the update to the HBAFE board schematics; a graphical representation of all electronic components in the board, their function and how they interact. This review was held on January 20th.
- Placement review where we check the placement of the different group of components, without considering how they are connected. At this early stage, it is still possible to move positions on the board. Review scheduled for February 8th.
- Layout review, the layout shows the exact physical locations of every component on the PCB and shows the wires that connect them together. Scheduled for February 16th.
In December, the blocks prototyped in the previous phase of the development were potted by Reijnen, a company in the north of the Netherlands. The objective was to identify any issues during the procedure and to gain experience with the company. Once potted, and back at ASTRON, the boards were tested to spot early on any changes in the RF performance due to the potting material. The tests were successful and did not show any negative effects.
Another aspect we are working on is the development of the test setup, needed for full-scale production. This development is based on the knowledge and lessons learned from LOFAR1 production and is aligned with the activities to prepare the production test setup of the LOFAR2.0 boards.