Skip to main content

10 years of LOFAR highlights:The use of a monitor & control system that monitors a physically widely distributed instrument

By Henk Mulder

The day-to-day LOFAR operations require highly specialized monitoring and control systems. We use a system that easily enables us to visualize any values we put in our database in a graphic interface or time-sequenced graphs. ASTRON uses a Supervisory Control and Data Acquisition (SCADA) system called Simatic WinCC Open Architecture (WinCC OA). We can fill the database of this system in many ways: from custom-built interfaces to SNMP (Simple Network Management Protocol) input channels.

Published by the editorial team, 5 June 2020

Organisations in many countries use WinCC OA for large scale projects like the monitoring and controlling of gas/energy distribution or waterways. Even the Large Hadron Collider in CERN is controlled in this way. Because it is specialised in handling millions of data points on many distributed systems, WinCC OA is a perfect fit for us.

In the LOFAR control-room, you will always see the main panel opened. The operator can easily see all the stations in the LOFAR network with a birds-eye view, see how they are performing, and get alerts if action needs to be taken. We can set these alert thresholds (low/suspicious/high) when the need arises, to make sure elements that start having problems can be discovered even before they fail completely.

All kinds of events can be displayed like when a fan stops working, a power supply drops out, cabinet/board temperatures rise too high, or when there is a network hick-up or power cut. It also gives us the ability to turn off antennas remotely and put a note in the system when and why it was turned off, so we can send an engineer to the station to do a repair. This way we are always sure that the observation that is running only includes stations that are working perfectly and the faulty elements are not included in the data.

And when there is a large scale problem, we can stop the observation from the user interface. We do this to make sure no time is lost and we can restart observing later when the system is back in optimum condition.

Because all the elements can be viewed and the reason why they failed is in the database, it also gives us the option to make reports and set priorities for the maintenance cycle. The number of failed elements in HBA/LBA can be viewed and from there we can start making a priority list for the field engineers. We do this roughly four times per year. All the stations in the Dutch network get a maintenance visit every round, making sure we get the most out of LOFAR.

Control-room video wall in Dwingeloo showing “the Navigator”, our Monitoring & Control system

 

The main panel shows the whole system in colour-coded boxes. Hovering over them gives us more information when we need it, clicking on them takes us deeper into the system. As alarms cascade through the system, even the smallest event deep in the system will be viewable on the top level. Here you see two stations in software level 2 for maintenance. And the German stations do not have a network connection right now as they have been switched to local mode.
There are all kinds of different processes/software/firmware and daemons that we monitor during running observations in the monitoring system

 

HBA faulty elements above / LBA elements below (green = on in this specific observation, gray = off, red = faulty). This view also shows us the layout and position of the station elements so we can guide the people in the field.

 

Cabinet view: you can go as deep as you want into it. From LOFAR -> Station -> Cabinet -> Rack -> Subrack -> Board -> Element -> Software / Firmware / etc

 

Turning off elements and leaving a note for the field engineers we usually go with short codes like: OSCILLATION (OSC) / FLAT (FLAT) / HIGH NOISE (HN) / SHORT (SHORT) / DOWN (DOWN) / SUMMATOR NOISE (SN) / LOW NOISE (LN). And afterwards, we leave a note when and what was done to repair it. So if it happens again we know what repair was already done.

On 12 June 2020, LOFAR celebrates its tenth anniversary. The radio telescope is the world’s largest low frequency instrument and is one of the pathfinders of the Square Kilometre Array (SKA), which is currently being developed. Throughout its ten years of operation, LOFAR has made some amazing discoveries. It has been a key part of groundbreaking research, both in astronomy and engineering. Here we feature some – but definitely not all – of these past highlights, with surely more to come in the future.

Sources
Related

@astron

Subscribe to our newsletter. For previous editions, click here.

searchclosechevron-downlinkedin-squarefacebookbarsenvelopelinkedinxingyoutube-playinstagrampaper-planefacebook-officialpinterest-pwhatsappcommentingcross