Daily Image
16-05-2018
LOFAR data quality inspection using machine learning
| Submitter: | Albert-Jan Boonstra |
| Description: | In modern radio telescopes, System Health Management systems are crucial for detection and identifying system errors. Due to the increasing scale and complexity of the systems involved, this is becoming becoming more and more difficult. LOFAR for example produces a few thousand spectrogram plots for each observation, and many other monitoring data as well. Spotting an error and identifying it is not an easy task and for the SKA this challenge will even be bigger. So why not try using artificial intelligence, or more specifically machine learning, for error detection? This was the charge of the ADDER project, an Alliance collaboration between ASTRON and the Netherlands eScience Center (NLeSC) in Amsterdam. Starting in 2016, we have created an automatic system that automatically groups common features, so called "clusters", in LOFAR visibility spectrogram plots. The machine learning algorithm was trained with roughly 100 LOFAR measurement sets, and the results are spectacular in the sense that it actually gives clusters that capture the kind of features we are looking for. Using this tool it is now possible to quickly trace anomalies. Another beauty of this approach is that it works well with the (about 100 MByte size HDF5) compressed data sets, so that the processing time required is minimal. The figure shows the graphical environment (GUI) that was used for the April ADDER busy day aimed at testing the approach. The "cluster space" graph shows the identified visibility clusters, each depicted with a colour, and containing baselines with specific common features in some abstract two-dimensional space. The "station locations" figure shows all Dutch and ILT telescope positions on a concentric logarithmic plane, so that it is easy to see which baselines belong to a specific cluster. The upper-right four figures show typical cluster members for the (XX, XY, YX and YY) correlation spectrograms. In this example it was one station that broke down, but we found many more subtle features and underlying causes as well. The automatic creation of the compressed LOFAR HDF5 files is now part of the LOFAR online system, integrating the new inspection tooling in day to day operations is a next step. The very successful ADDER project is now finished, and follow-up research will be conducted in a new collaboration that starts in the course of this year. |
| Copyright: | ASTRON / NLeSC |
| Tweet |  |