Daily Image
28-08-2019
Data compression aided AI for LOFAR system health management
| Submitter: | Maurine Santigny and Albert Jan Boonstra |
| Description: | LOFAR system health management (SHM) is charged with detecting and identifying system errors. LOFAR produces more than a thousand spectrogram plots for each observation and it’s not easy to detect an error. As manual inspection is very time consuming, research is on-going to see whether artificial intelligence (AI) could play a role in SHM. This track is currently pursued in one of the projects of the NWO Efficient Deep Learning (EDL) programme and also in a project of the Innovatie Cluster Drachten (ICD). But how about classic filtering approaches, could those be useful in feature and anomaly detection? Could these techniques be combined with AI or perhaps run stand-alone? To start answering these questions we have looked into ‘classic’ filters that select a subspace (compressing data) of each spectrogram in order to separate features that we see in LOFAR spectrogram plots. When features are separated it is easier to build detectors that can detect those features. The image shows a LOFAR spectrogram example of the L580401 data set. It shows the original spectrogram on the left, with time on the vertical axes and frequency on the horizontal axis. Visible is intermittent RFI and frequency dependent gain anomalies. The right figure shows a spectrogram reconstruction after wavelet filtering that removed (separated) vertical stripes. We also tested 2D Fourier transform filters and singular value decomposition (SVD) filters, and applied these on 53 spectrograms with 'typical' features. We found that eight of our developed filters were able to detect at least 85% of the features existing in our (small) dataset of preselected examples. Although the dataset used is small, we think these results are very promising and that this approach can lead to useful tooling to automatically generate data quality reports. |
| Copyright: | ASTRON |
| Tweet |  |