If the antennae of LOFAR are the senses of the radio telescope, then the central correlator is its brain. It is the place where all the data streams come together and are converted into astronomy data. LOFAR’s current brain is called COBALT 2.0 (Correlator and Beamforming Application platform for the LOFAR Telescope).
Published by the editorial team, 15 June 2020
The correlator team at ASTRON has delivered a remarkable piece of technology with COBALT 2.0. It is the successor to the previous COBALT correlator, which reached its end of life early 2019 as the warranty was going to expire. ‘The replacement was planned well in advance in the form of a LOFAR Mega Mode (LMM) proposal, led by Jason Hessels at ASTRON and University of Amsterdam,’ says Vishambhar Nath Pandey, system researcher at ASTRON. The main idea of the LMM proposal was to turn LOFAR into a truly multitasking radio telescope thus significantly boosting the science returns per observing hour, without changing the hardware on the ground. It received an NWO-M grant in 2017 where NWO (Nederlandse Organisatie voor Wetenschappelijk Onderzoek) and ASTRON funded the project together.
The works on this new brain commenced in 2018, starting with the detailed design and procurement phase through an open EU wide tender. Pandey: ‘COBALT 2.0 is state of the art and future proof in terms of its design and flexibility. It is also amongst the most energy-efficient High Performance and High Throughput Computing (HPC/HTC) correlators possible with the current technology.’ The hardware for COBALT 2.0 arrived in early 2019. Pandey: ‘After successful validation tests for hardware compliance in February 2019, the existing COBALT software was installed, and optimized where necessary, which was made easier due to hardware-software codesign of the correlator.’ In the summer of 2019, after it had passed all tests successfully, COBALT 2.0 was officially put into regular use. ‘Thus far it has been a great success.’
Mega Mode
‘LOFAR’s antennae fields spread across Europe, each generates about 3.1 Gb/s in data streams. These arrive at the central correlator, which appropriately processes and combines all the data streams into a readily usable data form for astronomers’, says Pandey. These are the data that researchers use in their studies. The central correlator COBALT2.0 is capable to simultaneously receive and process the data streams from both 48 high-band antennae fields and 96 low-band antennae fields. Those combined data streams are 148 Gb/s and 296 Gb/s respectively, which roughly translates to the equivalent data rate of a million single-sided single layer Digital Video Discs (DVD-5) per day.
Usually, a correlator can only process the data of a single project at a time. COBALT 2.0 however, can simultaneously process the data of several projects simultaneously. Pandey: ‘We call that the LOFAR Mega Mode.’ The concept is as follows: for a certain research project, LOFAR looks at a specific part of the sky, from which data streams are collected. ‘It may well be possible that this part of the sky encompasses the area which is subject of the project of another researcher,’ Pandey says. ‘That means that the data streams can be used for that second research project as well. You have to appropriately combine the data streams for that second research project, but that is something COBALT 2.0 is capable of due to its immense computing and throughput power.’
Another thing that COBALT 2.0 can do, is to produce both imaging data and beamformed data from the same data streams. Imaging data are used to produce high angular resolution images of a certain part of the sky; beamformed data can be used for high time and frequency resolution applications, like discovery and high cadence monitoring of exotic pulsars, solar and planetary studies, for example. In a very simplified way to produce imaging data, the data streams are appropriately multiplied; by appropriately adding the data streams the correlator can produce beamformed data. And COBALT 2.0 can produce both simultaneously.
Presently the team is busy implementing the new features in LOFAR Mega Mode to the existing COBALT software in order to realize the full potential of COBALT 2.0. Once completed, the resulting increased efficiency and capabilities of LOFAR will lead to new discoveries to help understand and unravel more scientific mysteries of the universe in the years to come.
Extremely energy efficient
The COBALT 2.0 system has an optimal combination of the most appropriate set of components, CPUs, GPUs, network topologies and most energy-efficient power supplies. A CPU, a central processing unit, is skilled in performing lots of different calculations simultaneously; a GPU, a graphic processing unit – the name stems from the fact that these processors were originally developed for graphic cards – is extremely skilled in performing one specific type of programmable calculation. COBALT 2.0 optimally utilizes the CPUs and GPUs for tasks where they are most efficient at.
The new correlator is amongst the most energy-efficient ones. Given the fact that the apparatus will be running 24/7, 365 days a year, that is an important feat. Pandey: ‘Despite COBALT 2.0 being several times faster than its predecessors and its LOFAR Mega Mode capability to process several projects simultaneously, its energy budget even at peak load is well within around € 1.000 a month.’
Already a big success
COBALT 2.0 isn’t only a future highlight for LOFAR, but it can already be called a success: Scientists and Engineers of the French NenuFAR telescope chose the COBALT 2.0 design after comparisons with other possible designs for their own correlator named NICKEL (NenuFAR Imager Correlation Kluster Elaborated from LOFAR’s). Pandey: ‘Our shared work and knowledge saves them years of work, also because they are reusing the correlator software suite developed by ASTRON for LOFAR. They are actually a few years ahead due to this. So, this is an example implementation of ASTRON’s philosophy of open source technical collaboration.’
In addition, the availability of Tensor Cores in Volta V100 GPUs in COBALT 2.0, which can further speed up performance for half-precision (than presently used full precision) computations by about a factor of 5, opens up the exciting future possibility of exploring future new astronomical observing modes which can be carried out with lower precision.
COBALT 2.0 will play a pivotal role in defining the future scientific capabilities of LOFAR. The success of this project is possible due to the pioneering modern digital design of the LOFAR telescope, and learnings from involvement in adapting state of the art new technologies during the GPU based COBALT (2013) and the earlier CPU based IBM Blue Gene -L/P (2004/2008) correlators.
Specifications of COBALT and COBALT 2.0
| COBALT | COBALT 2.0 | |
| Size | 9 nodes | 13 nodes |
| CPU’s | 18x Intel Xeon E52660 2.20GHz | 26x Intel Xeon Gold SP6140 2.30GHz |
| GPU’s | 18x Tesla K10 | 26x Tesla V100 |
| InfiniBand | 18x FDR – 54 Gb/s | 26x EDR – 100 Gb/s |
| Data Input | 36x 10GbE | 104x 10GbE |
| CPU compute capacity | 6 TFLOP/s | 63 TFLOP/s |
| GPU compute capacity | 82 TFLOP/s | 364 TFLOP/s |
| Power consumption (peak) | 6 kWh | 10 kWh |
This was a coordinated effort of the correlator team, comprising of: V.N. Pandey, J. J. D. Mol, P. C. Broekema, J. Romein, C. Bassa, J. Hessels, R. Kaptijn, J. Klipic, R. Bokhorst, J. Schaap, K. Stuurwold, P. Boven, B. Veenboer, Y. Grange, A. Coolen along with the collaboration with the Center for Information Technology (CIT) at the University of Groningen.
On 12 June 2020, LOFAR celebrated its tenth anniversary. The radio telescope is the world’s largest low frequency instrument and is one of the pathfinders of the Square Kilometre Array (SKA), which is currently being developed. Throughout its ten years of operation, LOFAR has made some amazing discoveries. It has been a key part of groundbreaking research, both in astronomy and engineering. Here we feature some future highlights we expect LOFAR to play a pivotal role in.