Interferometric Mode

Major Observing modes

Signal Path

Antennas Description

Station Description and Configuration

Array Configuration

Imaging Capability and Sensitivity

Frequency, Subband Selection, and RFI Situation

Beam Definition 

Transient Buffer Boards

Data Products and Management and Long-Term Archive

Data Quality Inspection

CEP and LTA Computing Facilities

Functionality Enhancements

System Notes


    The interferometric imaging mode provides correlated visibility data, just like traditional aperture synthesis radio telescope arrays consisting of antenna elements. The goal of the LOFAR imaging mode is to achieve high fidelity, low noise images of a range of astronomical objects, using customizable observing parameters.

    In this operating mode, station beams are correlated at a central facility (the COBALT correlator) to produce raw visibility data. Further processing, which consists of calibration and imaging, is handled off-line on the central processing cluster, CEP2/CEP4. Calibration is an iterative process of obtaining the best estimates of instrumental and environmental effects such as electronic station gains and ionospheric delays. The final data products for this mode include the calibrated uv data, optionally averaged in time and frequency, and corresponding image cubes. The visibility averaging reduces the data volume to a manageable level, while minimizing the effects of time and bandwidth smearing.

    In LOFAR Version 2 available in Cycle 6, on Low Band Antenna (LBA) and one High Band Antenna (HBA) observing mode with associated "recipes" for the imaging pipeline are offered for allocated observations.

    However because of the limited stability of the intra-network communication on CEP2, HBA imaging pipelines currently cannot be successfully completed. Therefore, until CEP4 is available, the RO will not offer to process data with the imaging pipeline in HBA. Only the calibration pipeline or pre-processing will be offered in HBA.

    Expert users can also request modified observing and analysis strategies, which will be considered at the time of the proposal evaluation.


Observations with the Low Band Antennas (LBA): 

Single observations that are continuous in time/Hour Angle: 

- Half of the available bandwidth (BW) on the target field (BW<=48 MHz, <=244 subbands) and half on a calibrator (same frequencies as the target, BW<=48 MHz, <=244 subbands). 

- Observations in the band of 10-80 MHz (either 10 MHz or 30 MHz filters are possible).

- Suggested range of correlator subbands is 114-357 

- Processing performed with the Standard Imaging Pipeline or Pre-processing Pipeline.

- Suggested averaging factors are 8 channels in frequency and 5 seconds in time.More details are given in the sections below.


Observations with the High Band Antennas (HBA): 

Observations can be specified in 4 different schemes, covering one of the 3 HBA bands: 110-190 MHz (with sampling clock 200 MHz), 170-230 MHz (with sampling clock 160 MHz) or 210-250 MHz (with sampling clock 200 MHz). 


i) Continuous in time/Hour Angle observation of the target bracketed by short calibrator runs. Alternatively, one could adopt the LBA strategy (Half of the available bandwidth on the target field and half on a calibrator) when a bright calibrator is present within the analogue beam of the HBA tiles (up to ~10 degrees from the target). 

-Processing performed with the Standard Imaging Pipeline or Pre-processing Pipeline.


ii) Two scans, one on the calibrator (5-10 min) and a long continuous run on the target. 

- Up to the full available bandwidth (BW < 80 MHz).

- Processing performed with the Standard Imaging Pipeline or Pre-processing pipeline. 

- This is the optimal strategy to use if advanced faceted imaging (extreme peeling) needs to be used in the calibration process.  


iii) Interleaved calibrator observations (eg. 2 min) with target field (eg. ~ 30 min), quasi-continuous in HA. 

- Up to the full available bandwidth (BW < 80 MHz).

- Observations in one of two HBA bands: 110-190 MHz or 210-250 MHz 

- Processing performed with the Standard Imaging Pipeline or Pre-processing pipeline.  


iv) If the user has a good initial model of the target field at his/her disposal, observations could be performed using the full bandwidth on the target. 

- Processing performed with the Standard Imaging Pipeline (Calibrator pipeline)  Pre-processing Pipeline. 


Suggested ranges of correlator subbands for observations in band 110-190 MHz are: 51-442 (i.e. 110-186 MHz) for continuous bandwidth, or the ranges 77-356 (i.e. 115-169 MHz), 358-396 (i.e. 170-177 MHz), 407-456 (i.e. 179-189 MHz) which exclude the known RFI bands. Suited ranges of correlator subbands for observations in band 170-230 MHz will be advertised soon (commissioning ongoing).

Suggested averaging factors are 4 channels in frequency and 5 seconds in time.

More details about the processing are given in the sections below.


3C196 or 3C295 are strongly recommended as flux calibrators, either by including them on a long run or by observing them for 5 min at the beginning and at the end of the observation. In the HBA, 3C147 and 3C48 may also be used. For calibrating European baselines, the proposer is adviced to use 3C196, 3C147, or 3C48. Note that for accurate gain calibration at LOFAR-NL or European scales, these sources need resolved models. In Table A we summarize the relevant parameters of the flux calibrators.

    A. Flux calibrators 

    Source Kind
    RA (h m s) DEC (o ‘ ")  I at 150 MHz   spectral index
    3C196 Seyfert 1 Galaxy LBA+HBA 08 13 36.07 +48 13 02.58  83.084  -0.699, -0.110
    3C295 Seyfert 2 Galaxy HBA 14 11 20.52 +52 12 09.86  97.763  -0.582,-0.298, 0.583,-0.363
    3C147 Seyfert 1 Galaxy LBA+HBA 05 42 36.26 +49 51 07.08  66.738 -0.022,-1.012,0.549 
    3C48 Quasar LBA+HBA 01 37 41.30 +33 09 35.12  64.768 -0.387,-0.420,0.181 
    3C286 Quasar  LBA+HBA 13 31 08.3  +30 30 33   27.477 -0.158,0.032,-0.180 
    3C287 Quasar LBA+HBA 13 30 37.7 +25 09 11   16.367  -0.364
    3C380 Quasar LBA+HBA 18 29 31.8  +48 44 46  77.352  -0.767

    Table A: calibrators to be used in LOFAR observations


    The list of polarized sources, required for monitoring ionospheric RM changes for accurate polarimetry, is given in Table B:

    B. Polarized sources

    Source Kind Band RA ( h m s) DEC (o ‘ ")
    Western hotspot DA240 Radio galaxy hotspot HBA 07 49 48.02 +55 54 22.1
    PSR B0834+06 Bright pulsar HBA 08 37 05.64 +06 10 14.56
    PSR B1642-03 Bright pulsar LBA+HBA 16 45 02.04 -03 17 58.32
    PSR B1919+21 Bright pulsar LBA+HBA 19 21 44.82 +21 53 02.25
    PSR B1937+21 Bright pulsar HBA 19 39 38.56 +21 34 59.14
    PSR B2217+47 Bright pulsar LBA+HBA 22 19 48.14 +47 54 53.93

    Table B: polarized sources for LOFAR observations


    The "Standard Imaging Pipeline"

    Processing of the raw uv data, which consists of calibration and imaging, is handled offline via a series of automated pipelines (see "Software Pipelines"). Calibration is an iterative process to obtain the best estimates of instrumental and environmental effects such as electronic station gains, clock and ionospheric delays


    Figure 1: Layout of The Imaging Pipeline. A short overview of the pipeline is given by Heald et al (2010)[PDF].


    The Standard Imaging Pipeline is under development, and the first version oroginally deployed in LOFAR Version 1.0 and still used in Version 2.0 is graphically illustrated in Figure 1.

    The first standard data processing steps, described below, are called the Pre-processing Pipeline and MSSS-standard pipelines in which are encapsulated the Calibrator, Target and Imaging pipeline:


    (a) Pre-Processing Pipeline: Flags the data in time and frequency, and optionally averages them in time, frequency, or both (the software that performs this step is labeled NDPPP - New Default Pre-Processing Pipeline). This stage of the processing also includes, if requested, a subtraction of the contributions of the brightest sources in the sky (the so called "A-team": Cygnus A, Cassiopeia A, Virgo A, etc...) from the visibilities through the 'demixing' algorithm (B. van der Tol, PhD thesis).  Currently, users should specify if demixing is to be used, and which sources should be demixed. In the next months the "smart demix" will be implemented in the pipeline, this algorithm is able to decide by itself if demix is needed and for which part of the observation. 

    (b) Calibrator pipeline: After the initial pre-processing (as per the Pre-Processing Pipeline), this uses the BBS software to compute the antenna gains, based on a calibrator for which the spectrum is well known at the LOFAR frequencies (see Table A). The solutions of the BBS run are stored in the .INST files.

    (c) The Target pipeline, starts with initial pre-processing. This step is followed by the BBS calibration, which applies the externally generated solutions for the calibrator to the target field. In the future, the calibration of the complex station gains will be achieved using a local sky model (LSM). This will be generated from the Global Sky Model (GSM), which will soon be provided by the MSSS survey.

    (d) Imaging pipeline. 

    The sub-bands are collected and concatenated in frequency according to the size of the bandwidth you requested for your images (or using a default size 2 MHz). Since, after the solution transfer from the calibrator to the target, low S/N in the solution from the calibrator pipelines could introduce spikes in your data, these are flagged. A run of phase calibration with BBS is performed using a GSM model extracted from the VLSS field as a skymodel. Before the imaging step the sub-bands are concatenated in time when observations are performed in snapshot. After this procedure, AWImager is performed and the images are produced in hdf5 format. AWImager is able to perform both W-projection and A-projection, a scheme that can potentially take into account all direction-dependent effects in the deconvolution step. Source finding software is used to identify the sources detected in the image, and generate an updated local sky model.

(e) Self-calibration pipeline.

This pipeline is an extended version of the above described imaging pipeline (d).  A defined number of 'major cycles' loops are performed by starting from the imaging pipeline data products. Several loops of phase calibration, imaging and source extraction are performed in order to update the concatenated dataset improving the initial model (local sky model). Concatenated Measurement Sets, images and source catalogues of intermediate loops are the final products of this pipeline.

The self-calibration pipeline that is being implemented in the Radio Observatory operational software is a version zero of the more sophisticated pipeline being developed in the framework of the Calibration and Imaging Tiger Team (CITT). The self-calibration pipeline implemented for the Radio Observatory is currently being commissioned, tailored and characterized in such a way that the operations of the overall telescope will not be affected (e.g. limiting the number of threads and memory usage).

Please carefully read below some aspects about this version of self-calibration pipeline that should be taken into account when adopting it:

-  The pipeline will produce a predefined set of images with resolution up to 20 arcsec. Intermediate resolutions (not decided yet) will be fixed and set by default by the Radio Observatory; these will be chosen in order to optimize the distribution in the uv-plane.

-  The data from each sub band will be averaged to one channel, before being concatenated into groups, independent from the resolution requested for the target pipeline.

-  Occasionally, a systematic shift in position has been observed in the final image. The entity of it might depend on the field configuration and/or frequency. The reason of it is implicit on the definition of self-calibration for which the absolute information of the phase are lost. While investigations are going on in order to prevent this or minimize it, we want to solicit the user to carefully check the astrometry of the delivered images.

-  The development of the self-calibration pipeline has been optimized for HBA datasets and limited to a certain resolution since lower frequencies and higher resolutions could only be achieved using more sophisticated algorithms which adopt direction-dependent calibration methods.              

The recent commissioning of the self-calibration pipeline in production has highlighted that some fundamental aspects of the current pipeline framework need to be improved in order to make it robust and supportable on CEP2. Therefore the Radio Observatory cannot offer this pipeline in production as yet. Work will continue during the coming period to address these points and fully commission and characterize it. The aim of the Radio Observatory is still to ultimately support it on the CEP2 processing cluster as soon as possible during Cycle 6.

Meanwhile, for Cycle 6 proposals, CEP3 processing time can be requested by the users to perform self-calibration in a standalone fashion, as described in the LOFAR Imaging Cookbook.  If you are willing to adopt this CEP3 offline option in the period before standard CEP2 self-calibration becomes operational, please make this clear in your proposal by answering the relevant question in the technical section of the proposal "Off-line data processing on RO facilities (CEP3) requirement". Alternatively, proposers may describe how they plan offline processing to their required image quality on their own compute resources.


    Considerations about the performance of the above pipelines can be read in the following sections.

    The entire end-to-end process, from performing the observation through obtaining the final images, is overseen by the SCHEDULER, software specially developed for LOFAR, which manages both the observational and the computational resources available. 


    Final Imaging Data Products 

    The final data products include the calibrated uv data, optionally averaged in time and frequency, corresponding images/image cubes and source catalogues. Visibility averaging is performed to a level that reduces the data volume to a manageable level, while minimizing the effects of time and bandwidth smearing. The averaging parameters, as well as the estimated storage capacity required in the LOFAR Long Term Archive, are also specified by the users through the North Star proposal submission tool.

    These final products will be stored at the LOFAR Long Term Archive where, in the future, significant computing facilities may become available for further re-processing. It is possible to routinely export datasets from the LTA to investigators for reduction and analysis using their own resources or through the use of suitable resources on the GRID.

    Immediately following the observation the user will receive a notification in which they will be requested to carefully inspect the raw visibility plots within 24 hours from the notification.  The data will then be processed and archived automatically according to the specifications given by users in their proposal.

    Additionally, in case the user plans to perform further post-processing on the commissioning cluster (currently CEP3), upon user request in the proposal, Science Support can copy the processed data there. It is then the users responsibility to back up the data to their own facilities. Data will be automatically deleted from CEP3 after 4 weeks from the copy. 
    From the moment the data are made available to the users at the LTA or, if requested, on CEP3, the users will have two weeks available to check the quality of their data and report problems to the Observatory. After this time window has passed, no requests for re-observation will be considered.


Current Imaging Status and Performance


    The imaging performance of LOFAR has been assessed using the products of both data reduction strategies performed by experienced commissioners and the Standard Imaging Pipeline. The results of the two approaches are reported below.

    Manual Reduction Strategies

    Expert users have adopted standard LOFAR software to produce images, exploring various analysis strategies, in some cases involving also self-calibration. Their results are reported in Table 1 for both HBA and LBA observations.


    1. Cycle0 and commissioning fields - HBA & LBA



    van Weeren








    Target Field





    Total observing time (hrs)










    Resolution $ (arcsec)

    6 x 6

    6 x 6 20 x 20 11 x 11

    Imaged FOV (deg)

    12 7 7.5 4.5 

    Final RMS Noise (mJy/beam)





    Equivalent noise over
    2 MHz bandwidth and
    6 hours (mJy/beam) 





    Noise /thermal noise ratio





     Calibration strategy BBS intial
    calibration with 2000 sources,
    robust SAGECAL
    with 20000 sources, excon imaging 
    Facet calibration extreme peeling. Transfer of
    phase calibration
    against GSM
    (vlss) model,
    1 self-calibration cycle.  
    Transfer of 
    SAGECAL for
    Ateam removal, 




    de Gasperin






    Target Field




    Total observing time (hrs)



    Bandwidth (MHz)




    Resolution $ (arcsec)

    23x16 23x15 310x210

    Imaged FOV (deg)

    7 9 15

    Final RMS Noise (mJy/beam)




    Equivalent noise over
    2 MHz bandwidth and
    6 hours (mJy/beam)




    Noise /thermal noise ratio



     Calibration strategy Transfer of
    time smooth
    of the solutions,
    phase calibration 
    Transfer of 
    DDE calibration,
    clock/TEC separation,
    phase only self-calibration 
$ The resolution depends on the stations used for the imaging

Table 1: Examples of sensitivities reached in Cycle0 and commissioning observations in HBA and LBA


In the LBA band, expert users have demonstrated that, using a simple analysis strategy that does not employ position dependent calibration or self-calibration, total intensity sensitivities of ~10 times the theoretical thermal noise in relatively long observations (6-10 hours) can be reached. Using more involved calibration techniques (as self-calibration or direction dependent calibration), sensitivities of 4-5 times the theoretical thermal noise have already been achieved.

For the HBA, experienced users using simple analysis strategies reached sensitivities of the order of 6-10 times the theoretical thermal noise in images at a resolution of 10"-20". On the other hand, after investing considerable post-processing effort and an high quality models for the phase calibration a noise of 1-2 times the thermal noise has been achieved.

Despite the good performance in terms of noise level, it is worth to mention that the image fidelity of LOFAR images is not yet fully accomplished. When the image noise is below 1 mJy/beam, artifacts around bright sources of about 1 Jy, become more prominent limiting by far the reliability of the low brightness features in the image. To try to quantify this effect we define local dynamic range as the ratio between the peak source flux and the brightness of the highest artifact around that source; the typical value of local dynamic range for LOFAR images is below 100. 


Imaging Pipeline Performance

The Standard Imaging Pipeline is under development and is based on the experience gained by analyzing commissioning observations with different strategies. In its implementation in LOFAR Version 2.0, it produces calibrated data and initial images using only one imaging step. Development of extensions and refinements including "Major Cycle" calibration, direction dependent calibration, clock/TEC separation, fit and correction for an ionospheric phase screen are under way through the Calibration and Imaging Tiger Team (CITT).


Part 1: To assess the performance of the Imaging pipeline, HBA and LBA observations were performed and reduced using the standard strategies mentioned above. Specifically, a typical field with no dominant sources was selected from the MSSS catalogue: L227+69, centered at RA = 15h07m49s and DEC = +69d14m24s. To explore any possible dependence by observing frequency, the data were analyzed in datasets of ~2 MHz bandwidth separated by ~ 2 MHz from each other. For the HBA, the final bands were distributed between 115 and 185 MHz, for the LBA between 54 and 75 MHz. The imaging was performed for the full FOV and using various physical baseline selections, specifically 3, 6, 12, 24, 48, and 79 km. Uniform weighting was used to make the maps and deconvolution was applied using 2500 cleaning cycles. The results for 4 bands spread uniformly over the full observing bandwidth are reported in table 2A for the HBA band and 2B for the LBA band. The detailed overviews of the behaviour of the noise, resolution, and imaging time as a function of baseline cut are available here for the HBA and here for the LBA. Note that the thermal noise has been computed though the LOFAR Image Noise Calculator, which uses the relevant relations reported here. The adopted factors assume an ideal performace of the instrument.



2A. L227+69 - HBA

Central Frequency HBA band 116 MHz 141 MHz 160 MHz 180 MHz
Resolution $ (arcsec) 15 x 6 14 x 6 13 x 6 13 x 6
Field of View Imaged (deg) 6
Achieved RMS noise over
2MHz bandwidth and
6 hours (mJy/beam)
3.5 2.2 1.5 1.4
Noise/thermal noise ratio 16 13 9 7


2B. L227+69 - LBA


Central Frequency LBA band
57 MHz 65 MHz 71 MHz 75 MHz

Resolution $ (arcsec)

28 x 13

26 x 11
23 x 10
26 x 9

Field of View Imaged (deg)


Achieved RMS noise over
2 MHz bandwidth and
6 hours (mJy/beam)





Noise /thermal noise ratio





Table 2A & 2B : characteristics of the images of the MSSS field L227+69 in HBA (top) and LBA (bottom). The field does not host dominant sources (their typical peak flux is < 15 Jy).


Part 2: After the characterisation Part 1, even though no major changes were made in the calibration software, a few features in the hardware were upgraded: syncopic boards installed in all the Dutch stations improving their sensitivity, all core stations were syncronized to the same clock and two remote stations (RS409 and RS310) were added to the array. For this reason a new characterisation effort took place in 2013, repeating the same exercise of observing the field L227+69 for one hour (instead of 6 hours) and imaging it at 3.5, 50 and 82 km. Despite the expectations there are no major differences between the Part 1 and Part 2 results so that the hardware improvements were not directly detectable in term of S/N ratio. Below we mention a couple of reasons which explain our results:

- The most important to take into account is the fact that the ionosphere is being more active this year due to higher Solar activity. This affects the visibilities at LOFAR frequencies, in particular in the LBA band where rapid phase variation could not be calibrated over short time scales, with such bad ionospheric conditions that the ratio between the sensitivity and thermal noise could increase by a factor of two or more. Ionospheric delays have also some effects on the stability of the antenna gains obtained for the calibrator, introducing extra uncertainty and noise when solutions are transferred to the target field.

- The images used for Part1 and Part2 were made with casapy in the first case and with the new awimager in the second case. It is possible that in Part 1 the contribution of bright offset sources was suppressed but, since awimager properly takes into account the variation of the beam during the observations, bright offset sources could cause the noise of Part2 images to be limited by artifacts.

- Another important effect has to be attributed to the artifact level. This is higher in Part 2 since the uv-coverage was sampled with only one hour of observing time compared with the six hours of time used for Part 1.           


When analysing these results it is important to realise that significant improvement can be made in the final images by adopting more complex reduction strategies. The low frequency radio sky is very bright and the FOV seen by dipole instruments such as LOFAR is very wide. The rapid density fluctuations of the ionosphere induce artefacts associated with strong sources located within the FOV; this severely limits the dynamic range of the final images made by adopting simplified reduction strategies such as those currently offered by the Standard Imaging Pipeline. To improve the map quality it is crucial to apply direction-dependent calibration towards the offending sources. This technique is known to be compute-time intensive, although recent developments made it more efficient and applicable to large low-frequency datasets. The noise estimates reported here will significantly improve once this technique is part of the Standard Imaging Pipeline.

Since 2013, efforts are invested in improving the performance of the Standard imaging Pipeline, but these enhancements still need to be included in the production system. Therefore, for the preparation of the Cycle 6 proposals, the user should still take into account the values listed in Table 2A and 2B. 

To assess the limitations induced by off-axis errors, an observation of an empty field (Elais, RA = 16h14m00s and DEC = +54d29m59s) was performed in HBA. The data were processed by Vibor Jelic through automatic scripts performing similar steps as the Standard Imaging Pipeline. The results for 4 bands uniformly distributed over the full observing bandwidth are reported in Table 3. The detailed overview of the noise dependence with frequency is reported here.


3. Elais field - HBA


Central Frequency HBA band
138 MHz 141 MHz 160 MHz 180 MHz

Resolution $ (arcsec)

15 x 6

14 x 6
13 x 6
13 x 6

Imaged FOV (deg)


Achieved RMS noise over
2 MHz bandwidth and
6 hours (mJy/beam)





Noise /thermal noise ratio





Table 3: image characteristics for the Elais field


When comparing these results with those obtained for L227+69 in HBA, it is clear that direction-dependent effects are one of the major effects that limit the quality of the images of low-frequency fields hosting strong off-axis sources. Direction-dependent calibration will be included in the Standard Imaging Pipeline in the near future, but will not be available for the remainder of Cycle 2.

A few quality metrics were investigated in the final maps of L227+69 and Elais, specifically the positional and flux accuracy of the sources within the primary beam. The positions of the sources in the LOFAR maps are correct to within 10" with respect to the catalogue positions. Since gain transfer from a calibrator was adopted during the data reduction, the flux accuracy of the sources in the target field depends on the accuracy of the calibrator fluxes, which is about 5-10%. The accuracy of the primary beam correction was also investigated during the commissioning work and it showed that primary beam corrected fluxes are within 10-20% from the interpolated VLSS-WENSS values up to 3 degrees from the field center.

Preliminary polarization calibration and full-Stokes imaging has already been accomplished during the commissioning phase, but the performance of this application is currently not yet well characterized. While work is underway to offer polarization imaging as a standard mode, it currently requires a carefully argued technical case by groups with sufficient expertise.


Recent and Forthcoming Improvements in the Standard Imaging Pipeline

Over the next few months, the performance of the instrument will improve due to several factors that are currently under development.

Several analysis steps are planned to be included in the Standard Imaging Pipeline as part of the development towards the next LOFAR software version. These points are all being addressed by the Calibration and Imaging Tiger Team (CITT). Some of these are major new software elements while others are simple additions to existing procedures:

Direction-dependent calibration has shown to improve significantly the quality of the images. An improvement by a factor of 3-5 in the image noise has been obtained when applied to MSSS LBA observations (see B. Adebahr's report at: and in HBA commissioning observations using SAGECal (see E. Orru' report at: Direction dependent calibration is still a bottleneck for computing resources. There are several possibilities to address this; improving BBS computational efficiency, using the SAGECAL algorithm, using a script to automatically derive the sources to be peeled and perform peeling using BBS. Furthermore, investigations of the quality and the computational efficiency of different algorithms and implementations (SAGECal, BBS) should be continued.

Clock/TEC separation: Initial tests demonstrated that frequency dependent effect due to the clock delays between stations and delays due to TEC differences can in principle be addressed in order to separate instrumental effects from ionospheric direction-dependent effects. Future commissioning and development is needed in order to fully understand the outcome of phase residuals after clock/TEC phase delays have been "removed"  from the data (see M. Mevius report at:

Ionospheric correction is crucial in order to reach the thermal noise and high quality images at LOFAR wavelengths. Two main approaches have been attempted and are currently under investigation in order to address the issue. One involves solving for many directions during the calibration phase (DDC) on short time scales so that the ionospheric effects are absorbed in the calibration solutions (see Yatawatta et al. 2013, A&A, 550A, 136Y). The other involves fitting a phase screen to the directional TEC phase solutions and applying this during the imaging stage (method under commission inspired on SPAM algorithm (see Intema et al. 2009, A&A, 501, 1185I). 


Computational Requirements for the Standard Imaging Pipeline

The computational requirements of the imaging mode can be substantial and depend both on observing parameters and image characteristics.

In the following, we present practical estimates for the Processing over Observing time ratio (P/O ratio) separately for the pre-processing and the imaging steps. Note that when considering the computational requirements for the observing proposals, users should account for BOTH of these factors.

a) Pre-processing Time

Each of the software elements in the pre-processing pipelines has a varied and complex dependence on both the observation parameters and the cluster performance, and hence a scaling relation is difficult to determine.

To have realistic estimates of pipeline processing times, typical LBA and HBA observations with durations longer than 2 hours and adopting the full Dutch array were selected from the LOFAR pipeline archive and were statistically analyzed. The results are summarized in the following table:


Nr Demixed Sources

Nr SB 

P/O ratio

0 244






0 80
1 80 0.3









2 244 4.5



1 122


0 366











Table 4: Pre-processing performance for >2h observations with different observation parameters and settings for demix for HBA and LBA. Although the case of 3 demixed sources has not been characterized, a large increase of the P/O ratio for both LBA and HBA is expected.

These guidelines have been implemented in NorthStar, such that pipeline durations are automatically computed for the user.  

Note that:

- Only a standard set up with 64 frequency channels per sub band has been characterised and it is fully supported. Set up with 256 frequency channels are supported for observing but because of the large amount of computing resources needed for the processing, cause swapping on the CEP2 cluster, the use of such a set up will be evaluated by the LOFAR technical panel during the review process of cycle proposals. It will be granted only if strictly needed for reaching science goals claimed in the requesting proposal.

 - The case of 3 demixed sources is expected to drastically increase in terms of P/O ratio for both LBA and HBA and of claimed computing resources. To safeguard the overall operations of the LOFAR system, the Radio Observatory does not support 3 demixed sources on the CEP2 cluster.   


b) Imaging Time

Imaging with AWimager is a computationally intensive task and, given the finite computing resources of the post-processing cluster, special consideration is needed in deriving the proposed parameters of the final images, particularly the resolution and the imaged field of view (FoV).

The ratio of imaging processing time over observing time (P/O) has been estimated for the L227+69 observations described above (Imaging Pipeline Performance Part2). The following considerations can be made:

a) For the LBA imaging in LBA outer the P/O ratio is about 0.8 for uvranges restricted to 3.5 km (core) and field of view of 12 degrees, is  ~1.2 when uv-ranges restricted to 50 km (core + inner remote stations) and the field of view is 8 degrees. When including all remote stations, the imaging process becomes too intensive. In particular, swap memory is then used on the locus machines, which both puts the locus nodes in danger and does not allow the imaging to make any progress. For this reason, when including more remote stations in imaging, it is necessary to restrict the field of view to ~ 3 degrees. Imaging LBA inner datasets for uv-ranges > 3.5 km is higly discouraged unless conspicuous restrictions are placed on the FOV. 

b) For the HBA the P/O ratio is about 0.9 for uvranges up to 3.5 km and field of view of 5 degrees. For uv-ranges of 50 km and imaging of the 5 degrees FoV the P/O ratio reaches 1.3 or above. When including baselines longer than 50 km, the imaging process becomes too intensive and starts using swap memory on CEP2. In this case, successful imaging runs need the FOV restricted to 3 degrees.

It becomes evident that with the current available processing power, in order to image a dataset, the Field of View has to be limited, in order to include long baselines and produce high resolution images.

Note that (sub)arcsec resolution imaging, using the international stations, is also possible, but scarcity of computational resources on CEP limit it practically to only a limited (1-2 arcmin) area of the full station field of view. While work is under way to offer this as a standard mode, this application currently requires appropriate "VLBI" expertise and dedicated processing resources (see "Long Baseline Imaging with LOFAR").




Current Self-Calibration Status and Performance


Important extensions to the set of standard observing and processing capabilities are being developed, including the automatic self-calibration pipeline. Significant work is going on simultaneously on the underlying pipeline framework within the CEP2 network. Amongst other critical improvements, this will allow robust running of the self-calibration pipeline, which requires stable intra-network communication through long turnaround times. Characterization of the current automatic self-calibration pipeline is not thus yet warranted. Furthermore, the underlying pipeline framework will need to be ported to the successor of CEP2, CEP4, expected to be brought online during Cycle 5. In the mean time, also, a more advanced version of the self-calibration pipeline has been developed in parallel for standalone use. This version is described in the LOFAR Imaging Cookbook.

With regard to self-calibration for Cycle 6, therefore, the Radio Observatory is following a stepwise approach. It is now anticipated that a robust self-calibration pipeline will become available as part of the automatic data processing on the CEP2/CEP4 clusters during the cycle. Users will be informed and consulted when this is becomes relevant for their projects. In the mean time, users can request processing time on ILT computing resources (CEP3) to perform self-calibration using the most recent tested version of the standalone pipeline, if they do not have the requisite resources available themselves. If you are interested in this offline option in the period before standard CEP2/CEP4 self-calibration becomes operational, please make this clear in your proposal by answering the relevant question in the technical section of the proposal "Off-line data processing on RO facilities (CEP3) requirement". Alternatively, proposers may describe how they plan offline processing to achieve the required image quality on their own compute resources.

Based on users experience one CEP3 node can sustain a maximum of six parallel self-calibration jobs, which require in total a P/O ~ 10. Assuming that groups of 10 sub bands are processed in a single run, it will be possible to process 60 sub bands at the same time, while the others will have to follow sequentially.  Therefore a typical observation of 480 sub bands grouped in blocks of 10 sub bands will need a P/O ~ 80 to be fully processed. Consequently a typical 8-hour observation will require 640 hours to be processed on one node, which is within the amount of time of a default CEP3 reservation block. Based on the above please explicitly state the number of CEP3 processing hours you will need to reduce your observations through self-calibration.  

Please note that if in the meantime the RO pipeline implementation will be fully commissioned and characterized by the time that Cycle 6 starts, the Radio Observatory might convert the CEP3 requests into supported self-calibration runs on CEP2.

The noise level of the images obtained by using the Standalone self-calibration pipeline can reach 8 times the thermal noise (calculated using the noise calculator tool). These values are based on a limited set of cases and on a fraction of the total frequency band. We advise the user to take this number into account as an indication of the best possible result achievable with this pipeline. More detailed information could be found in the document here.    



Installing the LOFAR Software Stack


The Lofar LTA software stack is the collection of software that is needed to run the Lofar imaging pipeline. That includes all needed libraries with a specific version. An overview of the LOFAR Software Stack, together with a discussion of various aspects of the software stack, are discussed at this Wiki page.




Design: Kuenst.    Development: Dripl.    © 2016 ASTRON