Source directory /data/lofarschool/data/Exercise-05-DPPP
Contact person V.N. Pandey (pandey -at-
Adriaan Renting (renting -at-

NB. This page can be used for help for exercise 5

The goal of this exercise is to acquaint you with the Default Pre-Processing steps (referred as DPPP from now) of LOFAR data using the software DPPP. DPPP is intended to operate as an integral part of the standard imaging pipeline. Though we will run it manually, it is actually designed to be run in an automated and distributed fashion as an when observations are carried out. You will probably be reminded of this fact now and then in the course of this exercise.

We will not explore running it in a distributed way due to practical network bandwidth and writing permission restrictions in the exercise. We will, however, explore a few aspects of its RFI flagging, compression along frequency axis, time axis, and both.


Very basic notion of radio data reduction, the MeasurementSet format, and casaviewer or any available data viewer.

The goal of this exercise is to get hands-on experience with the reduction of LOFAR data in general and with the DPPP software package in particular. In the course of the exercise you will:

  • Inspect a raw observed LOFAR data
  • Perform a simple run for RFI detection with DPPP
  • Inspect the output to ensure that RFI has been flagged satisfactorily
  • Perform Data compression along Frequency axis
  • Perform Data compression along Time axis
  • Inspect the compressed Measurement Sets

The following is a list of the files present in the data directory for this exercise (/data/lofarschool/data/Exercise-05-DPPP/) along with a short description of each file.

  |-- CS1_IDPPP.log_prop        # DPPP logger properties file.  
  |-- CS1_IDPPP.parset          # A sample DPPP parset.
  |-- L2008_09315_SB12_ORIG.MS  # Original LOFAR measurement Set with
  |                             # auto-correlations flagged.
  |-- L2008_09315_SB12_ORIG_CASAVIEWER.MS
  |                             # Original LOFAR measurement Set to be
  |                             # viewed with casaviewer.
  |-- set_A_CS1_IDPPP.parset    # DPPP parset for exercise 5A
  |-- set_B_CS1_IDPPP.parset    # DPPP parset for exercise 5B
  |-- set_C_CS1_IDPPP.parset    # DPPP parset for exercise 5C
  |-- set_D_CS1_IDPPP.parset    # DPPP parset for exercise 5D
  |-- outdp3_seta_L2008_09315_SB12_ORIG.MSs  # Output measurement set
  |                                          # corresponding to DPPP 
  |                                          # parset for exercise 5A.
  |-- outdp3_setb_L2008_09315_SB12_ORIG.MSs  # Output measurement set
  |                                          # corresponding to DPPP 
  |                                          # parset for exercise 5B.
  |-- outdp3_setc_L2008_09315_SB12_ORIG.MSs  # Output measurement set
  |                                          # corresponding to DPPP 
  |                                          # parset for exercise 5C.
  |-- outdp3_setd_L2008_09315_SB12_ORIG.MSs  # Output measurement set
  |                                          # corresponding to DPPP 
  |                                          # parset for exercise 5D.
  |-- sample_parset_with_explanation.txt     # DPPP parset with some 
  |                                            explanation of terms

This section starts with a step-by-step guide to correctly setup the environment for this exercise. Below that are the step-by-step instructions for the exercise itself.

Setting up the environment

Initialize the LOFAR software environment.

source /users/renting/LOFAR/installed/gnu/lofarinit.csh 

Create a private directory below /data/lofarschool/users. NB. Please do not use spaces or special characters to avoid quoting issues later.

cd /data/lofarschool/users
mkdir <your directory>

Copy all files needed for the exercise. All of them can be found under /data/lofarschool/data/Exercise-05-DPPP.

cd <your directory>
cp /data/lofarschool/data/Exercise-05-DPPP/* .
cp -r /data/lofarschool/data/Exercise-05-DPPP/L2008_09315_SB12_ORIG.MS .
cp -r /data/lofarschool/data/Exercise-05-DPPP/L2008_09315_SB12_ORIG_CASAVIEWER.MS .

The executable for DPPP is named CS1_IDPPP. Once you have set the environment as above, it will be by default visible to you. You can cross check it by typing the command:

which CS1_IDPPP # should print: /users/renting/LOFAR/installed/gnu/bin/CS1_IDPPP

Inspecting observed data

The measurement set you picked is one of the 46 subbands of the LBA (Low Band Antenna) measurement called L2007_09315.MS. This measurement was performed on October 24, 2008. It is a 3-hour measurement (14:04hrs to 17:04hrs UTC) (about 466 timeslots of 30 s. each) using 12 single antennas, targeted at the North celestial pole. The selected subband 12 has been taken as original except that auto-correlations have been flagged. It has 256 contiguous channels, whose total bandwidth is approx. 195 KHz centered around 39.452 MHz.

We will start by inspecting the original measurement set. Use any of your favorite tools for data inspection and have a look at the data. Otherwise for simplicity you can choose to use a simple tool called uvplot. It can be invoked by /app/uvplot/uvplot/uvplot; Once you invoke the tool, you can open your measurement set from the File Menu and afterwards press on the LOAD button. It shows uv data for a single baseline defined by antenna 1 and antenna 2 as shown. You can fiddle with that and also the amplitude scale to inspect data. By default the flagged data is shown in blue. Some help on color scale in uvplot representation can be found in ColorScaleuvplot. You can see that all the autocorrelations have been flagged. This is a simple tool and you are free to choose other ones as well. One can also choose casaviewer. It can be invoked directly as /app/Casa/x86_64/bin/casaviewer. NB. Please only use casaviewer on L2008_09315_SB12_ORIG_CASAVIEWER.MS or on the output measurement sets of DPPP. The measurement set clearly shows lot of bad data.

RFI flagging

Next our aim is try to flag bad or RFI infected data from the measurement set. Open the file set_A_CS1_IDPPP.parset and inspect the contents. Please refer to the parameter documentation for a list of supported parameters and their meaning.

Next, copy set_A_CS1_IDPPP.parset to CS1_IDPPP.parset and run the CS1_IDPPP program. (By default, CS1_IDPPP looks for CS1_IDPPP.parset for input and output details.)

cp set_A_CS1_IDPPP.parset CS1_IDPPP.parset

The software will start juggling out a lot of information. It should take about 10-12 minutes to run and give the output. The output will be output measurement set outdp3_seta_L2008_09315_SB12_ORIG.MSs. This measurement set has been flagged by the DPPP program to remove corrupted data. Inspect this masurement set and convince for yourself that this is indeed the case.

Now for Exercise B, C, D also you can replace CS1_IDPPP.parset with set_B_CS1_IDPPP.parset, set_C_CS1_IDPPP.parset, set_D_CS1_IDPPP.parset. After each run, do not forget to inspect the output. In case of B, the same operation as A is carried out followed by compression of the measured visibilities in frequency axis. In case of C, again the same operation as in A has been carried out followed by compression along time axis. In case of D, operation A is followed by compression along time and frequency axis together. More details will be explained during the hands-on session itself.

Clean up

If you want to keep (part of) your work, make sure you copy it to an external machine (for instance using scp). Once this is done, please remove your directory:

rm -rf /data/lofarschool/users/<your directory>

Data Processing School

  • Last modified: 2017-03-08 15:27
  • by