Table of Contents

How to use DTC package: details of each tool

In this section, functionality, usage and parameters are presented. Two main categories of tools are available:

DTC contains 3 sub-packages: CEP2, Stand-alone and Tools sub-packages. this table contains an overview on the tools:

Sub-Package Tool Function
CEP2 Beam2Dynspec-Complete Beam formed data complete conversion to Dynspec format
CEP2 Beam2Dynspec-Rebin Beam formed data conversion to Dynspec formet with a Time/Frequency rebin and/or selection
CEP2 Beam2Dynspec-Quick Dynspec Quicklook generation from Beam formed data
Stand-alone Beam2Dynspec-Standalone-Complete Stand-alone Station Beam formed data conversion to Dynspec formet with a Time/Frequency rebin and/or selection
Stand-alone Beam2Dynspec-Standalone-Rebin Stand-alone Station Beam formed data conversion to Dynspec formet with a Time/Frequency rebin and/or selection
Stand-alone Beam2Dynspec-Standalone-Quick Dynspec Quicklook generation from Beam formed data Stand-alone Station
Tools Dynspec-LinPol Convert a Dynspec with I,Q,U,V polarization to a Dynspec with I, Linear Polarization, Position Angle, Total Polarisation
Tools Dynspec-Rebin Allow to rebin/select in time and frequency a dynspec (avoid to reprocess all original data)
Tools Dynspec-Substract Generate a dynspec which is the subtraction of 2 dynspec: dynspec1-(k.dynspec2) where k is a number
Tools Dynspec-Visu Visualization tool for all dyspec

CEP2/4 sub-package

To use Dynspec on CEP2, execute

use Dynspec
use Pythonlibs
use LUS

For CEP4 there is a docker image. The run command may need to be adapted for write access on /data/.. :

docker-run.sh dynspec:latest /bin/bash
Beam2Dynspec-Complete

This tool does a complete conversion from Beam formed data (ICD3) to Dynspec data (ICD6). It should be used only on small datasets (at least if the resulting file is to be visualized with Dynspec-Visu). This tool is compatible with LOFAR observations generated by CEP, i.e. observations using the core or remote stations, but not for single international stations in standalone mode (See Stand-alone sub-package for that).

If the user types «Beam2Dynspec-Complete», the usage is explained:

MISSING Parameters
Usage: Beam2Dynspec-Complete --id=Lxxxxxx  --obsDIR= --outputDIR= --RAM=(x in Gb) --Npart= --RebinAll=(yes or no) [if Rebinall=yes default => 0: else --NSAP=]

The user has to provide the following parameters:

Optional:

Beam2Dynspec-Rebin

This tool does a conversion from Beam formed data (ICD3) to Dynspec data (ICD6). With this tool, it is possible to select the interesting time and frequency windows and also to rebin time and frequency channels to reduce the size of the dataset. The final dataset will usually be visualized easily. This tool is compatible with LOFAR observations generated by CEP, i.e. observations using the core or remote stations, but not for single international stations in standalone mode (See Stand-alone sub-package for that).

if the user types «Beam2Dynspec-Rebin», the usage is explained:

MISSING Parameters
Usage: Beam2Dynspec-Rebin --id=Lxxx  --obsDIR= --outputDIR=  --tmin= --tmax=  --tscale= --fmin= --fmax= --chanPerSubband= --RAM=(x in Gb) --Npart=  --RebinAll=(yes or no)  
[if Rebinall=yes default => 0: else --NSAP=]

The user has to provide the following parameters:

Rebining and Selection parameters:

Note: Some parameters are automatically checked and adapted to the right value: for example if fmin<Observation frequency, the code will provide a WARNING and fix fmin=minimum observation frequency, the same for the maximum frequency. Similarly, the parameters are corrected if tmin<0 ⇒ tmin=0, and a warning is issued.

Optionnal:

Beam2Dynspec-Quick

This tool generates a quick look of the data. It is recommended to use just after the observation and visualize the data to determine which part of the data is useful, and which part can be removed. The user provides to this tool a percent of the time and of the frequency that he would like to visualize every seconds (the time scale is fixed to 1s) for each subbands (the frequency scale is 1 channel/subband). Every second and subband, the code will take x% of the data for 1 second and y% of the data for 1 subband and average them to obtain the value of the pixel. Usually 1-5 percent of the time and 10-20 percent of the frequency channel are often enough to obtain a good quicklook. This tool is able to generate a quicklook of several hundreds of GByte observation in less than 5 minutes. Quicklooks are produced in jpeg and h5 format (which can be visualize with the DTC visualisation tool). This tool is compatible with LOFAR observations generated by CEP, i.e. observations using the core or remote stations, but not for single international stations in standalone mode (See Stand-alone sub-package for that).

if the user types «Beam2Dynspec-Quick», the usage is explained:

MISSING Parameters
Usage: Beam2Dynspec-Quick --id=Lxxx --obsDIR= --outputDIR= --percentTimeData=[0->1] --percentSpectralData=[0->1] --transpose=[yes or no] --nofPart=

The user has to provide the following parameters:

Stand-alone sub-package

Beam2Dynspec-Standalone-Complete

This tool does a complete conversion from Beam formed data (ICD3) to Dynspec data (ICD6). It should be used only on small datasets (at least if the resulting file is to be visualized with Dynspec-Visu). The Stand-alone package is for Beam formed data obtained on single stations in stand-alone mode. It does exactly the same as «Beam2Dynspec-Complete» but for single station data produced with the «Pelican» system. (Note: We have to test with data from the new system «Lump» system).

if the user types «Beam2Dynspec-Standalone-Complete», the usage is explained:

MISSING Parameters
Usage: ./ICD3-Complete-Standalone.py --id=Dxxxxxxxxxxxxxxxx.0Zxx...xxx  --obsDIR= --outputDIR=  --RAM=(x in Gb)

The user has to provide the following parameters:

Beam2Dynspec-Standalone-Rebin

This tool does a conversion from Beam formed data (ICD3) to Dynspec data (ICD6). With this tool, it is possible to select the interesting time and frequency windows and also to rebin time and frequency channels to reduce the size of the dataset. The final dataset will usually be visualized easily. The Stand-alone package is for Single station Beam formed data observation only. The Stand-alone package is for Beam formed data obtained on single stations in stand-alone mode. It does exactly the same as «Beam2Dynspec-Rebin» but for single station data produced with the «Pelican» system. (Note: We have to test with data from the new system «Lump» system).

If the user types «Beam2Dynspec-Standalone-Rebin», the usage is explained:

MISSING Parameters
MISSING Parameters
Usage: ./ICD3-Rebin-Standalone.py --id=Dxxxxxxxxxxxxxxxx.0Zxx...xxx  --obsDIR= --outputDIR=  --tmin= --tmax=  --tscale= --fmin= --fmax= --chanPerSubband= --RAM=(x in Gb)]

The user has to provide the following parameters:

Rebining and Selection parameters:

N.B: Some parameters are automatically checked and adapted to the right value: for example if fmin<Observation frequency, the code will provide a WARNING and fix fmin=minimum observation frequency, the same for the maximum frequency. Also if tmin<0 ⇒ tmin=0 with a warning.

Beam2Dynspec-Standalone-Quick

This tool generates a quick look of the data. It is recommended to use just after the observation and visualize the data to determine which part of the data is useful, and which part can be removed. The user provide to this tool a percent of the time and of the frequency that he would like to visualize every seconds (the time scale is fixed to 1s) for each subbands (the frequency scale is 1 channel/subband). Actually, every second and subband, the code will take x% of the data for 1 second and y% of the data for 1 subband and average them to obtain the value of the pixel. Usually 1-5 percent of the time and 10-20 percent of the frequency channel are often enough to obtain a good quicklook. This tool is able to generate a quicklook of several hundreds of GByte observation in less than 5 minutes. Quicklook are in jpeg and h5 format (could be visualize with the DTC visualisation tool). It does exactly the same as «Beam2Dynspec-Quick» but for single station data produced with the «Pelican» system. (Note: We have to test with data from the new system «Lump» system).

If the user types «Beam2Dynspec-Quick», the usage is explained:

MISSING Parameters
Usage: Beam2Dynspec-Quick --id=Lxxx --obsDIR= --outputDIR= --percentTimeData=[0->1] --percentSpectralData=[0->1] --transpose=[yes or no]

The user has to provide the following parameters:

Tools sub-package

Dynspec-LinPol

After Dynspec generation with a tool from CEP2 or Stand-alone sub-package, users can convert the I, Q, U, V Dysnpec to I, Linear Polarization, Position Angle, Total Polarization.

If the user types «Dynspec-LinPol», the usage is explained:

MISSING Parameters
Usage: .Dynspec-LinPol --outDir= --ID=[what you want:Determine the output filename] --filename=[Name of the ICD6 file (*.h5)] --RAM=

The user has to provide the following parameters:

Dynspec-Rebin

This tool is for rebinng of a dynspec file (ICD6, producing another ICD6 file). It avoids having to go back to the original beam-formed data when further rebinning is required (i.e. the user wishes a lower time/frequency resolution or a smaller time/frequency windows). The Visualization tool is not able to image large dynspec files. After a first conversion (with or without rebin/selection), it is possible that the user needs to reduce the size of the dynspec for an easy visualization. This tool is very similar to the rebin tools contained in CEP2 and Stand-alone sub-packages, but it rebin dynspec (ICD6) not Beam formed (ICD3) data.

If the user types «Dynspec-LinPol», the usage is explained:

MISSING Parameters
Usage: Dynspec-Rebin --Pathfile=  --obsname=Lxxx[for metadata] --outputFile=[choose the complete output name]  --tmin= --tmax=  --tscale= --fmin= --fmax= --chanPerSubband= 
--RAM=(x in Gb)]

The user has to provide the following parameters:

Rebining and Selection parameters:

Note: Some parameters are automatically checked and adapted to the right value: for example if fmin<Observation frequency, the code will provide a WARNING and fix fmin=minimum observation frequency, the same for the maximum frequency. Also if tmin<0 ⇒ tmin=0 with a warning.

Dynspec-Subtract

This tool is for Dynspec subtraction. Frequently, observations have 2 beam, one on the target, one off-axis to have a measure of the noise, RFI, etc…
This tool does a subtraction of 2 beams which can either be in the same or different dynspec files. The only condition is that the data must have the same format! Finally, it is possible to put a multiplicator for the subtraction like: New_dynspec = dynspec1-(k.dynspec2) where k is a number.
:!:For clarification, before dynspec conversion, beams are in different files. After, beams are in the same dynspec file, and their nomenclature (inside the dynspec file, see hdfview above, or the tutorial) is DYNSPEC_xxx.

If the user types «Dynspec-LinPol», the usage is explained:

MISSING Parameters
Usage: Dynspec-Substract --outDir= --id1=[what you want:Determine the output filename => id1-kxid2 or id1_dynspec1-kxdynspec2; depends if file1 & file2 are the same] 
--file1=[Name of the ICD6 file (*.h5)]  --dynspec1=[Name of the dynspec Group ex: DYNSPEC_000] --id2= --file2= --dynspec2= --k=[scaling factor: dynspec1-(k*dynspec2)] 
--RAM=

The user has to provide the following parameters:

Dynspec-Visu

This is the Visualization tool. It allows to visualize any Dynspec (ICD6) file, and inside the file any DYNSPEC_xxx (i.e Beam) and any Stokes parameters (also DYNSPEC_XXX which has Linear, PA and Total polarization). It is easy to use. There are still very minor visualization bugs (due to an error in matplotlib).
During the conversion or the data processing («Dynspec-Lin» or «Dynspec-Subtract») data are still real data ! But for visualization, we use the first tencile (10%) for visualization. At on time, we generate a cumulative distribution of the frequency channels, and compute the first tencile of this distribution. Then for all pixels at this time, the plotted value is I' = I Real/First tencile. For Q,U and V, it is plotted Q'=Q/I, U/I and V/I. So, for Q, U and V, a percentage of the polarisation is plotted (the same for linear, PA and Total Polarization).

An example of the use of Dynspec-Visu below:
Type in a terminal:

Dynspec-Visu


(click for larger version!)
A grey window appears (in case you are working on a remote server, be sure you have a graphical session opened i.e ssh -XY …).

Click on “File” in the upper left corner.
(click for larger version!)
A tab appears with a new command button: 'Load a Dynamic Spectrum'.

Click on 'Load a Dynamic Spectrum':
(click for larger version!)
and a selecting window appears.
:!: In the current version, the cancel button does not really work (it creates a bad objet in the graphical window). Avoid to use it.

After the file selection, The visualisation structure appears.
(click for larger version!)
At this step, you see on the top left (in the box “Dynamic Spectra”) the different dynamic spectra contained in your file (DYNSPEC_000, DYNSPEC_001, …).

Select a dynamic spectrum by double-clicking its name (e.g. “DYNSPEC_000”):
(click for larger version!)
In the “Metadata” window, the metadata of this dynamic spetrum appear (observation ID, Target, RA, DEC, LOFAR stations involved, frequency and time resolution, etc …). In the “Select Stoke” window just below, you see which Stokes parameters (I, Q, U, V, or I, lin, PA, total polarization) are available for the selected dynamic spctrum. And, in the “Resume selection” window, you can see that DYNSPEC_000 is selected, but no Stokes paramter has been chosen yet.

In the “Select Stokes” window, double-click on the Stokes component you are interested in (e.g. “Stokes: I” for the beginning):
(click for larger version!)
Now, the selected Stokes parameter appears in the “Resume Selection” window. With this, the display buttons (Non-continous and continous) become available in the “Display” section. When your observation covers a continuous frequency band, you will not see a difference. When your observation covers several frequency bands separated by gaps, the behaviour differs:


Click (:!: only once) on either “continuous” or “non-continuous”.
Be patient, data display can be long, especially of your file is > 1 GB. A progress bar indicates approximately what fraction the file is processed (don't be surprised if it does not move for some time, its granularity is low).
(click for larger version!)
On a laptop, a 300 MB file took 20 seconds to load for Stokes I, and 120 seconds for other Stokes parameters (which are normalized by Stokes I, requiring extra calculations). Congratulations, your dynamic spectrum is now loaded!

Click in the dynamic spectrum plot (in the window “Plotting The LOFAR Dynamic Spectrum Data”):
(click for larger version!)
The position of your cursor in time and frequency is indicated below the plot. Also, frequency and time profiles through this point appear on the windows “Plotting TimeS pectra” and “Plotting Frequency Spectra” to the right.

To change the contrast in the main window (“Plotting The LOFAR Dynamic Spectrum Data”), you can change the minimum and maximum values (boxes “Minimum” and “Maximum” just above the plot), and click on the respective “Update” buttons:
(click for larger version!)
:!: Don't believe the value with which the “Minimum” and “Maximum” boxes are initialized.
Now the scale of your dynamic spectrum is changed!

Currently, the color-scale is linear. In some cases, it is useful to use a non-linear color scale. For this, change the value of the “Gamma Correction” (the default value is 1. :!: Use a value close to 1!):
(click for larger version!)
The dynamic spectrum display changed! :!: Don't believe the value with which the “Gamma Correction” box is initialized.
Note: Updating the gamma correction can be slow.

To inverse the time and frequency axes, click on the “Transpose” button:
(click for larger version!)
Time and frequency axis are inverted. Some people prefer to display time-frequency instead of frequency-time.

Now select “DYNSPEC_005”, “Stokes: V” and display with “continuous”:
(click for larger version!)
Now, we visualize DYNSPEC_005 and Stokes V. Note that the scale now goes from -1 to 1. (For Stokes Q, U and V, the display shows Q/I, U/I and V/I.)

Below each of the three plot, a few buttons are displayed.

It is recommended now to do the Tutorial.

Return to main page: dynspec

Continue with Tutorials: