Dynspec Toolkit Container (DTC)

This page is be the central information point for the Dynspec Toolkit Container (DTC). Author: vilchez@astron.nl

DTC (for Dynamic spectrum Toolkit Container) is a complete package of tools to reduce, process and convert Beam formed data (ICD3 format) to Dynspec data (ICD6 format) and, finally, visualize LOFAR dynspec data. LOFAR observations can be recorded in datafiles in the Beam Formed data format (ICD3 format, see below). To convert files from the Beam formed data format to Dynspec data format (ICD6 format, see below), the DTC is required. Dynspec data are written in hdf5 format.

A beam formed observation has (or can have) several SAP (Sub-Array Pointing, i.e. multiple direction, for example one on-source and one off-source), several Beam (e.g. different stations), several Stokes, several Parts (used only for large observation, where the data are split in several parts). To process a given datafile, one has to know the SAP, the Beam, and the corresponding Part. These can be be easily identified through the ICD3 nomenclature: For each observation, there will be one h5 file and one raw file for a given SAP, a given Beam, a given Stoke and a given Part. So for one observation: ID=L123456, SAP n°2, Beam n°6, Stokes Q and part n°0, there will be 2 files:

• Lxxxxxx_SAP002_BEAM006_S1_P000.h5
• Lxxxxxx_SAP002_BEAM006_S1_P000.raw

Within the .raw file, the data are stored in a 2 dimensional matrix (time and frequency).

The DTC package converts ICD3 files to ICD6 format. Here, all Beams and all Stokes are grouped in the same dynspec.h5 file. You note that data are not separated in a raw file anymore, and we obtain one dynspec for one SAP (i.e one dynspec per source). Dynspec nomenclature will be:

• Dynspec_(rebinned or complete)_Lxxxxxx_SAPxxx.h5, where Lxxxxxx is the ID of the observation and SAPxxx is the ID of the SAP.

Within the .h5 file, the data are stored in a 3 dimensional matrix (time,frequency and Stokes).

On CEP, you can use « hdfview » for ICD3 or ICD6 data visualization. To activate it on CEP, just type:

use hdfview

Below, just a screen shot using hdfview of a dynspec file and a Beam formed file. Differences in the file organization between ICD3 and ICD6 format are evident, as are the different data dimensions.

(click for larger version!)

DTC provides a number of possibilites (example of data processing using DTC):

• ICD3 observations (h5+raw data) are copied to a directory
• Use the «Quicklook tool» to obtain a quicklook image (in jpeg format and a dynspec in h5 format). It takes generally less than one minute to generate a quicklook of several hundreds GByte). NB: it possible to use the «Visualization tool» to visualise the quicklook.h5 file.
• If you are interested in a particular event in your dynamic spectrum, you can extractr it. Select the frequency and time range you want to isolate. Because Beam Formed data are usually very voluminous, it is recommended to select only a part of the dynspec and also rebin this part in frequency and in time.
• To convert the Beam formed (ICD3) data (and/or select, rebin your data) to Dynspec (ICD6) data, it is required to use the «Rebin» tool (or «Complete» tool, if you want just a conversion without rebining or selecting).
• Use the «Visualization» tool to visualize the obtained dynspec (this will also allow to save jpeg images of the dynamic spectrum of zoomed versions thereof).

Extra tools (with optional functionality):

• Use the <> tool to do the subtraction of 2 dynspec beams (using Z=X-(k*Y) where k is a free scaling factor). The dynspec obtained in this way can be visualized with the «Visualization» tool.
• Use the «LinPol» tool, to convert your I,Q,U,V dysnpec to I,linear,PA,Total dynspec (only avaible for I,Q,U,V observation of course). The obtained dynspec can be visualized with the «Visualization» tool.
• Use the «Dynspec Rebin» tool to rebin (and/or select) a dynspec. This option allows to reprocess the data without having to start from the beginning (with ICD3 to ICD6 conversion).

Note: The «Visualization» can visualize only dynspec with a size comparable to the available memory (RAM) without swapping.

DTC dependencies, DTC install and the use of DTC on CEP cluster are explained on this web page: DTC install

DTC usage and tool functionalities are described on this web page: DTC usage

If you want to use DTC on CEP, continue reading here: dynspec_cep

DTC usage and tool functionalities for ASTRON. are described on this web page: dynspec_cep_ASTRON

Several tutorials are avaible here on different targets:

• Sun tutorial: The Sun.
• Cas A tutorial: Cas A.
• Jupiter tutorial: Jupiter.