public:user_software:dynspec

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
public:user_software:dynspec [2014-11-13 08:16] – [DTC Tutorials] Arno Schoenmakerspublic:user_software:dynspec [2020-11-04 08:46] (current) Bernard Asabere
Line 1: Line 1:
-====== Dynspec Toolkit Container (DTC)  ======+====== Dynspec Toolkit Container (DTC) ======
  
-This page is be the central information point for the Dynspec Toolkit Container (DTC). +This page is be the central information point for the Dynspec Toolkit Container (DTC). Author: vilchez@astron.nl
-Author: vilchez@astron.nl+
  
 +==== What's DTC ====
  
-====What's DTC==== +DTC (for Dynamic spectrum Toolkit Container) is a complete package of tools to reduce, process and convert Beam formed data (ICD3 format) to Dynspec data (ICD6 format) and, finally, visualize LOFAR dynspec data. LOFAR observations can be recorded in datafiles in the Beam Formed data format (ICD3 format, see below). To convert files from the Beam formed data format to Dynspec data format (ICD6 format, see below), the DTC is required. Dynspec data are written in hdf5 format.
-DTC (for Dynamic spectrum Toolkit Container) is a complete package of tools to reduce, process and convert Beam formed data (ICD3 format) to Dynspec data (ICD6 format) and, finally,  visualize LOFAR dynspec data.  +
-LOFAR observations can be recorded in datafiles in the Beam Formed data format (ICD3 format, see below). To convert files from the Beam formed data format to Dynspec data format (ICD6 format, see below), the DTC is required. Dynspec data are written in hdf5 format. +
  
 +==== ICD3 Format ====
 +
 +A beam formed observation has (or can have) several SAP (Sub-Array Pointing, i.e. multiple direction, for example one on-source and one off-source), several Beam (e.g. different stations), several Stokes, several Parts (used only for large observation, where the data are split in several parts). To process a given datafile, one has to know the SAP, the Beam, and the corresponding Part. These can be be easily identified through the ICD3 nomenclature: For each observation, there will be one h5 file and one raw file for a given SAP, a given Beam, a given Stoke and a given Part. So for one observation: ID=L123456, SAP n°2, Beam n°6, Stokes Q and part n°0, there will be 2 files:
  
-====ICD3 Format==== 
-A beam formed observation has (or can have) several SAP (Sub-Array Pointing, i.e. multiple direction, for example one on-source and one off-source), several Beam (e.g. different stations), several Stokes, several Parts (used only for large observation, where the data are split in several parts). 
-To process a given datafile, one has to know the SAP, the Beam, and the corresponding Part. 
-These can be be easily identified through the ICD3 nomenclature:  
-For each observation, there will be one h5 file and one raw file for a given SAP, a given Beam, a given Stoke and a given Part. So for one observation: ID=L123456, SAP n°2, Beam n°6, Stokes Q and part n°0, there will be 2 files: 
   * Lxxxxxx_SAP002_BEAM006_S1_P000.h5   * Lxxxxxx_SAP002_BEAM006_S1_P000.h5
   * Lxxxxxx_SAP002_BEAM006_S1_P000.raw   * Lxxxxxx_SAP002_BEAM006_S1_P000.raw
 +
 Within the .raw file, the data are stored in a 2 dimensional matrix (time and frequency). Within the .raw file, the data are stored in a 2 dimensional matrix (time and frequency).
  
 +==== ICD6 Format ====
  
 +The DTC package converts ICD3 files to ICD6 format. Here, all Beams and all Stokes are grouped in the same dynspec.h5 file. You note that data are not separated in a raw file anymore, and we obtain one dynspec for one SAP (i.e one dynspec per source). Dynspec nomenclature will be:
  
 +  * Dynspec_(rebinned or complete)_Lxxxxxx_SAPxxx.h5, where Lxxxxxx is the ID of the observation and SAPxxx is the ID of the SAP.
  
-====ICD6 Format==== 
-The DTC package converts ICD3 files to ICD6 format. Here, all Beams and all Stokes are grouped in the same dynspec.h5 file. You note that data are not separated in a raw file anymore, and we obtain one dynspec for one SAP (i.e one dynspec per source). 
-Dynspec nomenclature will be:  
-  * Dynspec_(rebinned or complete)_Lxxxxxx_SAPxxx.h5, where Lxxxxxx is the ID of the observation and SAPxxx is the ID of the SAP.  
 Within the .h5 file, the data are stored in a 3 dimensional matrix (time,frequency and Stokes). Within the .h5 file, the data are stored in a 3 dimensional matrix (time,frequency and Stokes).
  
 +==== Visualization with hdfview ====
  
 +On CEP, you can use << [[http://www.hdfgroup.org/products/java/hdfview/|hdfview]] >> for ICD3 or ICD6 data visualization. To activate it on CEP, just type:
 +<file>
  
 +use hdfview
  
-====Visualization with hdfview==== +</file>
-On CEP, you can use << [[http://www.hdfgroup.org/products/java/hdfview/|hdfview]] >> for ICD3 or ICD6 data visualization. To activate it on CEP, just type:  +
-<file>use hdfview</file> +
-Below, just a screen shot using hdfview of a dynspec file and a Beam formed file.  +
-Differences in the file organization between ICD3 and ICD6 format are evident, as are the different data dimensions.+
  
-{{:engineering:hdfview.png?600|}} (click for larger version!)+Below, just a screen shot using hdfview of a dynspec file and a Beam formed file. Differences in the file organization between ICD3 and ICD6 format are evident, as are the different data dimensions.
  
 +{{:engineering:hdfview.png?600}}(click for larger version!)
  
 +==== DTC capabilities ====
  
- 
-====DTC capabilities==== 
 DTC provides a number of possibilites (example of data processing using DTC): DTC provides a number of possibilites (example of data processing using DTC):
 +
   * ICD3 observations (h5+raw data) are copied to a directory   * ICD3 observations (h5+raw data) are copied to a directory
-  * Use the <<Quicklook tool>> to obtain a quicklook image (in jpeg format and a dynspec in h5 format). It takes generally less than one minute to generate a quicklook of several hundreds GByte). NB: it possible to use the <<Visualization tool>> to visualise the quicklook.h5 file.  +  * Use the <<Quicklook tool>> to obtain a quicklook image (in jpeg format and a dynspec in h5 format). It takes generally less than one minute to generate a quicklook of several hundreds GByte). NB: it possible to use the <<Visualization tool>> to visualise the quicklook.h5 file. 
-  * If you are interested in a particular event in your dynamic spectrum, you can extractr it. Select the frequency and time range you want to isolate. Because Beam Formed data are usually very voluminous, it is recommended to select only a part of the dynspec and also rebin this part in frequency and in time. +  * If you are interested in a particular event in your dynamic spectrum, you can extractr it. Select the frequency and time range you want to isolate. Because Beam Formed data are usually very voluminous, it is recommended to select only a part of the dynspec and also rebin this part in frequency and in time.
   * To convert the Beam formed (ICD3) data (and/or select, rebin your data) to Dynspec (ICD6) data, it is required to use the <<Rebin>> tool (or <<Complete>> tool, if you want just a conversion without rebining or selecting).   * To convert the Beam formed (ICD3) data (and/or select, rebin your data) to Dynspec (ICD6) data, it is required to use the <<Rebin>> tool (or <<Complete>> tool, if you want just a conversion without rebining or selecting).
   * Use the <<Visualization>> tool to visualize the obtained dynspec (this will also allow to save jpeg images of the dynamic spectrum of zoomed versions thereof).   * Use the <<Visualization>> tool to visualize the obtained dynspec (this will also allow to save jpeg images of the dynamic spectrum of zoomed versions thereof).
  
 Extra tools (with optional functionality): Extra tools (with optional functionality):
-  * Use the <<subtraction>> tool to do the subtraction of 2 dynspec beams (using Z=X-(k*Y) where k is a free scaling factor). The dynspec obtained in this way can be visualized with the <<Visualization>> tool.+ 
 +  * Use the <> tool to do the subtraction of 2 dynspec beams (using Z=X-(k*Y) where k is a free scaling factor). The dynspec obtained in this way can be visualized with the <<Visualization>> tool.
   * Use the <<LinPol>> tool, to convert your I,Q,U,V dysnpec to I,linear,PA,Total dynspec (only avaible for I,Q,U,V observation of course). The obtained dynspec can be visualized with the <<Visualization>> tool.   * Use the <<LinPol>> tool, to convert your I,Q,U,V dysnpec to I,linear,PA,Total dynspec (only avaible for I,Q,U,V observation of course). The obtained dynspec can be visualized with the <<Visualization>> tool.
-  * Use the <<Dynspec Rebin>> tool to rebin (and/or select) a dynspec. This option allows to reprocess the data without having to start from the beginning (with ICD3 to ICD6 conversion).      +  * Use the <<Dynspec Rebin>> tool to rebin (and/or select) a dynspec. This option allows to reprocess the data without having to start from the beginning (with ICD3 to ICD6 conversion). 
 + 
 +Note: :!: The <<Visualization>> can visualize only dynspec with a size comparable to the **available memory**  (RAM) without swapping.
  
-Note: :!: The <<Visualization>> can visualize only dynspec with a size comparable to the **available memory** (RAM) without swapping.+==== INSTALL DTC ====
  
 +DTC dependencies, DTC install and the use of DTC on CEP cluster are explained on this web page: [[:public:user_software:dysnpec_install|DTC install]]
  
-====INSTALL DTC==== +==== How to use the DTC package: details of each tool ====
-DTC dependencies, DTC install and the use of DTC on CEP cluster are explained on this web page:  +
-[[http://www.lofar.org/operations/doku.php?id=dysnpec_install | DTC install]]+
  
 +DTC usage and tool functionalities are described on this web page: [[:public:user_software:dynspec_tools|DTC usage]]
  
-====How to use the DTC package: details of each tool==== +==== How to use the DTC package on CEP ====
-DTC usage and tool functionalities are described on this web page: +
-[[http://www.lofar.org/operations/doku.php?id=dynspec_tools | DTC usage]]+
  
-====How to use the DTC package on CEP====  +If you want to use DTC on CEP, continue reading here: [[:public:user_software:dysnpec_cep|dynspec_cep]]
  
-If you want to use DTC on CEP, continue reading here: [[dysnpec_cep|dynspec_cep]]+==== LOFAR Radio Observatory: How to use DTC package (Duty scientist ONLY) ====
  
-====LOFAR Radio Observatory: How to use DTC package (Duty scientist ONLY) ==== +DTC usage and tool functionalities for ASTRON. are described on this web page: [[:operations:dysnpec_cep_ro|dynspec_cep_ASTRON]]
-DTC usage and tool functionalities  for the R.O. are described on this web page: +
-[[operations:dysnpec_cep_ro|dynspec_cep_RO]]+
  
-====DTC Tutorials==== +==== DTC Tutorials ====
-Several tutorials are avaible here on different targets:  +
-  * Sun tutorial: [[dynspec_tutorial | The Sun]]. +
-  * Cas A tutorial: [[dynspec_tutorial_casa | Cas A]]. +
-  * Jupiter tutorial: [[dynspec_tutorial_jupiter| Jupiter]].+
  
 +Several tutorials are avaible here on different targets:
  
 +  * Sun tutorial: [[:public:user_software:dynspec_tutorial|The Sun]].
 +  * Cas A tutorial: [[:public:user_software:dynspec_tutorial_casa|Cas A]].
 +  * Jupiter tutorial: [[:public:user_software:dynspec_tutorial_jupiter|Jupiter]].
  
  
  • Last modified: 2014-11-13 08:16
  • by Arno Schoenmakers