public:user_software:dynspec_tutorial

Dynspec Toolkit Container (DTC): The Sun Tutorial

Use the following link to download the ICD3 Sun data (3.5 GB): DATA

Unzip:

unzip Sun.zip

:!: Throughout this tutorial, the following variables are used:
Observation directory (where all ICD3 files are located - that's the files you just downloaded): $OBS_DIR
Output directory (where the Dynspec will be generated): $OUTPUT_DIR
Dynspec filename: $FILENAME
Best define them now.

:!: Throughout the tutorial, the FULL path must be provided.

Note: DTC use the Linux command find to identify all ICD3 files for a given observation, i.e. data could everywhere in $OBS_DIR (i.e. in different sub-directories). This method allows to launch DTC from a hoover node (which has access to every nodes) when data are on different nodes (e.g. on CEP nodes at the Radio Observatory).

Before process the data, it is important to have observation parameters.

ICD3 nomenclature

We notice that all file name start with L248605*. So, the ID of the observation is: L248605.
Using the ICD3 nomenclature, this observation contains 2 (*.h5+.raw) x 8 files.
We see S0, S1, S2 and S3, so means the observation contains 4 Stokes components.
Then 2 SAP, (SAP000 and SAP001), i.e. 2 targets (in our case: the Sun and a beam offset from the Sun).
We see only files which finished by P000, so there is only one part (the data are not split in time).

HDFVIEW

Using hdfview give to the user access to metadata of the observation.
Open any h5 file with hdfview (all h5 files have exactly the same structure and the same time and frequency metadata). For example, open L248605_SAP000_B000_S0_P000_bf.h5. Click on L248605_SAP000_B000_S0_P000_bf.h5. At the bottom of the hdfview window, the metadata of the observation are shown:

  • OBSERVATION_FREQUENCY_MAX = 67.474365234375
  • OBSERVATION_FREQUENCY_MIN = 19.818115234375
  • OBSERVATION_FREQUENCY_UNIT = MHz
  • OBSERVATION_ID = 248605
  • OBSERVATION_NOF_SUB_ARRAY_POINTINGS = 2
  • OBSERVATION_START_MJD = 56946.55972222222
  • OBSERVATION_START_UTC = 2014-10-16T13:26:00.000000000Z
  • OBSERVATION_STATIONS_LIST = CS002LBA,CS003LBA,CS004LBA,CS005LBA,CS006LBA,CS007LBA
  • TARGETS = Sun,OffSun
  • TOTAL_INTEGRATION_TIME = 299.97662208
  • etc …

Navigate to the subgroup: SUB_ARRAY_POINTING_000/BEAM_000/COORDINATES/COORDINATES_0. This is the time axis (“COORDINATE_TYPE = Time”). The time step is given by INCREMENT = 0.01048576. So the time scale is 0.01048576 second.
Then navigate to the data group: SUB_ARRAY_POINTING_000/BEAM_000/STOKES_0. In the metadata, you now see NOF_CHANNELS = 16,16,…., NOF_SAMPLES = 28608 and NOF_SUBBANDS = 244. The observation thus contains 244 sub-bands of 16 channels each.

Observation overview

At this step, just with the name of the observation and hdfview, we have the main parameters of the observation:

  • Observation ID L248605
  • 2 SAP (Sun and offset)
  • 1 Beam in SAP000 (there is another beam in SAP001, open a L248605_SAP001_B000_S0_P000_bf.h5 to verify)
  • 4 Stokes
  • 1 Part
  • fmin=19.818 MHz
  • fmax=67.474 MHz
  • 16 channels per sub-band
  • 244 sub-bands
  • ~5 minutes duration (299.976 s)
  • time scale: 0.01048576 second/time pixel

Exit hdfview.

To generate a quicklook, we use Beam2Dynspec-Quick.

Launch this command:

Beam2Dynspec-Quick --id=L248605 --obsDIR=$OBS_DIR --outputDIR=$OUTPUT_DIR --percentTimeData=0.01 --percentSpectralData=0.1 --transpose=no --nofPart=1

N.B: If the output directory do not exists, it will be created if PATH in PATH/$OUTPUT_DIR exists !

The following text will appear on your terminal:

vilchez@lce010:/data/scratch/vilchez/Sun$ Beam2Dynspec-Quick --id=L248605 --obsDIR=/data/scratch/vilchez/Sun/L248605/ 
--outputDIR=/data/scratch/vilchez/CasA/Dynspec/ --percentTimeData=0.01 --percentSpectralData=0.1 --transpose=no --nofPart=1

Part 0 Started
-----------------------------------

ICD3 files list etablished and take: 0.0 s

SAP number 0 started
SAP number 1 started

Duration of processing: 0.23 s

plot generation started
Quicklook generation finished and take: 1.33 s
Part 0 Finished
-----------------------------------

Quicklook data generation is finished for All Part and takes 1.34 s

Open the QuickLook_Continous_L248605_SAP000_000_P000.jpg, and check you dynamic spectrum. Note: some observation has a non-continuous bandwidth, that is why you obtain continuous and non-continuous quicklooks. In the present case, *Continous* and the *Non_Continous* files are identical because the bandwidth is continuous.

You can also visualize the newly created files QuickLook_L248605_SAP000_P000.h5 and QuickLook_L248605_SAP001_P000.h5 using Dynspec-Visu which gives you more flexibility than the fixed jpg version. This tool is explained here: Visualization tool.

(click for larger version!) (click for larger version!)

In this tutorial we will focus on a time window of 100-250 s, a frequency window of 30-60 MHz, a time rebin to 0.1s and a frequency rebin to 4 channel/subband.

In case of large observation, it is absolutly not recommended to do a complete converse. In this tutorial, our observation has only 3 GB, so that a full conversion is not unreasonable.

Launch this command:

Beam2Dynspec-Complete --id=L248605  --obsDIR=$OBS_DIR --outputDIR=$OUTPUT_DIR --RAM=1 --Npart=1 --RebinAll=yes

N.B: If the output directory do not exists, it will be created if PATH in PATH/$OUTPUT_DIR exists ! if RebinAll=no, you have to provide NSAP=number of the SAP that you want to process.

The following text will appear on your terminal:

vilchez@lce010:/data/scratch/vilchez/Sun$ Beam2Dynspec-Complete --id=L248605  --obsDIR=/data/scratch/vilchez/Sun/L248605/ 
--outputDIR=/data/scratch/vilchez/Sun/Dynspec/ --RAM=1 --Npart=1 --RebinAll=yes
find /data/scratch/vilchez/Sun/L248605/ -name L248605_SAP*bf.h5  > /data/scratch/vilchez/Sun/Dynspec/ICD3-Complete-list.txt

ICD3 files list etablished and take: 0.0 s

SAP number 0 started
SAP N° 0 and BEAM N° 0 Stokes: 4 and Part: 0 is done
  
SAP number 0 done 
  
SAP number 1 started
SAP N° 1 and BEAM N° 0 Stokes: 4 and Part: 0 is done
  
SAP number 1 done 
  

Duration of processing: 18.71 s

In the output directory, you have 2 Dynspec of 3.5 GB (named Dynspec_complete_L248605_SAP000.h5 and Dynspec_complete_L248605_SAP001.h5). It is possible to visualize these files with Dynspec-Visu.
Note: The visualization of a complete converted dynspec is long (even for this small observation). Be patient, it takes a few (up to 10) minutes (depending the available RAM). (Remember to click ONLY once in the “Display” section ⇒ see Visualization tool)

(click for larger version!)

:!: This dynspec (SAP_000) has a size of 1.7 GB (the total observation of 3.5GB is split in 2 dynspec: the target and the offset). Its data matrix contains about 1.2E8 elements. The visualization tool is limited to 2E8 elements. Thus, the maximum file size which can be visualized by «Dynspec-Visu» is ~2.8 GB, and will take ~5-15 minutes at least to plot it. Visualization of a file after “complete conversion” is not a good idea except for small observation less than ~2 GB.
Below, an example of the error message that appears in case the data file is too large for the visualization tool.
(click for larger version!)

As mentioned in the previous part of the tutorial, using a complete conversion is not really convenient. Thanks to the Quicklook, we defined a region of interest (certainly just doing a select in this case is enough, but in majority of cases a rebin is required) in time and frequency, and also rebin parameters in time and frequency.

Selected parameter:

  • time window of 100-250 s
  • frequency window of 30-60 MHz
  • time rebin to 0.1s
  • frequency rebin of 4 channel/subband

Launch this command:

Beam2Dynspec-Rebin  --id=L248605  --obsDIR=$OBS_DIR --outputDIR=$OUTPUT_DIR --tmin=100 
--tmax=250  --tscale=0.1 --fmin=30 --fmax=60 --chanPerSubband=4 --RAM=1 --Npart=1  --RebinAll=yes

N.B: If the output directory do not exists, it will be created if PATH in PATH/$OUTPUT_DIR exists! if RebinAll=no, you have to provide NSAP=number of the SAP that you want to process.

The following text will appear on your terminal:

vilchez@lce010:/data/scratch/vilchez/Sun$ Beam2Dynspec-Rebin  --id=L248605  --obsDIR=/data/scratch/vilchez/Sun/L248605/ 
--outputDIR=/data/scratch/vilchez/Sun/Dynspec/  --tmin=100 --tmax=250  --tscale=0.1 --fmin=30 --fmax=60 --chanPerSubband=4 --RAM=1 --Npart=1  --RebinAll=yes
find /data/scratch/vilchez/Sun/L248605/ -name L248605_SAP*bf.h5  > /data/scratch/vilchez/Sun/Dynspec/ICD3-Rebin-list.txt

ICD3 files list etablished and take: 0.0 s

SAP number 0 started
SAP N° 0 and BEAM N° 0 Stokes: 4 and Part: 0 is done
  
SAP number 0 done 
  
SAP number 1 started
SAP N° 1 and BEAM N° 0 Stokes: 4 and Part: 0 is done
  
SAP number 1 done 
  

Duration of processing: 3.87 s

Now, in the output directory, you have 2 Dynspec of 15 MB (named Dynspec_rebinned_L248605_SAP000.h5 and Dynspec_rebinned_L248605_SAP001.h5). Files of this size are easy to visualize with Dynspec-Visu!
Compare to the original datafile (e.g. by looking at QuickLook_Continous_L248605_SAP000_000_P000.jpg). Also, try to zoom in, and check if the level of fine structure is sufficient. If now, rebin with different parameters (remember, we just used “–tscale=0.1” and “–chanPerSubband=4”).

The two images below show the dynamic spectrum on the left in Stokes I, on the right in Stokes V. Modify the minimum, the maximum and the Gamma correction (must be ~1) to amplify contrast (see Visualization tool for details).
Note that it takes more time to visualize Stokes Q, U and V than Stokes I. This is due to the fact that Q, U and V are displayed in percent of polarization, i.e. Q/I, U/I and V/I, while the datafile really contains I, Q, U and V. Displaying Q/I, U/I and V/I this requires an extra conversion.

(The Sun Stokes:I, click for larger version!) (The Sun Stokes:V, click for larger version!)

With the previous step of this tutorial, we obtain 2 dynspec (Dynspec_rebinned_L248605_SAP000.h5 and Dynspec_rebinned_L248605_SAP001.h5), which both contain Stokes I, Q, U and V.

Using Dynspec-LinPol, Stokes I, Q, U and V can be converted to I, Linear, PA and total polarisation.

Reminder:

  • Circular = V
  • Linear polarization = sqrt(Q*Q+U*U)
  • Position Angle = 1/2 ArcTg(U/Q) (from p187 Handbook of Pulsar Astronomy)
  • Total Polarization = (Linear+V+PA)/I

Type the following command line in a terminal:

Dynspec-LinPol --outDir=$OUTPUT_DIR --ID=Sun_LinPol --filename=$FILENAME --RAM=1

where $FILENAME should be one of the two newly created *rebinned* files.

The following text will appear on your terminal:

vilchez@lce010:/data/scratch/vilchez/Sun$ Dynspec-LinPol --outDir=/data/scratch/vilchez/Sun/Dynspec/ --ID=Sun_LinPol --filename=/data/scratch/vilchez/Sun/Dynspec/Dynspec_rebinned_L248605_SAP000.h5 --RAM=1
Linear Polarisation Process Finished
  

Duration of processing: 0.12 s

In your output directory, you have a file named: Dynspec_Linear_Polar_Sun_LinPol.h5. Now, visualize the Linear polarization with Dynspec-Visu:

(Sun: linear polarisation, click for larger version!)

Now, we will do a dynspec substraction. To do that we will use complete converted dynspec files. Dynspec-Substract is able to do a substraction like: dynspec1-(k.dynspec2) where k is a number (usually, k=1).
Parameters are:

  • file1= Dynspec_complete_L248605_SAP000.h5 (generated before)
  • file2= Dynspec_complete_L248605_SAP001.h5

Type the following command line in a terminal:

Dynspec-Substract --outDir=$OUTPUT_DIR --id1=Sun --file1=PATH/Dynspec_complete_L248605_SAP000.h5  --dynspec1=DYNSPEC_000 --id2=Sun_offset 
--file2=PATH/Dynspec_complete_L248605_SAP001.h5 --dynspec2=DYNSPEC_000 --k=1 --RAM=1

The following text will appear on your terminal:

vilchez@lce010:/data/scratch/vilchez/Sun$ Dynspec-Substract --outDir=/data/scratch/vilchez/Sun/Dynspec/ --id1=Sun --file1=/data/scratch/vilchez/Sun/Dynspec
/Dynspec_complete_L248605_SAP000.h5  --dynspec1=DYNSPEC_000 --id2=Sun_offset --file2=/data/scratch/vilchez/Sun/Dynspec/Dynspec_complete_
L248605_SAP001.h5 --dynspec2=DYNSPEC_000 --k=1 --RAM=1
Substraction Process Finished
  

Duration of processing: 11.11 s

Now, you have in the output directory a dynspec named: Dynspec_Subsracted_Sun-1xSun_offset.h5. You can see the results with Dynspec-Visu:

(Dynspec of the Sun (ON beam), click for larger version!)
(Dynspec of the Sun offset (OFF beam), click for larger version!)
(Dynspec of the subtraction, click for larger version!)

Using Dynspec-Rebin, you can do a rebin of a dynspec. Do a dynspec rebin directly on the complete substracted dynspec produced in the previous section. For this, we will use the the same parameters used at the beginning of this tutorial, in the section Selection and rebin ICD3 conversion to dynspec :

Selected parameter: * time window of 100-250 s * frequency window of 30-60 MHz * time rebin to 0.1s * frequency rebin of 4 channel/subband**

Type the following command line in a terminal:

Dynspec-Rebin --Pathfile=$FILENAME  --obsname=L248605 --outputFile=$OUTPUT_FILENAME  --tmin=100 --tmax=250  --tscale=0.1 --fmin=30 --fmax=60 --chanPerSubband=4 --RAM=1

The following text will appear on your terminal:

vilchez@lce010:/data/scratch/vilchez/Sun$  Dynspec-Rebin --Pathfile=/data/scratch/vilchez/Sun/Dynspec/Dynspec_Subsracted_Sun-1xSun_offset.h5  --obsname=L248605 
--outputFile=/data/scratch/vilchez/Sun/Dynspec/Dynspec_Substracted_Sun_Rebinned.h5  --tmin=100 --tmax=250  --tscale=0.1 --fmin=30 --fmax=60 --chanPerSubband=4 --RAM=1
Dynspec N° 0 is re-processed

Duration of processing: 11.75 s

You can visualize the resulting file with Dynspec-Visu:
(Dynspec of the subtracted and rebinned dynspec of the Sun, click for larger version!)

Now, we investigated all DTC functionalities.
In case of questions mail to: vilchez@astron.nl

Return to main page: dynspec

Continue with other tutorials:

  • Last modified: 2017-03-08 15:27
  • (external edit)