Dynspec Toolkit Container (DTC): Cas A Tutorial

This page provide a tutorial for DTC users. DTC is documented on this page: http://www.lofar.org/operations/doku.php?id=dynspec

This tutorial follows exactly the same structure as the Sun tutorial. No extra functions are presented.

Use the following link to download the ICD3 Cas A (3.5 GB): DATA

Unzip:

unzip CasA.zip

Throughout this tutorial, the following variables are used:
Observation directory (where all ICD3 files are located - that's the files you just downloaded): $OBS_DIR
Output directory (where the Dynspec will be generated): $OUTPUT_DIR
Dynspec filename: $FILENAME
Best define them now.

Throughout the tutorial, the FULL path must be provided.

Note: DTC use the Linux command find to identify all ICD3 files for a given observation, i.e. data could everywhere in $OBS_DIR (i.e. in different sub-directories). This method allows to launch DTC from a hoover node (which has access to every nodes) when data are on different nodes (e.g. on CEP nodes at the Radio Observatory).

Before process the data, it is important to have observation parameters.

We notice that all file name start with L248607*. So, the ID of the observation is: L248607.
Using the ICD3 nomenclature, this observation contains 2 (*.h5+.raw) x 8 files.
We see S0, S1 ,S2 and S3, so means the observation contains 4 Stokes components.
Then 2 SAP, (SAP000 and SAP001), i.e. 2 targets (in our case: the Cas A and a beam offset from Cas A).
We see only files which finished by P000, so there is only one part (the data are not split in time).

Using hdfview give to the user access to metadata of the observation.
Open any h5 file with hdfview (all h5 files have exactly the same structure and the same time and frequency metadata). For example, open L248607_SAP000_B000_S0_P000_bf.h5. Click on L248607_SAP000_B000_S0_P000_bf.h5, appears metadata:

• OBSERVATION_FREQUENCY_MAX = 67.474365234375
• OBSERVATION_FREQUENCY_MIN = 19.818115234375
• OBSERVATION_FREQUENCY_UNIT = MHz
• OBSERVATION_ID = 248607
• OBSERVATION_NOF_SUB_ARRAY_POINTINGS = 2
• OBSERVATION_START_MJD = 56946.555555555555
• OBSERVATION_START_UTC = 2014-10-16T13:20:00.000000000Z
• OBSERVATION_STATIONS_LIST = CS002LBA,CS003LBA,CS004LBA,CS005LBA,CS006LBA,CS007LBA
• TARGETS = CasA,OffCasA
• TOTAL_INTEGRATION_TIME = 299.97662208
• etc …

Navigate to the subgroup: SUB_ARRAY_POINTING_000/BEAM_000/COORDINATES/COORDINATES_0. This is the time axis (“COORDINATE_TYPE = Time”). The time step is given by INCREMENT = 0.01048576. So the Time scale is 0.01048576 second.
Then navigate to the data group: SUB_ARRAY_POINTING_000/BEAM_000/STOKES_0. It data metadata, you see NOF_CHANNELS = 16,16,…., NOF_SAMPLES = 28608 and NOF_SUBBANDS = 244. The observation thus contains 244 sub-bands of 16 channels each.

At this step, just with the name of the observation and hdfview, we have the main parameters of the observation:

• Observation ID L248607
• 2 SAP (Cas A and offset)
• 1 Beam in SAP000 (there is another beam in SAP001, open a L248607_SAP001_B000_S0_P000_bf.h5 to verify)
• 4 Stokes
• 1 Partition
• fmin=19.818 MHz
• fmax=67.474 MHz
• 16 channels/sub-band
• 244 sub-bands
• ~5 minutes duration (299.976 s)
• time scale: 0.01048576 second/time pixel

Exit hdfview.

To generate a quicklook, we use Beam2Dynspec-Quick.

Launch this command:

Beam2Dynspec-Quick --id=L248607 --obsDIR=$OBS_DIR --outputDIR=$OUTPUT_DIR --percentTimeData=0.01 --percentSpectralData=0.1 --transpose=no --nofPart=1

N.B: If the output directory do not exists, it will be created if PATH in PATH/$OUTPUT_DIR exists !

The following text will appear on your terminal:

vilchez@lce010:/data/scratch/vilchez/CasA$ Beam2Dynspec-Quick --id=L248607 --obsDIR=/data/scratch/vilchez/CasA/L248607/ 
--outputDIR=/data/scratch/vilchez/CasA/Dynspec/ --percentTimeData=0.01 --percentSpectralData=0.1 --transpose=no --nofPart=1


Part 0 Started
-----------------------------------

ICD3 files list etablished and take: 0.01 s

SAP number 0 started
SAP number 1 started

Duration of processing: 0.23 s

plot generation started
Quicklook generation finished and take: 1.33 s
Part 0 Finished
-----------------------------------

Quicklook data generation is finished for All Part and takes 1.34 s

Open the QuickLook_Continous_L248607_SAP000_000_P000.jpg, and check you dynamic spectrum. Note: some observation has a non-continuous bandwidth, that is why you obtain continuous and non-continuous quicklooks. In the present case, *Continous* and the *Non_Continous* files are identical because the bandwidth is continuous.

You can also visualize the newly created files QuickLook_L248607_SAP000_P000.h5 and QuickLook_L248607_SAP001_P000.h5 using Dynspec-Visu which gives you more flexibility than the fixed jpg version. This tool is explained here: Visualization tool.

(click for larger version!) (click for larger version!)

In this tutorial we will focus on a time window of 100-299 s, a frequency window of 20-50 MHz, a time rebin to 0.1s and a frequency rebin to 4 channel/subband.

In case of large observation, it is absolutly not recommended to do a complete converse. In this tutorial, our observation has only 3 GB, so that a full conversion is not unreasonable.

Launch this command:

Beam2Dynspec-Complete --id=L248607  --obsDIR=$OBS_DIR --outputDIR=$OUTPUT_DIR --RAM=1 --Npart=1 --RebinAll=yes

N.B: If the output directory do not exists, it will be created if PATH in PATH/$OUTPUT_DIR exists ! if RebinAll=no, you have to provide NSAP=number of the SAP that you want to process.

The following text will appear on your terminal:

vilchez@lce010:/data/scratch/vilchez/CasA$ Beam2Dynspec-Complete --id=L248607  --obsDIR=/data/scratch/vilchez/CasA/L248607/ 
--outputDIR=/data/scratch/vilchez/CasA/Dynspec/ --RAM=1 --Npart=1 --RebinAll=yes
find /data/scratch/vilchez/CasA/L248607/ -name L248607_SAP*bf.h5  > /data/scratch/vilchez/CasA/Dynspec/ICD3-Complete-list.txt

ICD3 files list etablished and take: 0.01 s

SAP number 0 started
SAP N° 0 and BEAM N° 0 Stokes: 4 and Part: 0 is done
  
SAP number 0 done 
  
SAP number 1 started
SAP N° 1 and BEAM N° 0 Stokes: 4 and Part: 0 is done
  
SAP number 1 done 
  

Duration of processing: 18.42 s

In the output directory, you have 2 Dynspec of 3.5 GB (named Dynspec_complete_L248607_SAP000.h5 and Dynspec_complete_L248607_SAP001.h5). It is possible to visualize these files with Dynspec-Visu.
Note: The visualization of a complete converted dynspec is long (even for this small observation). Be patient, it takes a few (up to 10) minutes (depending the available RAM). (Remember to click ONLY once in the “Display” section ⇒ see Visualization tool)

(click for larger version!)

This dynspec (SAP_000) has a size of 1.7 GB (the total observation of 3.5GB is split in 2 dynspec: the target and the offset). Its data matrix contains about 1.2E8 elements. The visualization tool is limited to 2E8 elements. Thus, the maximum file size which can be visualized by «Dynspec-Visu» is ~2.8 GB, and will take ~5-15 minutes at least to plot it. Visualization of a file after “complete conversion” is not a good idea except for small observation less than ~2 GB.
Below, an example of the error message that appears in case the data file is too large for the visualization tool.
(click for larger version!)

As mentioned in the previous part of the tutorial, using a complete conversion is not really convenient. Thanks to the Quicklook, we defined a region of interest (certainly just doing a select in this case is enough, but in majority of cases a rebin is required) in time and frequency, and also rebin parameters in time and frequency.

Selected parameter:

• time window of 100-299 s
• frequency window of 20-50 MHz
• time rebin to 0.1s
• frequency rebin of 4 channel/subband

Launch this command:

Beam2Dynspec-Rebin  --id=L248607  --obsDIR=$OBS_DIR--outputDIR=$OUTPUT_DIR --tmin=100 
--tmax=250  --tscale=0.1 --fmin=30 --fmax=60 --chanPerSubband=4 --RAM=1 --Npart=1  --RebinAll=yes

N.B: If the output directory do not exists, it will be created if PATH in PATH/$OUTPUT_DIR exists ! if RebinAll=no, you have to provide NSAP=number of the SAP that you want to process.

The following text will appear on your terminal:

vilchez@lce010:/data/scratch/vilchez/CasA/Dynspec$ Beam2Dynspec-Rebin  --id=L248607  --obsDIR=/data/scratch/vilchez/CasA/L248607/ 
--outputDIR=/data/scratch/vilchez/CasA/Dynspec/  --tmin=100 --tmax=250  --tscale=0.1 --fmin=30 --fmax=60 --chanPerSubband=4 --RAM=1 --Npart=1  --RebinAll=yes
find /data/scratch/vilchez/CasA/L248607/ -name L248607_SAP*bf.h5  > /data/scratch/vilchez/CasA/Dynspec/ICD3-Rebin-list.txt

ICD3 files list etablished and take: 0.01 s

SAP number 0 started
SAP N° 0 and BEAM N° 0 Stokes: 4 and Part: 0 is done
  
SAP number 0 done 
  
SAP number 1 started
SAP N° 1 and BEAM N° 0 Stokes: 4 and Part: 0 is done
  
SAP number 1 done 
  

Duration of processing: 3.87 s

Now, in the output directory, you have 2 Dynspec of 15 MB (named Dynspec_rebinned_L248607_SAP000.h5 and Dynspec_rebinned_L248607_SAP001.h5). Files of this size are easy to visualize with Dynspec-Visu!
Compare to the original datafile (e.g. by looking at QuickLook_Continous_L248607_SAP000_000_P000.jpg). Also, try to zoom in, and check if the level of fine structure is sufficient. If now, rebin with different parameters (remember, we just used “–tscale=0.1” and “–chanPerSubband=4”).

The two images below show the dynamic spectrum on the left in Stokes I, on the right in Stokes V. Modify the minimum, the maximum and the Gamma correction (must be ~1) to amplify contrast (see Visualization tool for details).
Note that it takes more time to visualize Stokes Q, U and V than Stokes I. This is due to the fact that Q, U and V are displayed in percent of polarization, i.e. Q/I, U/I and V/I, while the datafile really contains I, Q, U and V. Displaying Q/I, U/I and V/I this requires an extra conversion.

(Cas A Stokes:I, click for larger version!) (Cas A Stokes:V, click for larger version!)

With the previous step of this tutorial, we obtain 2 dynspec (Dynspec_rebinned_L248607_SAP000.h5 and Dynspec_rebinned_L248607_SAP001.h5), which both contain Stokes I, Q, U and V.

Using Dynspec-LinPol, Stokes I, Q, U and V can be converted to I, Linear, PA and total polarisation.

Reminder:

• Circular = V
• Linear polarization = sqrt(Q*Q+U*U)
• Position Angle = 1/2 ArcTg(U/Q) (from p187 Handbook of Pulsar Astronomy)
• Total Polarization = (Linear+V+PA)/I

Dynspec-LinPol --outDir=$OUTPUT_DIR --ID=CasA_LinPol --filename=$FILENAME --RAM=1

where $FILENAME should be one of the two newly created *rebinned* files.

The following text will appear on your terminal:

vilchez@lce010:/data/scratch/vilchez/CasA$ Dynspec-LinPol --outDir=/data/scratch/vilchez/CasA/Dynspec/ --ID=CasA_LinPol --filename=/data/scratch/vilchez/CasA/Dynspec/Dynspec_rebinned_L248607_SAP000.h5 --RAM=1
Linear Polarisation Process Finished
  

Duration of processing: 0.12 s

In your output directory, you have a file named: Dynspec_Linear_Polar_CasA_LinPol.h5. Now, visualize the Linear polarization with Dynspec-Visu:

(Cas A Linear Polarisation, click for larger version!)

Now, we will do a dynspec substraction. To do that we will use complete converted dyncpecs. Dynspec-Substract is able to do a substraction like: dynspec1-(k.dynspec2) where k is a number (usually, k=1).
Parameters are:

• file1= Dynspec_complete_L248607_SAP000.h5 (generated before)
• file2= Dynspec_complete_L248607_SAP001.h5

Type the following command line in a terminal:

Dynspec-Substract --outDir=$OUTPUT_DIR --id1=Sun --file1=PATH/Dynspec_complete_L248607_SAP000.h5  --dynspec1=DYNSPEC_000 --id2=Sun_offset 
--file2=PATH/Dynspec_complete_L248605_SAP001.h5 --dynspec2=DYNSPEC_000 --k=1 --RAM=1

The following text will appear on your terminal:

vilchez@lce010:/data/scratch/vilchez/CasA$ Dynspec-Substract --outDir=/data/scratch/vilchez/CasA/Dynspec/ --id1=CasA --file1=/data/scratch/vilchez/CasA/Dynspec/
Dynspec_complete_L248607_SAP000.h5  --dynspec1=DYNSPEC_000 --id2=CasA_offset --file2=/data/scratch/vilchez/CasA/Dynspec/Dynspec_complete_L248607_SAP001.h5 
--dynspec2=DYNSPEC_000 --k=1 --RAM=1
Substraction Process Finished
  

Duration of processing: 11.21 s

Now, you have in the output directory a dynspec named: Dynspec_Subsracted_CasA-1xCasA_offset.h5. You can see the results with Dynspec-Visu:

(Dynspec of the Cas A, click for larger version!)
(Dynspec of the Cas A offset, click for larger version!)
(Dynspec of the subtraction, click for larger version!)

Using Dynspec-Rebin, you can do a rebin of a dynspec. Do a dynspec rebin directly on the complete substracted dynspec produced in the previous section. For this, we will use the the same parameters used at the beginning of this tutorial, in the section Selection and rebin ICD3 conversion to dynspec :

Selected parameter: * time window of 100-299 s * frequency window of 20-50 MHz * time rebin to 0.1s * frequency rebin of 4 channel/subband**

Type the following command line in a terminal:

Dynspec-Rebin --Pathfile=$FILENAME  --obsname=L248605 --outputFile=$OUTPUT_FILENAME  --tmin=100 --tmax=299  --tscale=0.1 --fmin=20 --fmax=50 --chanPerSubband=4 --RAM=1

The following text will appear on your terminal:

vilchez@lce010:/data/scratch/vilchez/CasA$ Dynspec-Rebin --Pathfile=/data/scratch/vilchez/CasA/Dynspec/Dynspec_Subsracted_CasA-1xCasA_offset.h5  --obsname=L248607 
--outputFile=/data/scratch/vilchez/CasA/Dynspec/Dynspec_Subsracted_CasA_Rebinned.h5  --tmin=100 --tmax=299  --tscale=0.1 --fmin=20 --fmax=50 --chanPerSubband=4 --RAM=1
Dynspec N° 0 is re-processed

Duration of processing: 14.42 s

Now, you can visualize with Dynspec-Visu:
(Dynspec of the subtracted and rebinned dynspec of Cas A, click for larger version!) (Dynspec of the subtracted and rebinned dynspec of Cas A, zoom on a region, click for larger version!)

Now, we investigated all DTC functionalities.
In case of questions mail to: vilchez@astron.nl

Return to main page: dynspec

Continue with other tutorials:

• Sun tutorial: The Sun.
• Jupiter tutorial: Jupiter.