public:documents:raw_olap_data_formats

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
public:documents:raw_olap_data_formats [2010-10-21 14:26] – Jan David Molpublic:documents:raw_olap_data_formats [2017-03-08 15:27] (current) – external edit 127.0.0.1
Line 1: Line 1:
-==== Raw OLAP data formats ====+===== Raw OLAP data formats (obsolete) ====
  
-OLAP produces several data formats, which are intended to be replaced by their final format, such as HDF5. The formats below are not officially supported and subject to change without notice.+OLAP produces several data formats, which are intended to be replaced by their final format, such as HDF5.
  
-=== Beamformed Data ===+===== After 2011-10-24 =====
  
-Beamformed data can be recorded as either complex voltages (yielding X and Y polarisations) or one or more stokes. In either case, a sequence of blocks will be stored, each of which consists of a header and data. The header is defined as:+Files adhere to the following naming scheme: ''Liiiii_SAPsssss_Bbbb_Sz_bf.raw'', with: 
 + 
 +  - ''iiiii'' = SAS observation ID 
 +  - ''sssss'' = Station beam number (SAP) 
 +  - ''bbb'' = Tied-array beam number (TAB) 
 +  - ''z'' = Stokes number 
 + 
 +The stokes numbers are to be interpreted as follows: 
 + 
 +  - Complex Voltages: 
 +     - z = 0 -> Xr (X polarisation, real part) 
 +     - z = 1 -> Xi (X polarisation, imaginary part) 
 +     - z = 2 -> Yr (Y polarisation, real part) 
 +     - z = 3 -> Yi (Y polarisation, imaginary part) 
 +  - Coherent/incoherent Stokes: 
 +     - z = 0 -> I 
 +     - z = 1 -> Q 
 +     - z = 2 -> U 
 +     - z = 3 -> V 
 + 
 +The data is encoded as follows. Each .raw file is a multiple of the following structure. All data is written as big-endian 32-bit IEEE floats. 
 + 
 +<code> 
 +struct block { 
 +  float sample[SUBBANDS][CHANNELS]; 
 +}; 
 +</code> 
 + 
 +The constants used can be derived from the parset: 
 + 
 +<code> 
 +  SUBBANDS = len(parset["Observation.subbandList"]) 
 + 
 +  if (complex voltages || coherent stokes) { 
 + 
 +    CHANNELS = parset["OLAP.CNProc_CoherentStokes.channelsPerSubband"] 
 +    if (CHANNELS == 0) CHANNELS = parset["Observation.channelsPerSubband"] 
 + 
 +  } elif (incoherent stokes) { 
 + 
 +    CHANNELS = parset["OLAP.CNProc_IncoherentStokes.channelsPerSubband"] 
 +    if (CHANNELS == 0) CHANNELS = parset["Observation.channelsPerSubband"] 
 + 
 +  } 
 +</code> 
 + 
 +The sampling rate can be derived as follows: 
 + 
 +<code> 
 +  # clock frequency (f.e. 200 MHz) 
 +  clock_hz = parset["Observation.sampleClock"] * 1.0e6 
 + 
 +  # subband frequency (f.e. 195 kHz) 
 +  base_subband_hz = clock_hz / 1024 
 + 
 +  # channel frequency (f.e. 763 Hz) 
 +  base_nrchannels = parset["Observation.channelsPerSubband"] 
 +  base_channel_hz = base_subband_hz / base_nrchannels 
 + 
 +  if(complex voltages || coherent stokes) { 
 +    cs_temporalintegration = parset["OLAP.CNProc_CoherentStokes.timeIntegrationFactor"] 
 + 
 +    sample_hz = base_channel_hz / cs_temporalintegration 
 + 
 +  } elif(incoherent stokes) { 
 + 
 +    is_temporalintegration = parset["OLAP.CNProc_IncoherentStokes.timeIntegrationFactor"] 
 + 
 +    sample_hz = base_channel_hz / is_temporalintegration 
 +  } 
 +     
 +</code> 
 + 
 +===== Before 2011-10-24 ===== 
 + 
 +Data can be recorded as either complex voltages (yielding X and Y polarisations) or one or more stokes. In either case, a sequence of blocks will be stored, each of which consists of a header and data. The header is defined as:
 <code C> <code C>
 struct header { struct header {
Line 14: Line 89:
 in which sequence_number starts at 0, and is increased by 1 for every block. Missing sequence numbers implies missing data. The padding can have any value and is to be ignored. in which sequence_number starts at 0, and is increased by 1 for every block. Missing sequence numbers implies missing data. The padding can have any value and is to be ignored.
  
-== Complex Voltages ==+==== Complex Voltages ====
  
 Each (pencil) beam produces two files: one containing the X polarisation, and one containing the Y polarisation. The names of these files adhere to the following scheme: Each (pencil) beam produces two files: one containing the X polarisation, and one containing the Y polarisation. The names of these files adhere to the following scheme:
  
-|Lxxxxx_Byyy_S0-bf.raw|X polarisations of beam yyy of observation xxxxx| +|Lxxxxx_Byyy_S0_bf.raw|X polarisations of beam yyy of observation xxxxx| 
-|Lxxxxx_Byyy_S1-bf.raw|Y polarisations of beam yyy of observation xxxxx|+|Lxxxxx_Byyy_S1_bf.raw|Y polarisations of beam yyy of observation xxxxx| 
 + 
 +Proposed is the following scheme: 
 + 
 +|Lxxxxx_Byyy_S0_bf.raw|X polarisation (real part) of beam yyy of observation xxxxx| 
 +|Lxxxxx_Byyy_S1_bf.raw|X polarisation (imaginary part) of beam yyy of observation xxxxx| 
 +|Lxxxxx_Byyy_S2_bf.raw|Y polarisation (real part) of beam yyy of observation xxxxx| 
 +|Lxxxxx_Byyy_S3_bf.raw|Y polarisation (imaginary part) of beam yyy of observation xxxxx|
  
 Each file is a sequence of blocks of the following structure: Each file is a sequence of blocks of the following structure:
Line 26: Line 108:
 struct block { struct block {
   struct header header;   struct header header;
 +
 +  /* each block contains SAMPLES samples. The data structure is two samples larger (|2) for
 +     technical reasons, but those two samples do not actually exist, and thus should be read
 +     and immediately discarded. Time should just be incremented SAMPLES samples per block. */
  
   /* big endian */   /* big endian */
 +  // 2010-09-20 release and later:
   fcomplex voltages[SAMPLES|2][SUBBANDS][CHANNELS];   fcomplex voltages[SAMPLES|2][SUBBANDS][CHANNELS];
-}+ 
 +  /* 
 +  // 2010-06-29 release and earlier stored data per subband instead of per beam: 
 +  fcomplex voltages[BEAMS][CHANNELS][SAMPLES|2][POLARIZATIONS]; 
 +  */ 
 +};
 </code> </code>
  
-== Stokes ==+Older releases: 
 +2010-09-20: 
 +  - filenames ended in -bf.raw instead of _bf.raw
  
-Each (pencil) beam produces one or four files: one containing the Stokes I (power) values, and optionally three files for Stokes Q, U, and V, respectively. The names of these files adhere to the following scheme:+==== Coherent Stokes ====
  
-|Lxxxxx_Byyy_S0-bf.raw|Stokes I of beam yyy of observation xxxxx| +Each (pencil) beam produces one or four files: one containing the Stokes I (power) values, and optionally three files for Stokes Q, U, and V, respectively. The names of these files adhere to the following scheme:
-|Lxxxxx_Byyy_S1-bf.raw|Stokes Q of beam yyy of observation xxxxx| +
-|Lxxxxx_Byyy_S2-bf.raw|Stokes U of beam yyy of observation xxxxx| +
-|Lxxxxx_Byyy_S3-bf.raw|Stokes V of beam yyy of observation xxxxx|+
  
-Currently (release 2010-09-20), the Stokes U and V are multiplied by a factor of 1/2, but that will be changed in a subsequent release.+|Lxxxxx_Byyy_S0_bf.raw|Stokes I of beam yyy of observation xxxxx| 
 +|Lxxxxx_Byyy_S1_bf.raw|Stokes Q of beam yyy of observation xxxxx| 
 +|Lxxxxx_Byyy_S2_bf.raw|Stokes U of beam yyy of observation xxxxx| 
 +|Lxxxxx_Byyy_S3_bf.raw|Stokes V of beam yyy of observation xxxxx|
  
 Each file is a sequence of blocks of the following structure: Each file is a sequence of blocks of the following structure:
  
 <code C> <code C>
 +// Since 2011-10-24, Stokes are just a continuous stream of samples:
 +struct block {
 +  float stokes[SAMPLES][SUBBANDS][CHANNELS];
 +};
 +
 +// Before 2011-10-24:
 struct block { struct block {
   struct header header;   struct header header;
 +
 +  /* each block contains SAMPLES samples. The data structure is two samples larger (|2) for
 +     technical reasons, but those two samples do not actually exist, and thus should be read
 +     and immediately discarded. Time should just be incremented SAMPLES samples per block. */
  
   /* big endian */   /* big endian */
 +  // 2010-09-20 release and later:
   float stokes[SAMPLES|2][SUBBANDS][CHANNELS];   float stokes[SAMPLES|2][SUBBANDS][CHANNELS];
-}+ 
 +  /* 
 +  // 2010-06-29 release and earlier stored data per subband instead of per beam: 
 +  fcomplex voltages[BEAMS][CHANNELS][SAMPLES|2][STOKES]; 
 +  */  
 +}; 
 +</code> 
 + 
 +Older releases: 
 +2010-09-20: 
 +  - Values of Stokes U and V are multiplied by 1/2 
 +  - filenames ended in -bf.raw instead of _bf.raw 
 + 
 +==== Incoherent Stokes ==== 
 + 
 +Incoherent stokes are stored per subband, with one or four stokes per file, using the following naming convention: 
 + 
 +|Lxxxxx_SByyy_bf.incoherentstokes|Stokes of subband yyy of observation xxxxx| 
 + 
 +Each file is a sequence of blocks of the following structure: 
 + 
 +<code C> 
 +struct block { 
 +  struct header header; 
 + 
 +  /* each block contains SAMPLES samples. The data structure is two samples larger (|2) for 
 +     technical reasons, but those two samples do not actually exist, and thus should be read 
 +     and immediately discarded. Time should just be incremented SAMPLES samples per block. */\ 
 + 
 +  /* big endian */ 
 +  // 2010-10-25 release and later: 
 +  float stokes[STOKES][CHANNELS][SAMPLES|2]; 
 + 
 +  /* 
 +  // 2010-09-20 release: 
 +  float stokes[STOKES][SAMPLES|2][CHANNELS]; 
 + 
 +  // 2010-06-29 release and earlier: 
 +  float stokes[CHANNELS][SAMPLES|2][STOKES]; 
 +  */ 
 +}; 
 +</code> 
 + 
 +The order in which the Stokes values are stored is: I, Q, U, V. 
 + 
 +Older releases: 
 +2010-09-20: 
 +  - Values of Stokes U and V are multiplied by 1/2 
 +  - filenames ended in -bf.raw instead of _bf.raw 
 +  - data order changed 
 + 
 +==== BFRaw format ==== 
 + 
 +Raw station data can be stored in a format called BFRaw. This format is used for debugging purposes and is not a regular observation mode, it takes more manpower to record it. The BFRaw format is recorded below for those who need to access it. 
 + 
 +A BFRaw file starts with a file header containing the configuration: 
 + 
 +<code C> 
 +struct file_header 
 +{ 
 +  // 0x3F8304EC, also determines endianness 
 +  uint32_t    magic; 
 +  // The number of bits per sample (16) 
 +  uint8_t     bitsPerSample; 
 +  // The number of polarizations (2) 
 +  uint8_t     nrPolarizations; 
 +  // Number of subbands, maximum of 62 
 +  uint16_t    nrSubbands; 
 +  // 155648 (160Mhz) or 196608 (200Mhz) 
 +  uint32_t    nrSamplesPerSubband; 
 +  // Name of the station 
 +  char      station[20]; 
 +  // The sample rate: 156250.0 or 195312.5 .. double (number of samples per second for each subband) 
 +  double   sampleRate; 
 +  // The frequencies within a subband 
 +  double   subbandFrequencies[62]; 
 +  // The beam pointing directions (RA, DEC in J2000) 
 +  double   beamDirections[8][2]; 
 +  // mapping from subbands to beams (SAPs) 
 +  int16_t     subbandToSAPmapping[62]; 
 +  // Padding to circumvent 8-byte alignment 
 +  uint32_t    padding; 
 +}; 
 +</code> 
 + 
 +After the file header, there is a series of blocks until the end of file, configured using values from the file header: 
 + 
 +<code C> 
 +struct block 
 +  // 0x2913D852 
 +  uint32_t      magic; 
 + 
 +  // per-SAP information (up to 8 SAPs can be defined, but typically only 1 is used) 
 + 
 +  // number of samples the signal is shifted to align the station beam to the reference 
 +  // phase center (=Observation.referencePhaseCenter in the parset) 
 +  int32_t       coarseDelayApplied[8];  
 +  // Padding to circumvent 8-byte alignment 
 +  uint8_t       padding[4]; 
 + 
 +  // the sub-sample delay which still has to be compensated for (in seconds), 
 +  // at the beginning and at the end of the block 
 +  double     fineDelayRemainingAtBegin[8]; 
 +  double     fineDelayRemainingAfterEnd[8]; 
 +  // Compatible with TimeStamp class (see below) 
 +  int64_t      time[8]; 
 + 
 +  struct marshalledFlags 
 +  { 
 +    // up to 16 ranges of flagged samples within this block 
 +    uint32_t      nrFlagsRanges; 
 +    struct range 
 +    { 
 +      uint32_t    begin; // inclusive 
 +      uint32_t    end;   // exclusive 
 +    } flagsRanges[16]; 
 +  } flags[8]; 
 + 
 +  std::complex<int16_t> 
 +    samples[fileHeader.nrSubbands][fileHeader.nrSamplesPerSubband][fileHeader.nrPolarizations]; 
 +}; 
 +</code> 
 + 
 +To convert a TimeStamp-compatible int64_t to a C-readable timestamp, use 
 +<code C> 
 +/* clockspeed is in Hz */ 
 +int64 nanoseconds = (int64) (timestamp * 1024 * 1e9 / clockspeed); 
 + 
 +struct timespec ts; 
 +ts.tv_sec  = nanoseconds / 1000000000ULL; 
 +ts.tv_nsec = nanoseconds % 1000000000ULL;
 </code> </code>
  
Line 71: Line 306:
 Constants can be computed using the parset file. Below is a translation between the C constants used above and their respective parset keys: Constants can be computed using the parset file. Below is a translation between the C constants used above and their respective parset keys:
  
-|SAMPLES |The number of time samples in a block           |OLAP.CNProc.integrationSteps|+|SAMPLES |The number of time samples in a block           |OLAP.CNProc.integrationSteps / OLAP.Stokes.integrationSteps|
 |SUBBANDS|The number of subbands (beamlets) specified     |len(Oberservation.subbandList)| |SUBBANDS|The number of subbands (beamlets) specified     |len(Oberservation.subbandList)|
 |CHANNELS|The number of channels per subband              |Observation.channelsPerSubband| |CHANNELS|The number of channels per subband              |Observation.channelsPerSubband|
 +|STOKES  |The number of stokes calculated (1 or 4)        |len(OLAP.Stokes.which)|
  
 ==== Useful routines ==== ==== Useful routines ====
Line 84: Line 320:
  
 <code C> <code C>
-uint32 swap_uint32( uint32 x )+#include <stdint.h> // for uint32_t. On Windows, use UINT32. 
 + 
 +uint32_t swap_uint32( uint32_t x )
 { {
   union {   union {
     char c[4];     char c[4];
-    uint32 i;+    uint32_t i;
   } src,dst;   } src,dst;
  
Line 108: Line 346:
     char c[4];     char c[4];
     float f;     float f;
-  } dst;return+  } dst;
  
   dst.c[0] = x[3];   dst.c[0] = x[3];
Line 133: Line 371:
   myarray[1][2] = 1.0;   myarray[1][2] = 1.0;
  
-  /* note: &myarray[0][0] is the address of the first element, which can be used if the full +  /* note: &myarray[0][0] (or myarray.origin()) is the address of the first element, which can be 
-     array needs to be read from disk. */+     used if the full array needs to be read from disk. */
  
   return 0;   return 0;
Line 192: Line 430:
  
 <code C> <code C>
-// TODO: Test this code 
- 
 #include "boost/multi_array.hpp" #include "boost/multi_array.hpp"
-#include <stdint.h> // for uint32. On Windows, use UINT32.+#include <cstdio> 
 +#include <stdint.h> // for uint32_t. On Windows, use UINT32.
  
 struct header { struct header {
-  uint32 sequence_number;+  uint32_t sequence_number;
   char padding[508];       char padding[508];    
 }; };
Line 204: Line 441:
 int is_bigendian() { int is_bigendian() {
   union {   union {
-    char ct[4]; +    char c[4]; 
-    uint32 i;+    uint32_t i;
   } u;   } u;
  
   u.i = 0x12345678;   u.i = 0x12345678;
   return u.c[0] == 0x12;   return u.c[0] == 0x12;
 +}
 +
 +uint32_t swap_uint32( uint32_t x )
 +{
 +  union {
 +    char c[4];
 +    uint32_t i;
 +  } src,dst;
 +
 +  src.i = x;
 +  dst.c[0] = src.c[3];
 +  dst.c[1] = src.c[2];
 +  dst.c[2] = src.c[1];
 +  dst.c[3] = src.c[0];
 +
 +  return dst.i;
 } }
  
Line 217: Line 470:
     char c[4];     char c[4];
     float f;     float f;
-  } dst;return+  } dst;
  
   dst.c[0] = x[3];   dst.c[0] = x[3];
Line 229: Line 482:
 int main() int main()
 { {
-  unsigned NRSUBBANDS = 248; +  // example file (60MB!) is available at 
-  unsigned NRCHANNELS = 256; +  // http://www.astron.nl/~mol/L09330_B000_S0-example-stokes-I-248-subbands-16-channels-763-samples.raw 
-  unsigned NRSAMPLES  = 768;+ 
 +  unsigned SUBBANDS = 248;        // |Observation.subbandList| 
 +  unsigned CHANNELS = 16;         // Observation.channelsPerSubband 
 +  unsigned SAMPLES  = 12208 / 16; // OLAP.CNProc.integrationSteps / OLAP.Stokes.integrationSteps 
 +  unsigned FLOATSPERSAMPLE = 1;   // 1 for Stokes, 2 for Complex Voltages (real and imaginary parts)
  
   struct header header;   struct header header;
Line 237: Line 494:
  
   // the raw_array is read from disk and converted to the float_array   // the raw_array is read from disk and converted to the float_array
-  // the extra dimension [2] covers the real and imaginary parts of each fcomplex 
   // the extra dimension [4] covers the size of a float in chars in the raw_array   // the extra dimension [4] covers the size of a float in chars in the raw_array
-  boost::multi_array<char,4> raw_array(boost::extents[NRSAMPLES|2][NRSUBBANDS][NRSAMPLES][2][4]); +  boost::multi_array<char,5> raw_array(boost::extents[SAMPLES|2][SUBBANDS][CHANNELS][FLOATSPERSAMPLE][4]); 
-  boost::multi_array<float,3> float_array(boost::extents[NRSAMPLES|2][NRSUBBANDS][NRSAMPLES][2]);+  boost::multi_array<float,4> float_array(boost::extents[SAMPLES|2][SUBBANDS][CHANNELS][FLOATSPERSAMPLE]);
  
-  FILE *f = fopen( "L12345_B000_S0-bf.raw", "rb" );+  FILE *f = fopen( "L09330_B000_S0-example-stokes-I-248-subbands-16-channels-763-samples.raw", "rb" );
   if (!f) {   if (!f) {
     puts( "Could not open input file." );     puts( "Could not open input file." );
Line 249: Line 505:
  
   while( !feof(f) ) {   while( !feof(f) ) {
 +    // read header
     if( fread( f, &header, sizeof header, 1 ) < 1 )     if( fread( f, &header, sizeof header, 1 ) < 1 )
       break;       break;
 +
 +    if( swap_endian )
 +      header.sequence_number = swap_uint32( header.sequence_number );
  
     printf( "Reading block %u...\n", header.sequence_number );     printf( "Reading block %u...\n", header.sequence_number );
  
 +    // read data
     if( swap_endian ) {     if( swap_endian ) {
       if( fread( f, raw_array.origin(), raw_array.num_elements(), 1 ) < 1 )       if( fread( f, raw_array.origin(), raw_array.num_elements(), 1 ) < 1 )
         break;         break;
  
-      for( unsigned i = 0; i < float_array.num_elements(); i++ ) +      // swap all data regardless of array dimensions 
-        *(float_array.origin()+i) = swap_float( raw_array.origin()+i*4 );+      char  *src = raw_array.origin(); 
 +      float *dst = float_array.origin(); 
 + 
 +      for( unsigned i = 0; i < float_array.num_elements(); i++ ) { 
 +        *dst = swap_float( src ); 
 +        dst++; src += 4; 
 +      }
     } else     } else
       if( fread( f, float_array.origin(), float_array.num_elements(), 1 ) < 1 )       if( fread( f, float_array.origin(), float_array.num_elements(), 1 ) < 1 )
         break;         break;
 +
 +    // process block here
   }   }
  
Line 269: Line 538:
 } }
 </code> </code>
 +
 +==== Changelog for each release ====
 +
 +|2010-10-25|Incoherent Stokes data order changed|
 +|          |File naming scheme changed (-bf -> _bf)|
 +|          |Stokes U and V are no longer multiplied by 1/2|
 +|2010-09-20|First release documented|
  • Last modified: 2010-10-21 14:26
  • by Jan David Mol