OLAP produces several data formats, which are intended to be replaced by their final format, such as HDF5. The formats below are not officially supported and subject to change without notice.

Data can be recorded as either complex voltages (yielding X and Y polarisations) or one or more stokes. In either case, a sequence of blocks will be stored, each of which consists of a header and data. The header is defined as:

struct header {
  uint32 sequence_number; /* big endian */
  char padding[508];
};

in which sequence_number starts at 0, and is increased by 1 for every block. Missing sequence numbers implies missing data. The padding can have any value and is to be ignored.

Each (pencil) beam produces two files: one containing the X polarisation, and one containing the Y polarisation. The names of these files adhere to the following scheme:

Each file is a sequence of blocks of the following structure:

struct block {
  struct header header;
 
  /* big endian */
  fcomplex voltages[SAMPLES|2][SUBBANDS][CHANNELS];
}

Each (pencil) beam produces one or four files: one containing the Stokes I (power) values, and optionally three files for Stokes Q, U, and V, respectively. The names of these files adhere to the following scheme:

Currently (release 2010-09-20), the Stokes U and V are multiplied by a factor of 1/2, but that will be changed in a subsequent release.

Each file is a sequence of blocks of the following structure:

struct block {
  struct header header;
 
  /* big endian */
  float stokes[SAMPLES|2][SUBBANDS][CHANNELS];
}

Incoherent stokes are stored per subband, with one or four stokes per file, using the following naming convention:

Currently (release 2010-09-20), the Stokes U and V are multiplied by a factor of 1/2, but that will be changed in a subsequent release.

Each file is a sequence of blocks of the following structure:

struct block {
  struct header header;
 
  /* big endian */
  float stokes[STOKES][SAMPLES|2][CHANNELS];
 
  /* format next release:
  float stokes[STOKES][CHANNELS][SAMPLES|2];
  */
}

The order in which the Stokes values are stored is: I, Q, U, V.

A 'float' is a 32-bit IEEE floating point number. An 'fcomplex' is a complex number defined as

struct fcomplex {
  float real;
  float imag;
};

Constants can be computed using the parset file. Below is a translation between the C constants used above and their respective parset keys:

The following routines might be useful when reading raw OLAP data.

Needed if you read data on a machine which used a different endianness. Typically, x86 machines (intel, amd) are little-endian, while the rest (sparc, powerpc, including the BlueGene/P) is big-endian.

uint32_t swap_uint32( uint32_t x )
{
  union {
    char c[4];
    uint32_t i;
  } src,dst;
 
  src.i = x;
  dst.c[0] = src.c[3];
  dst.c[1] = src.c[2];
  dst.c[2] = src.c[1];
  dst.c[3] = src.c[0];
 
  return dst.i;
}
 
/* Do NOT take a float as an argument. An incorrectly read float
   (because it has the wrong endianness) is subject to modification
   by the platform/compiler (normalisation etc). */
float swap_float( char *x )
{
  union {
    char c[4];
    float f;
  } dst;return
 
  dst.c[0] = x[3];
  dst.c[1] = x[2];
  dst.c[2] = x[1];
  dst.c[3] = x[0];
 
  return dst.f;
}

Since the dimensions of the arrays produced by OLAP depend on the parset, it's handy to have access to arrays with variable size. The easiest way is to use C++ and the boost library (which is often installed by default):

#include "boost/multi_array.hpp"
 
int main() {
  /* create an array of floats with 2 dimensions, and initialise it to have dimensions [2][3] */
  boost::multi_array<float,2> myarray(boost::extents[2][3]);
 
  /* getting and setting is the same as with regular C arrays */
  myarray[1][2] = 1.0;
 
  /* note: &myarray[0][0] (or myarray.origin()) is the address of the first element, which can be
     used if the full array needs to be read from disk. */
 
  return 0;
}

See also http://www.boost.org/doc/libs/1_43_0/libs/multi_array/doc/user.html

If you need to use C, things become a bit more cumbersome. You need to roll out your own multi-dimensional array, although you'll have to customise your code for each number of dimensions in order to keep your code readable. For example:

/* create an array of floats with 2 dimensions, max1 and max2 in size respectively */
struct myarray {
  float *data;
  unsigned max1,max2;
};
 
/* return myarray[one][two] */
float get( struct myarray *array, unsigned one, unsigned two ) 
{
  return *(myarray.data + one * myarray.max2 + two);
}
 
/* set myarray[one][two] to value */
void set( struct myarray *array, unsigned one, unsigned two, float value ) 
{
  *(myarray.data + one * myarray.max2 + two) = value;
}
 
int main() {
  /* create an array of floats */
  struct array myarray;
 
  /* allocate the array with dimensions [2][3] */
  myarray.max1 = 2;
  myarray.max2 = 3;
  myarray.data = malloc( myarray.max1 * myarray.max2 * sizeof *myarray );
 
  /* emulate myarray[1][2] = 1.0 */
  set(&myarray,1,2,1.0);
 
  /* note: myarray.data is the address of the first element, which can be used if the full
     array needs to be read from disk. */
 
  /* free the array */
  free( myarray.data );
 
  return 0;
}

Keep in mind that if you need to switch endianness as well, you first need to read into a char array, and convert it to a float array after reading from disk. This is included in the example below.

The following code reads raw complex voltages from disk.

// TODO: Test this code
 
#include "boost/multi_array.hpp"
#include <cstdio>
#include <stdint.h> // for uint32_t. On Windows, use UINT32.
 
struct header {
  uint32_t sequence_number;
  char padding[508];    
};
 
int is_bigendian() {
  union {
    char c[4];
    uint32_t i;
  } u;
 
  u.i = 0x12345678;
  return u.c[0] == 0x12;
}
 
uint32_t swap_uint32( uint32_t x )
{
  union {
    char c[4];
    uint32_t i;
  } src,dst;
 
  src.i = x;
  dst.c[0] = src.c[3];
  dst.c[1] = src.c[2];
  dst.c[2] = src.c[1];
  dst.c[3] = src.c[0];
 
  return dst.i;
}
 
float swap_float( char *x )
{
  union {
    char c[4];
    float f;
  } dst;return
 
  dst.c[0] = x[3];
  dst.c[1] = x[2];
  dst.c[2] = x[1];
  dst.c[3] = x[0];
 
  return dst.f;
}
 
int main()
{
  unsigned SUBBANDS = 248;
  unsigned CHANNELS = 256;
  unsigned SAMPLES  = 768;
 
  struct header header;
  int swap_endian = !is_bigendian();
 
  // the raw_array is read from disk and converted to the float_array
  // the extra dimension [2] covers the real and imaginary parts of each fcomplex
  // the extra dimension [4] covers the size of a float in chars in the raw_array
  boost::multi_array<char,5> raw_array(boost::extents[SAMPLES|2][SUBBANDS][CHANNELS][2][4]);
  boost::multi_array<float,4> float_array(boost::extents[SAMPLES|2][SUBBANDS][CHANNELS][2]);
 
  FILE *f = fopen( "L12345_B000_S0-bf.raw", "rb" );
  if (!f) {
    puts( "Could not open input file." );
    return 1;
  }
 
  while( !feof(f) ) {
    // read header
    if( fread( f, &header, sizeof header, 1 ) < 1 )
      break;
 
    if( swap_endian )
      header.sequence_number = swap_uint32( header.sequence_number );
 
    printf( "Reading block %u...\n", header.sequence_number );
 
    // read data
    if( swap_endian ) {
      if( fread( f, raw_array.origin(), raw_array.num_elements(), 1 ) < 1 )
        break;
 
      // swap all data regardless of array dimensions
      char  *src = raw_array.origin();
      float *dst = float_array.origin();
 
      for( unsigned i = 0; i < float_array.num_elements(); i++ ) {
        *dst = swap_float( src );
        dst++; src += 4;
      }
    } else
      if( fread( f, float_array.origin(), float_array.num_elements(), 1 ) < 1 )
        break;
 
    // process block here
  }
 
  fclose( f );
  return 0;
}