Mapping Data From MS Headers to Trees

Current MS-oriented trees require some information from the MS to function correctly:

Station positions
Array position (usually set to position of station 1 for simplicity, this is only needed to keep relative UVWs coordinates small, thus retaining a few extra digits of precision.)
Phase center

Note that meqserver does not deal with a MS directly, but rather with an abstracted visibility stream (see StreamControl), consisting of a VisHeader record, a number of VisTile objects, and a footer record. All required information is already present in the VisHeader record -- and is much easier to get at there -- but the issue is still how to move it from the header and into particular nodes in a policy-free way.

I have implemented a very open-ended mechanism for this. Meqserver now embeds the Python interpreter, and can be directed to run a Python script whenever it receives a visheader. The script then takes care of everything else. See MeqServer/test/meqsolve.g for a working example. More specifically:

1. The input record of meqserver (see StreamControl) has one new optional field called python_init. If present, this should give the name (or full path) of a Python script. Here's an example of an input record from meqsolve.g:

inputrec := [ sink_type = 'ms_in',
              ms_name = msname,
              data_column_name = 'DATA',
              tile_size=5,
              selection = [ channel_start_index=1,channel_end_index=1 ],
              python_init = 'read_msvis_header.py' ];

2. The Python script is executed, defines a "header handler" function, and registers this function with the kernel.

3. Whenever a visheader is received, the kernel passes it to the handler function, which takes care of updating relevant nodes.

Here is the script activated by meqsolve.g:

   1 # cvs path: LOFAR/Timba/MeqServer/test/read_msvis_header.py
   2 from Timba import dmi
   3 import meqserver
   4 
   5 def setState (node,**fields):
   6   """helper function to set the state of a node specified by name or
   7   nodeindex""";
   8   rec = dmi.record(state=dmi.record(fields));
   9   if isinstance(node,str):
  10     rec.name = node;
  11   elif isinstance(node,int):
  12     rec.nodeindex = node;
  13   else:
  14     raise TypeError,'illegal node argumnent';
  15   # pass command to kernel
  16   meqserver.mqexec('Node.Set.State',rec);
  17 
  18 def processVisHeader (hdr):
  19   """handler for the visheader""";
  20   # phase center
  21   (ra0,dec0) = hdr.phase_ref;
  22   setState('ra0',value=ra0);
  23   setState('dec0',value=dec0);
  24   # antenna positions
  25   pos = hdr.antenna_pos;
  26   if pos.rank != 2 or pos.shape[0] != 3:
  27     raise ValueError,'incorrectly shaped antenna_pos';
  28   nant = pos.shape[1];
  29   coords = ('x','y','z');
  30   for iant in range(nant):
  31     sn = str(iant+1);
  32     # since some antennas may be missing from the tree,
  33     # ignore errors
  34     try:
  35       for (j,label) in enumerate(coords):
  36         setState(label+'.'+sn,value=pos[j,iant]);
  37     except: pass;
  38   # array reference position
  39   for (j,label) in r
  40   enumerate(coords):
  41     setState(label+'0',value=pos[j,0]);
  42 
  43 # register the handler with meqserver
  44 meqserver.add_header_handler(processVisHeader);

The handler function simply picks apart the visheader, and issues Node.Set.State commands to assign values to specific nodes (which in this case are assumed to be MeqConstants; MeqParms would require minor changes to the script.)

Some further notes on this:

This script is not coupled to a specific MS -- all MS's yield visheaders of the same layout, defining a phase center and a 3xN array of antenna positions -- but it is coupled to meqsolve.g, or rather, the node naming convention employed therein. Hence there is no such beast as a globally applicable script; each tree generation scheme will require its own adaptation.
It is a simple matter to embed such a script as a string in the forest state record. (Still to be implemented, but relatively trivial). This makes a forest entirely self-contained.
Once Python tree generation works, this solution will integrate with it in a very elegant way.

A further point to ponder is the following. We can now execute arbitrary Python code directly inside the kernel! With a bit more interface work, Python can be allowed to talk directly to the forest and to C++ nodes. This can eventually become a very powerful tool for embedding all sorts of intelligence (which C++ is notoriously poor at) directly inside the kernel. We should be moving closer to a situation where C++ is only used for the "heavy lifting" (i.e. node implementation), and all the messy details are taken care of from Python.