Introduction
To encourage interoperability between our various frameworks, and to enable people to write standard "plug-and-play" modules for such things as Jones matrices, it is important to define some standard interfaces and behaviours. I will refer to these by contracts, since Python encourages describing interfaces in terms of guaranteed object behaviour (i.e. contract), rather than specific object types. This page is an attempt to describe a few standard contracts.
I have implemented a class tentatively called MeqMaker which makes use of all the contracts here to put together arbitrary MEs. Since this provides examples of how these contracts are used, I will refer to MeqMaker a lot here.
"Jones" contract
This represents a single Jones matrix. A "Jones" contract is simply a node that computes a scalar or a 2x2 matrix.
"Station Jones" contract
J is a "Station Jones" if for a given station index p, J(p) implements the "Jones" contract.
The most common example is an unqualified node. E.g. ns.G is a "Station Jones" if ns.G(p) are valid nodes representing Jones matrices for each station (p).
The reason to define this interface as a contract and not as simply an unqualified node is because people may want to implement some kind of class to represent station Jones. As long as that class defines the function call operator (which returns a single "Jones"), the class follows the contract. We may need to use this when we implement image-plane effects on extended sources.
Note that Meow.SkyComponent.corrupt() expects a "Station Jones" argument.
"Sky Jones" contract
J is a "Sky Jones set" if it implements these two behaviours:
- J(x) is a "Station Jones", where x is either a (source/direction) name, or an object. (Note that TDL can use object qualifiers: if obj is an object with a valid obj.name attribute, then J(obj) is the same as J(obj.name)).
- J(x,p) is a "Jones", and is the same as J(x)(p).
Note that we do not define what "x" are valid here. The range of "x" is specified by additional contracts, e.g. "Sky Jones for the following list of sources". See "Sky Jones Module" below for examples.
The most common example is an unqualified node. E.g. ns.E is a "Sky Jones" if ns.E(src,p) are valid nodes representing Jones matrices for each source and station.
"Station Pointings" contract
P is a "Station Pointings" if, for a given station index p, P(p) is a node returning a real-valued 2-vector. Note that this may be an absolute pointing, or a pointing offset.
"Module" contract
A "Module" is simply a self-contained unit implementing some functionality (Jones, sky models, etc.) This may be an actual Python module (e.g. oms_gain_models.py), or an object of some class (e.g. Meow.MeowLSM).
A "Module" will typically include a number of TdlOptions that show up in the GUI and in config files. Real Python modules may declare these as TDLCompileOptions/TDLRuntimeOptions; users of the module may trivially "suck" these into submenus, see TdlOptions for details.
The various specialized "Module" contracts below all follow the "Module" contract.
Optional "Module" behaviours
A "Module" may contain a solvable attribute, which is True if the module contains potentially solvable parameters, and False otherwise. If this attribute is missing, True should be assumed. The MeqMaker class can perform some important optimizations when dealing with a mix of solvable and non-solvable modules. Note that the Sky-Jones Module contract, below, provides a potential refinement of this behaviour, in cases where solvable
parameters apply to only a subset of sources.
The value of solvable need not be fixed -- it may in fact depend on the module's compile-time options. The final & effective value of solvable may be updated inside the module's compute_xxx() function (see below). MeqMaker will only interrogate solvable after compute_xxx() has been called.
A "Module" may implement a compile_options() function taking no arguments, and returning a list of compile-time TDLOptions used inside the module.
A "Module" may implement a runtime_options() function taking no arguments, and returning a list of run-time TDLOptions used inside the module.
For real Python modules, neither function is really necessary. The module can simply declare !TDLCompile/RuntimeOptions in its global scope in the usual way. The MeqMaker class will "suck in" these options automatically, and neatly arrange them inside sub-menus. There are two situations in which you may need to implement these functions:
If the "Module" is implemented as a class, where each class instance has its private set of options. See Meow.LSM for an example of this.
- If the module's runtime options are generated on-the-fly during compilation.
Note that declaring global-scope options vs. returning them via compile_/runtime_options() is an either-or proposition. If, e.g., compile_options() is implemented, then MeqMaker will NOT look for global-scope options.
Node naming issues
Generally, we don't want modules making up their own node names in the global node scope, since this can lead to naming conflicts between different modules. For this reason, modules are typically passed a node or nodes argument, which should be used to create all internal nodes. This leaves naming under control of the caller. Any intermediate nodes used in calculations should be created within a subscope of nodes:
ns = nodes.Subscope(); ns.foo << ...
Some Common Module Behaviours
Most module contracts listed below define a compute_xxx() function that implements the bulk of the module's behaviour. This has the following features that are common to all contracts:
the first argument is node (also called Jones where appropriate). This should be an unqualified node which is to be used by the module to create its nodes. Typically, the node argument is used to create per-station or per-baseline nodes, and returned "as is". This may be treated as only a "naming hint" though: the Module may choose to create some kind of class implementing its return value contract, and return an object of that class. In any case the node argument should be used as the "base node" for all nodes created inside the module, since that allows the caller to control naming conventions.
the function argument list MUST end with **kw. We may extend these contracts in the future with additional optional keyword arguments; older modules MUST have a **kw in here to "swallow" these optional arguments.
the function usually has an optional stations or ifrs argument giving a list of stations or ifrs for which the nodes are to be defined. If not supplied (or None), Meow.Context.array.stations() or ifrs() should be used, which gives a list of all stations/ifrs defined by the current MS. The caller may only want nodes for a specific subset of these, hence the stations/ifrs argument. It is not a big sin to ignore this argument and always use Meow.Context.array.stations() or ifrs(), since the only ill-effect is that some extra nodes are defined and presumably left unused.
the function usually has an optional inspectors argument. The module may choose to create its own set of inspector nodes, in which case they should be added here via inspectors.append(). See discussion on "Sky Jones Module", below, for an explanation. If you don't implement inspectors, you can omit this argument from the function definition (since it will be swallowed by **kw). Note that MeqMaker automatically creates inspectors for the return values of most modules.
the function may have an optional label argument, which can be used to specify, e.g., what kind of a Jones term the module is being used to implement (e.g. "G", "E", etc.) If you don't need to know the label, you can omit this argument from the function definition (since it will be swallowed by **kw).
the function may have an optional tags argument, which can be a string or a list of tags. The module may choose to add tags to "interesting" nodes such as MeqParms, if this is the case, then the supplied tags must be added to the nodes' tag list.
- if the module implements solvable parameters, these must be tagged with "solvable".
"Pointing Module" contract
A "Pointing Module" is a "Module" that implements a compute_pointings function with the following signature:
def compute_pointings(node,stations=None,tags=None,inspectors=[],**kw): ... return node;
node, stations and tags are specified in "Some Common Module Behaviours" above.
The return value must be a "Station Pointings", or None. Typically, the node argument is used to create per-station pointings as node(p), and then returned as-is. This is not strictly required, however: the "Pointing Module" may choose to create some kind of class implementing the "Station Pointings" contract, and return an object of that class. In any case the node argument should be used as the "base node" for all nodes created inside the module, since that allows the caller to control naming conventions.
The None return value can be used to indicate things like "no pointing errors".
"Stations Jones Module" contract
A "Station Jones Module" is a "Module" that implements a compute_jones function with the following signature:
def compute_jones (Jones,stations=None,tags=None,inspectors=[],**kw): ... return Jones;
Jones, stations and tags are specified in "Some Common Module Behaviours" above.
The return value must be a "Station Jones", or None. Typically, the Jones argument is used to create per-station Jones terms as Jones(p), and then returned as-is. This is not strictly required, however: the "Station Jones Module" may choose to create some kind of class implementing the "Station Jones" contract, and return an object of that class. In any case the Jones argument should be used as the "base node" for all nodes created inside the module, since that allows the caller to control naming conventions.
The None return value can be used to indicate "no Jones" (i.e. each Jones matrix is an identity, therefore no corruptions need to be applied.)
"Sky Jones Module" contract
A "Sky Jones Module" is a "Module" that implements a compute_jones function with the following signature:
def compute_jones (Jones,sources,stations=None,pointing_offsets=None,tags=None, inspectors=[],solvable_sources=sets.Set(),**kw): ... return Jones;- The return value must be a "Sky Jones".
- Everything else said about "Station Jones Modules" still applies here.
Jones, stations and tags are specified in "Some Common Module Behaviours" above.
The sources argument should be a list of sources (or perhaps simply source names). The only requirement is that the "Sky Jones" returned by this function must be valid for every source object (or every name) in the sources list.
The solvable_sources argument, if supplied, will be a Python set object. If the sky-Jones term is potentially solvable for a subset of the sources, then this set should be updated with the names of those sources (and the module's solvable attribute -- see above -- should be explicitly set to False.) If the sky-Jones term is potentially solvable for all sources, you may ignore this set and simply set the solvable attribute to True.
The MeqMaker class can perform some important optimizations when dealing with a mix of solvable and non-solvable sky-Jones terms.
The optional pointing_offsets argument, if supplied, is a "Stations Pointings", which should be treated as pointing offsets. Modules concerned with station beams should use this argument to introduce pointing errors. Other modules may safely ignore it (and omit it from the function declaration, since it will be swallowed by **kw).
An example "Sky Jones Module" is oms_ionosphere.py (in WH/contrib/OMS/Siamese), which computes a Zeta-Jones. Note that this module illustrates the intended use of the inspectors argument. MeqMaker usually automatically creates inspectors for the Jones terms returned by every module. However, it is not really interesting to look at Zeta-Jones per se: far more informative are the TECs and the ionospheric phase delays that go into making it. The oms_ionosphere module therefore makes its own inspectors for these things, MeqMaker then skips the standard inspector.
"Sky Model" contract
A "Sky Model" is a "Module" that implements a source_list() function with the following signature:
def source_list(**kw):
No arguments are currently passed in, but **kw is mandatory for future-proofing.
The return value must be a list of Meow.SkyComponent objects.
Optional "Sky Model" behaviours
A "Sky Model" may optionally implement an estimate_image_size() function with the following signature:
def estimate_image_size(**kw):
No arguments are currently passed in, but **kw is mandatory for future-proofing.
- The return value should be a recommended image size, in arc minutes, for encompassing all the objects in the sky model.
There are currently two "Sky Model" examples:
- Meow.LSM objects, which provide a "Sky Model"-compliant wrapper around the standard LSM code.
gridded_skies.py (in WH/contrib/OMS/Siamese)
Gridded_skies does implement estimate_image_size(), LSM currently doesn't
"Visibility Processing Module" contract
A VPM is a "Module" that implements a process_visibilities() function with the following signature:
def process_visibilities(nodes,input_nodes,ifrs=None,tags=None,inspectors=[],**kw):
- The return value can be None to indicate that no processing has been applied (and no new nodes created, presumably). Any other return value indicates success.
input_nodes is an unqualified node that must be qualified with an ifr (station pair) to obtain a visibility node. If the ifrs argument is supplied, then only ifrs in the list are guaranteed to provide initialized visibilty nodes. If the ifrs argument is not supplied, Context.array.ifrs() must be used.
nodes is an unqualified node for output visibilities. nodes must be qualified with an ifr, and output visibilities must be assigned to the resulting node.
tags and inspectors are specified in "Some Common Module Behaviours" above.
VPMs may be used for any function that operates on visibilities. This can be, e.g., a flagger, or a module implementing interferometer-based errors.
Reversible VPMs
An "Reversible VPM" additionally implements a correct_visibilities() function with the following signature:
def correct_visibilities(nodes,input_nodes,ifrs=None,tags=None,inspectors=[],**kw):
This function is meant to invert the processing applied by 'process_visibilities()`. For example, ifr-based errors are invertable, so MeqMaker can call this function when generating corrected visibilities. On the other hand, a flagger module would not implement this function at all.
All arguments are the same as for process_visibilities() above.
"Sky Visibility Processing Module" contract
A SVPM is a "Module" that implements a process_visibilities() function with the following signature:
def process_visibilities(nodes,input_nodes,sources,ifrs=None,tags=None,inspectors=[],**kw):
The sources argument should be a list of sources (or perhaps simply source names).
input_nodes is an unqualified node that must be qualified with a source and an ifr (station pair) to obtain a visibility node. If the ifrs argument is supplied, then only ifrs in the list are guaranteed to provide initialized visibilty nodes. If the ifrs argument is not supplied, Context.array.ifrs() must be used.
- All other behaviour as as per the "Visibility Processing Module" contract above.
SVPMs may be used for any function that operates on per-source visibilities. This can be, e.g., a module that applies smearing correction, etc.
Reversible SVPMs
An "Reversible SVPM" additionally implements one (or both) functions with the following signature:
def correct_visibilities(nodes,input_nodes,sources,ifrs=None,tags=None,inspectors=[],**kw): def correct_visibilities_for_direction(nodes,input_nodes,source,ifrs=None,tags=None,inspectors=[],**kw):
These functions are meant to invert the processing applied by 'process_visibilities()`. For example, ifr-based errors are invertable, so MeqMaker can call this function when generating corrected visibilities.
The first version corrects visibilities given by input_nodes(src,*ifr) for all sources in the list, and assigns the result to nodes(src,*ifr)
The second version corrects visibilities given by input_nodes(*ifr) for the direction of the given source. This is used to apply aky-based corrections when inverting measurement equations. Note that sematically, correct_visibilities(nodes,input_nodes,sources) should be equivalent to:
1 for src in sources: 2 correct_visibilities_for_direction(nodes(src),input_nodes(src),src);
All other arguments are the same as for process_visibilities() above.