Prefactor - The LOFAR pre-facet calibration pipeline

The official repository of prefactor is now on GitHub: https://github.com/lofar-astron/prefactor

I consists of parsets for the genericpipeline that do the first calibration of LOFAR data. Originally in order to prepare said data for the Factor facet calibration (https://github.com/lofar-astron/factor), but also useful if you don't plan to run Factor.

It includes:

• clock-TEC separation with transfer of clock from the calibrator to the target
• some flagging and averaging of amplitude solutions
• diagnostic plots
• at least some documentation

There are several pipeline parsets in this repository:

• Pre-Facet-Cal.parset : The “standard” pre-facet calibration pipeline, works on pre-NDPPP'ed data
• Pre-Facet-Cal-RawData-Single.parset : A pre-facet pipeline to work on raw (non NDPPP'ed) data
• Pre-Facet-Cal-RawData-PreAvg.parset : A pre-facet pipeline to work on raw (non NDPPP'ed) data that does the subband concatenating in the first NDPPP step. (To reduce the number of files on systems where this is a problem, e.g. JURECA)
• Initial-Subtract.parset : A pipeline that generates full FoV images and subtracts the sky-models from the visibilities. (Needed for facet-calibration, this could also be done as the first step of Factor.)

• the full “offline” LOFAR software installation version >= 2.15
  (With small modifications the Pre-Facet-Cal pipelines can be run with older versions, but that is not supported by the authors anymore.)
• LoSoTo
• Python-PP (see http://www.parallelpython.com/ or https://pypi.python.org/pypi/pp )
• Python matplotlib
• WSClean (for Initial-Subtract, version >=1.9)

There is a preliminary version of a cookbook chapter at: https://github.com/lofar-astron/prefactor/blob/pdf-doc/docs/cookbook_prefacet.pdf Please read that first!

• Don't edit the original parset files directly. Make a copy with a descriptive name (e.g. Pre-Facet-Cal-calibrator-3c295.parset) and edit that copy.
• Get someone to write a better documentation for the genericpipeline.

KeyError 'mapfile' : Your pipeline run fails like that:

2016-02-07 14:48:58 ERROR   genericpipeline: *******************************************
2016-02-07 14:48:58 ERROR   genericpipeline: Failed pipeline run: Pre-Facet-Cal
2016-02-07 14:48:58 ERROR   genericpipeline: Detailed exception information:
2016-02-07 14:48:58 ERROR   genericpipeline: <type 'exceptions.KeyError'>
2016-02-07 14:48:58 ERROR   genericpipeline: 'mapfile'
2016-02-07 14:48:58 ERROR   genericpipeline: *******************************************

That happens when one step didn't generate a mapfile. Usually that means that the pipeline was looking for its input data, but couldn't find any files that match. Please check your *_input_path and *_input_pattern in the parset file! (Note: ls -d *_input_path/*_input_pattern should find your data.)

PipelineStep_* missing : Your pipeline run fails like that:

2016-02-04 13:33:56 ERROR   genericpipeline: *******************************************
2016-02-04 13:33:56 ERROR   genericpipeline: Failed pipeline run: Pre-Facet-Cal
2016-02-04 13:33:56 ERROR   genericpipeline: Detailed exception information:
2016-02-04 13:33:56 ERROR   genericpipeline: <type 'exceptions.ImportError'>
2016-02-04 13:33:56 ERROR   genericpipeline: No module named PipelineStep_createMapfile
2016-02-04 13:33:56 ERROR   genericpipeline: *******************************************

(The exact name of the missing module varies.) You are probably missing one of the entries in the recipe_directories setting in your pipeline.cfg, or one of those entries doesn't work. Make sure both entries point to the correct directories, and that the missing module can be found in the ''plugins' subdirectory of one of those two entries.