Prefactor - The LOFAR pre-facet calibration pipeline

The official repository of prefactor is now on GitHub: https://github.com/lofar-astron/prefactor

I consists of parsets for the genericpipeline that do the first calibration of LOFAR data. Originally in order to prepare said data for the Factor facet calibration (https://github.com/lofar-astron/factor), but also useful if you don't plan to run Factor.

It includes:

• clock-TEC separation with transfer of clock from the calibrator to the target
• some flagging and averaging of amplitude solutions
• diagnostic plots
• at least some documentation

There are several pipeline parsets in this repository:

• Pre-Facet-Cal.parset : The “standard” pre-facet calibration pipeline, works on pre-NDPPP'ed data
• Pre-Facet-Cal-RawData-Single.parset : A pre-facet pipeline to work on raw (non NDPPP'ed) data
• Pre-Facet-Cal-RawData-PreAvg.parset : A pre-facet pipeline to work on raw (non NDPPP'ed) data that does the subband concatenating in the first NDPPP step. (To reduce the number of files on systems where this is a problem, e.g. JURECA)
• Initial-Subtract.parset : A pipeline that generates full FoV images and subtracts the sky-models from the visibilities. (Needed for facet-calibration, this could also be done as the first step of Factor.)

• the full “offline” LOFAR software installation version >= 2.15
  (With small modifications the Pre-Facet-Cal pipelines can be run with older versions, but that is not supported by the authors anymore.)
• LoSoTo
• Python-PP (see http://www.parallelpython.com/ or https://pypi.python.org/pypi/pp )
• Python matplotlib
• WSClean (for Initial-Subtract, version >=1.9)

There is a preliminary version of a cookbook chapter at: https://github.com/lofar-astron/prefactor/blob/pdf-doc/docs/cookbook_prefacet.pdf Please read that first!

• Don't edit the original parset files directly. Make a copy with a descriptive name (e.g. Pre-Facet-Cal-calibrator-3c295.parset) and edit that copy.
• Get someone to write a better documentation for the genericpipeline.

KeyError 'mapfile' : Your pipeline run fails like that:

2016-02-07 14:48:58 ERROR   genericpipeline: *******************************************
2016-02-07 14:48:58 ERROR   genericpipeline: Failed pipeline run: Pre-Facet-Cal
2016-02-07 14:48:58 ERROR   genericpipeline: Detailed exception information:
2016-02-07 14:48:58 ERROR   genericpipeline: <type 'exceptions.KeyError'>
2016-02-07 14:48:58 ERROR   genericpipeline: 'mapfile'
2016-02-07 14:48:58 ERROR   genericpipeline: *******************************************

That happens when one step didn't generate a mapfile. Usually that means that the pipeline was looking for its input data, but couldn't find any files that match. Please check your *_input_path and *_input_pattern in the parset file!