The official repository of prefactor is now on GitHub: https://github.com/lofar-astron/prefactor
It consists of parsets for the genericpipeline that do the first calibration of LOFAR data. Originally in order to prepare said data for the Factor facet calibration (https://github.com/lofar-astron/factor), but also useful if you don't plan to run Factor.
It includes:
There are several pipeline parsets in this repository:
Installation and setup is explained at the GitHub page: https://github.com/lofar-astron/prefactor
Please read that first!
Pre-Facet-Cal-calibrator-3c295.parset
) and edit that copy. runtime_directory
and working_directory
entries in the pipeline.cfg
for different pipeline runs. The pipeline framework will generate sub-directories with the job-name in there.recipe_directories = [%(pythonpath)s/lofarpipe/recipes,/home/rvweeren/software/prefactor]
BLAS Core affinity : Your pipeline runs slow. All NDPPP-/BBS-/whatever- processes use only little CPU time and only one core of the node is busy.
On clusters like CEP-3 the OpenBLAS library is built with threading affinity. This means that by default the different processes all try to use the same core(s). The use LofIm
and use Lofar
scripts set an environment variable that disables this threading affinity, but if the pipeline.cfg
file does not have the [remote]
section included, then this environment variable is not forwarded to the processes that are started by the pipeline.
So please set the [remote]
section in your pipeline.cfg
.
KeyError 'mapfile' : Your pipeline run fails like that:
2016-02-07 14:48:58 ERROR genericpipeline: ******************************************* 2016-02-07 14:48:58 ERROR genericpipeline: Failed pipeline run: Pre-Facet-Cal 2016-02-07 14:48:58 ERROR genericpipeline: Detailed exception information: 2016-02-07 14:48:58 ERROR genericpipeline: <type 'exceptions.KeyError'> 2016-02-07 14:48:58 ERROR genericpipeline: 'mapfile' 2016-02-07 14:48:58 ERROR genericpipeline: *******************************************
That happens when one step didn't generate a mapfile. Usually that means that the pipeline was looking for its input data, but couldn't find any files that match. Please check your *_input_path
and *_input_pattern
in the parset file! (Note: ls -d *_input_path/*_input_pattern
should find your data.)
PipelineStep_* missing : Your pipeline run fails like that:
2016-02-04 13:33:56 ERROR genericpipeline: ******************************************* 2016-02-04 13:33:56 ERROR genericpipeline: Failed pipeline run: Pre-Facet-Cal 2016-02-04 13:33:56 ERROR genericpipeline: Detailed exception information: 2016-02-04 13:33:56 ERROR genericpipeline: <type 'exceptions.ImportError'> 2016-02-04 13:33:56 ERROR genericpipeline: No module named PipelineStep_createMapfile 2016-02-04 13:33:56 ERROR genericpipeline: *******************************************
(The exact name of the missing module varies.) You are probably missing one of the entries in the recipe_directories
setting in your pipeline.cfg
, or one of those entries doesn't work. Make sure both entries point to the correct directories, and that the missing module can be found in the plugins
subdirectory of one of those two entries.
Missing “h5imp_cal_losoto.h5”: Your pipeline run fails with an “executable_args failed
” error and in the logfile you can find something like:
"/usr/local/lofar/losoto/current/lib/python2.7/site-packages/losoto-1.0.0-py2.7.egg/losoto/h5parm.py", line 40, in __init__ raise Exception('Missing file '+h5parmFile+'.') Exception: Missing file ./h5imp_cal_losoto.h5.
That was caused by a bug in an old version of the genericpipeline. Update the software, make sure you use the new version of the software when starting the pipeline, and check the pathes in pipeline.cfg
that they use the new version!
invalid value for ExecField executable: Your pipeline run fails like that:
2016-04-25 15:53:23 ERROR genericpipeline: ******************************************* 2016-04-25 15:53:23 ERROR genericpipeline: Failed pipeline run: Initial-Subtract 2016-04-25 15:53:23 ERROR genericpipeline: Detailed exception information: 2016-04-25 15:53:23 ERROR genericpipeline: <type 'exceptions.TypeError'> 2016-04-25 15:53:23 ERROR genericpipeline: /homea/htb00/htb001/prefactor/bin/InitSubtract_sort_and_compute.py is an invalid value for ExecField executable 2016-04-25 15:53:23 ERROR genericpipeline: *******************************************
The given path points to a file that either doesn't exist or that does not have the execute flag set on the file system (“chmod +x
”). Usually this affects executables that are defined in the pipeline parset. So make sure that the variables in the pipeline parset point to the right files, and check if the execute flag is set.
For feature requests or bug reports, please open an issue on the GitHub at: https://github.com/lofar-astron/prefactor/issues