public:user_software:prefactor

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
public:user_software:prefactor [2016-02-08 09:17] – [Documentation] Andreas Hornefferpublic:user_software:prefactor [2019-02-25 09:16] (current) – [To do list] David Rafferty
Line 5: Line 5:
 The official repository of prefactor is now on GitHub: https://github.com/lofar-astron/prefactor The official repository of prefactor is now on GitHub: https://github.com/lofar-astron/prefactor
  
-I consists of parsets for the [[http://www.astron.nl/citt/genericpipeline/| genericpipeline ]] that do the first calibration of LOFAR data. Originally in order to prepare said data for the Factor facet calibration (https://github.com/lofar-astron/factor), but also useful if you don't plan to run Factor.+It consists of parsets for the [[http://www.astron.nl/citt/genericpipeline/| genericpipeline ]] that do the first calibration of LOFAR data. Originally in order to prepare said data for the Factor facet calibration (https://github.com/lofar-astron/factor), but also useful if you don't plan to run Factor.
  
 It includes: It includes:
Line 14: Line 14:
  
 There are several pipeline parsets in this repository: There are several pipeline parsets in this repository:
-  * **Pre-Facet-Cal.parset** : The "standard" pre-facet calibration pipeline, works on pre-NDPPP'ed data +  * **Pre-Facet-Calibrator.parset** : The calibrator pipeline. 
-  * **Pre-Facet-Cal-RawData-Single.parset** : A pre-facet pipeline to work on raw (non NDPPP'ed) data +  * **Pre-Facet-Target.parset** : The target pipeline. 
-  * **Pre-Facet-Cal-RawData-PreAvg.parset** : A pre-facet pipeline to work on raw (non NDPPP'ed) data that does the subband concatenating in the first NDPPP step. (To reduce the number of files on systems where this is a problem, e.g. JURECA) +  * **Initial-Subtract.parset** : A pipeline that generates full FoV images and subtracts the sky-models from the visibilities. (Needed for Factor.)
-  * **Initial-Subtract.parset** : A pipeline that generates full FoV images and subtracts the sky-models from the visibilities. (Needed for facet-calibration, this could also be done as the first step of Factor.) +
 ==== Software requirements ==== ==== Software requirements ====
  
-  * the full "offline" LOFAR software installation version >= 2.15 \\ (With small modifications the Pre-Facet-Cal pipelines can be run with older versions, but that is not supported by the authors anymore.)+  * the full "offline" LOFAR software installation
   * LoSoTo   * LoSoTo
-  * Python-PP (see http://www.parallelpython.com/ or https://pypi.python.org/pypi/pp )+  * LSMTool (see https://github.com/darafferty/LSMTool)
   * Python matplotlib   * Python matplotlib
-  * WSClean (for Initial-Subtract, version >=1.9) +  * WSClean (for Initial-Subtract) 
 +  * Aplpy (see https://aplpy.github.io) 
 +  * See the prefactor GitHub page for more information
 ==== Documentation ==== ==== Documentation ====
  
-There is a preliminary version of a cookbook chapter at: https://github.com/lofar-astron/prefactor/blob/pdf-doc/docs/cookbook_prefacet.pdf +Installation and setup is explained at the GitHub page: https://github.com/lofar-astron/prefactor \\ 
 **Please read that first!** **Please read that first!**
  
 === Usage Notes === === Usage Notes ===
   * Don't edit the original parset files directly. Make a copy with a descriptive name (e.g. ''Pre-Facet-Cal-calibrator-3c295.parset'') and edit that copy.    * Don't edit the original parset files directly. Make a copy with a descriptive name (e.g. ''Pre-Facet-Cal-calibrator-3c295.parset'') and edit that copy. 
-  * Get someone to write a better documentation for the genericpipeline.+    * There is also no need to change the ''runtime_directory'' and ''working_directory'' entries in the ''pipeline.cfg'' for different pipeline runs. The pipeline framework will generate sub-directories with the job-name in there. 
 +  * calibrator and target data have to match: 
 +    * observed with the same selection of stations (If calibrator and target observations have the same number of stations but different stations, then the pipeline will not fail but produce wrong results.) 
 +    * observed close enough in time that calibration values can be transferred 
 +  * For each observation you should process all the calibrator data at once together. (Clock/TEC separation and flagging of bad amplitudes work better with the full bandwidth.) 
 +    * Calibrator and target data are typically processed separately. 
 +    * The target data can be processed one time- or frequency- block at a time, e.g. when you have limited disk space. (The processed data is a lot smaller than the input data.) 
 +  * Don't forget to update the recipe_directories entry in the pipeline.cfg to include the pre-factor plugins, for example: <code> recipe_directories = [%(pythonpath)s/lofarpipe/recipes,/home/rvweeren/software/prefactor]  </code> 
 + 
 + 
 +==== FAQ ==== 
 + 
 +**BLAS Core affinity :** Your pipeline runs slow. All NDPPP-/BBS-/whatever- processes use only little CPU time and only one core of the node is busy.\\ 
 +On clusters like CEP-3 the OpenBLAS library is built with threading affinity. This means that by default the different processes all try to use the same core(s). The ''use LofIm'' and ''use Lofar'' scripts set an environment variable that disables this threading affinity, but if the ''pipeline.cfg'' file does not have the ''[remote]'' section included, then this environment variable is not forwarded to the processes that are started by the pipeline.\\ So please set the ''[remote]'' section in your ''pipeline.cfg''.
  
-=== FAQ === 
  
 **KeyError 'mapfile' :** Your pipeline run fails like that: **KeyError 'mapfile' :** Your pipeline run fails like that:
Line 48: Line 59:
 </code> </code>
 That happens when one step didn't generate a mapfile. Usually that means that the pipeline was looking for its input data, but couldn't find any files that match. Please check your ''*_input_path'' and ''*_input_pattern'' in the parset file! (Note: ''ls -d *_input_path/*_input_pattern'' should find your data.) That happens when one step didn't generate a mapfile. Usually that means that the pipeline was looking for its input data, but couldn't find any files that match. Please check your ''*_input_path'' and ''*_input_pattern'' in the parset file! (Note: ''ls -d *_input_path/*_input_pattern'' should find your data.)
 +
 +** PipelineStep_* missing :** Your pipeline run fails like that:
 +<code>
 +2016-02-04 13:33:56 ERROR   genericpipeline: *******************************************
 +2016-02-04 13:33:56 ERROR   genericpipeline: Failed pipeline run: Pre-Facet-Cal
 +2016-02-04 13:33:56 ERROR   genericpipeline: Detailed exception information:
 +2016-02-04 13:33:56 ERROR   genericpipeline: <type 'exceptions.ImportError'>
 +2016-02-04 13:33:56 ERROR   genericpipeline: No module named PipelineStep_createMapfile
 +2016-02-04 13:33:56 ERROR   genericpipeline: *******************************************
 +</code>
 +(The exact name of the missing module varies.) You are probably missing one of the entries in the ''recipe_directories'' setting in your ''pipeline.cfg'', or one of those entries doesn't work. Make sure both entries point to the correct directories, and that the missing module can be found in the ''plugins'' subdirectory of one of those two entries.
 +
 +** Missing "h5imp_cal_losoto.h5":** Your pipeline run fails with an "''executable_args failed''" error and in the logfile you can find something like:
 +<code>
 +"/usr/local/lofar/losoto/current/lib/python2.7/site-packages/losoto-1.0.0-py2.7.egg/losoto/h5parm.py",
 +line 40, in __init__
 +    raise Exception('Missing file '+h5parmFile+'.')
 +Exception: Missing file ./h5imp_cal_losoto.h5.
 +</code>
 +That was caused by a bug in an old version of the genericpipeline. Update the software, make sure you use the new version of the software when starting the pipeline, and check the pathes in ''pipeline.cfg'' that they use the new version!
 +
 +**invalid value for ExecField executable:** Your pipeline run fails like that:
 +<code>
 +2016-04-25 15:53:23 ERROR   genericpipeline: *******************************************
 +2016-04-25 15:53:23 ERROR   genericpipeline: Failed pipeline run: Initial-Subtract
 +2016-04-25 15:53:23 ERROR   genericpipeline: Detailed exception information:
 +2016-04-25 15:53:23 ERROR   genericpipeline: <type 'exceptions.TypeError'>
 +2016-04-25 15:53:23 ERROR   genericpipeline: /homea/htb00/htb001/prefactor/bin/InitSubtract_sort_and_compute.py is an invalid value for ExecField executable
 +2016-04-25 15:53:23 ERROR   genericpipeline: *******************************************
 +</code>
 +The given path points to a file that either doesn't exist or that does not have the execute flag set on the file system ("''chmod +x''"). Usually this affects executables that are defined in the pipeline parset. So make sure that the variables in the pipeline parset point to the right files, and check if the execute flag is set. 
 +==== To do list ====
 +
 +For feature requests or bug reports, please open an issue on the GitHub at:
 +https://github.com/lofar-astron/prefactor/issues
  
  • Last modified: 2016-02-08 09:17
  • by Andreas Horneffer