public:user_software:prefactor

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
public:user_software:prefactor [2016-04-15 17:37] – [Software requirements] Reinout van Weerenpublic:user_software:prefactor [2019-02-25 09:16] (current) – [To do list] David Rafferty
Line 14: Line 14:
  
 There are several pipeline parsets in this repository: There are several pipeline parsets in this repository:
-  * **Pre-Facet-Cal.parset** : The "standard" pre-facet calibration pipeline, works on pre-NDPPP'ed data +  * **Pre-Facet-Calibrator.parset** : The calibrator pipeline. 
-  * **Pre-Facet-Cal-RawData-Single.parset** : A pre-facet pipeline to work on raw (non NDPPP'ed) data +  * **Pre-Facet-Target.parset** : The target pipeline. 
-  * **Pre-Facet-Cal-RawData-PreAvg.parset** : A pre-facet pipeline to work on raw (non NDPPP'ed) data that does the subband concatenating in the first NDPPP step. (To reduce the number of files on systems where this is a problem, e.g. JURECA) +  * **Initial-Subtract.parset** : A pipeline that generates full FoV images and subtracts the sky-models from the visibilities. (Needed for Factor.)
-  * **Initial-Subtract.parset** : A pipeline that generates full FoV images and subtracts the sky-models from the visibilities. (Needed for facet-calibration, this could also be done as the first step of Factor.) +
 ==== Software requirements ==== ==== Software requirements ====
  
-  * the full "offline" LOFAR software installation version >= 2.15 \\ (With small modifications the Pre-Facet-Cal pipelines can be run with older versions, but that is not supported by the authors anymore.)+  * the full "offline" LOFAR software installation
   * LoSoTo   * LoSoTo
   * LSMTool (see https://github.com/darafferty/LSMTool)   * LSMTool (see https://github.com/darafferty/LSMTool)
-  * Python-PP (see http://www.parallelpython.com/ or https://pypi.python.org/pypi/pp ) 
   * Python matplotlib   * Python matplotlib
-  * WSClean (for Initial-Subtract, version >=1.9)+  * WSClean (for Initial-Subtract)
   * Aplpy (see https://aplpy.github.io)   * Aplpy (see https://aplpy.github.io)
 +  * See the prefactor GitHub page for more information
 ==== Documentation ==== ==== Documentation ====
  
-Installation and setup is explained in the preliminary version of a cookbook chapter at: https://github.com/lofar-astron/prefactor/blob/pdf-doc/docs/cookbook_prefacet.pdf \\ +Installation and setup is explained at the GitHub page: https://github.com/lofar-astron/prefactor \\ 
 **Please read that first!** **Please read that first!**
  
Line 42: Line 40:
     * Calibrator and target data are typically processed separately.     * Calibrator and target data are typically processed separately.
     * The target data can be processed one time- or frequency- block at a time, e.g. when you have limited disk space. (The processed data is a lot smaller than the input data.)     * The target data can be processed one time- or frequency- block at a time, e.g. when you have limited disk space. (The processed data is a lot smaller than the input data.)
-  * Yes, updating the LOFAR software can be annoying. But it is worth ist! Trust me. 8-) 
-  * Get someone to write a better documentation for the genericpipeline. 
   * Don't forget to update the recipe_directories entry in the pipeline.cfg to include the pre-factor plugins, for example: <code> recipe_directories = [%(pythonpath)s/lofarpipe/recipes,/home/rvweeren/software/prefactor]  </code>   * Don't forget to update the recipe_directories entry in the pipeline.cfg to include the pre-factor plugins, for example: <code> recipe_directories = [%(pythonpath)s/lofarpipe/recipes,/home/rvweeren/software/prefactor]  </code>
  
Line 84: Line 80:
 That was caused by a bug in an old version of the genericpipeline. Update the software, make sure you use the new version of the software when starting the pipeline, and check the pathes in ''pipeline.cfg'' that they use the new version! That was caused by a bug in an old version of the genericpipeline. Update the software, make sure you use the new version of the software when starting the pipeline, and check the pathes in ''pipeline.cfg'' that they use the new version!
  
 +**invalid value for ExecField executable:** Your pipeline run fails like that: 
 +<code> 
 +2016-04-25 15:53:23 ERROR   genericpipeline: ******************************************* 
 +2016-04-25 15:53:23 ERROR   genericpipeline: Failed pipeline run: Initial-Subtract 
 +2016-04-25 15:53:23 ERROR   genericpipeline: Detailed exception information: 
 +2016-04-25 15:53:23 ERROR   genericpipeline: <type 'exceptions.TypeError'> 
 +2016-04-25 15:53:23 ERROR   genericpipeline: /homea/htb00/htb001/prefactor/bin/InitSubtract_sort_and_compute.py is an invalid value for ExecField executable 
 +2016-04-25 15:53:23 ERROR   genericpipeline: ******************************************* 
 +</code> 
 +The given path points to a file that either doesn't exist or that does not have the execute flag set on the file system ("''chmod +x''"). Usually this affects executables that are defined in the pipeline parset. So make sure that the variables in the pipeline parset point to the right files, and check if the execute flag is set. 
 ==== To do list ==== ==== To do list ====
  
-For prefactor please use the ToDo list on the GitHub wiki: +For feature requests or bug reports, please open an issue on the GitHub at:
-https://github.com/lofar-astron/prefactor/wiki \\ +
-Or set up an issue in the GitHub issue List:+
 https://github.com/lofar-astron/prefactor/issues https://github.com/lofar-astron/prefactor/issues
  
-== FACTOR == 
- 
- 
-  * Time correlation settings given by user. 
-  * Aplpy plots (Tammo has script on his github but change noise to 1mJy (current 5mJy)) 
-  * All WSclean (any aliasing issues?) 
-  * pybdsm boxes from restoring beam size. 
-  * outer uvrange option (maybe good for help create models for selfcalbration when working at full resolution) 
-  * Check that amplitudes are properly normalised 
-  * Second major iteration 
-  * High dynamic range option 
-  * optimize number of IO-intesive (NDPPP) processes 
  • Last modified: 2016-04-15 17:37
  • by Reinout van Weeren