Differences

This shows you the differences between two versions of the page.

--- public:user_software:prefactor [2016-02-25 17:32] – Tim Shimwell
+++ public:user_software:prefactor [2019-02-25 09:16] (current) – [To do list] David Rafferty
@@ Line 14: / Line 14: @@
 There are several pipeline parsets in this repository:
-  * **Pre-Facet-Cal.parset** : The "standard" pre-facet calibration pipeline, works on pre-NDPPP'ed data
+  * **Pre-Facet-Calibrator.parset** : The calibrator pipeline.
-  * **Pre-Facet-Cal-RawData-Single.parset** : A pre-facet pipeline to work on raw (non NDPPP'ed) data
+  * **Pre-Facet-Target.parset** : The target pipeline.
-  * **Pre-Facet-Cal-RawData-PreAvg.parset** : A pre-facet pipeline to work on raw (non NDPPP'ed) data that does the subband concatenating in the first NDPPP step. (To reduce the number of files on systems where this is a problem, e.g. JURECA)
+  * **Initial-Subtract.parset** : A pipeline that generates full FoV images and subtracts the sky-models from the visibilities. (Needed for Factor.)
-  * **Initial-Subtract.parset** : A pipeline that generates full FoV images and subtracts the sky-models from the visibilities. (Needed for facet-calibration, this could also be done as the first step of Factor.)
 ==== Software requirements ====
-  * the full "offline" LOFAR software installation version >= 2.15 \\ (With small modifications the Pre-Facet-Cal pipelines can be run with older versions, but that is not supported by the authors anymore.)
+  * the full "offline" LOFAR software installation
   * LoSoTo
   * LSMTool (see https://github.com/darafferty/LSMTool)
-  * Python-PP (see http://www.parallelpython.com/ or https://pypi.python.org/pypi/pp )
   * Python matplotlib
-  * WSClean (for Initial-Subtract, version >=1.9)
+  * WSClean (for Initial-Subtract)
+  * Aplpy (see https://aplpy.github.io)
+  * See the prefactor GitHub page for more information
 ==== Documentation ====
-There is a preliminary version of a cookbook chapter at: https://github.com/lofar-astron/prefactor/blob/pdf-doc/docs/cookbook_prefacet.pdf
+Installation and setup is explained at the GitHub page: https://github.com/lofar-astron/prefactor \\
 **Please read that first!**
@@ Line 40: / Line 38: @@
     * observed close enough in time that calibration values can be transferred
   * For each observation you should process all the calibrator data at once together. (Clock/TEC separation and flagging of bad amplitudes work better with the full bandwidth.)
-    * The target data can be processed one time- or frequency- block at a time.
+    * Calibrator and target data are typically processed separately.
-  * Yes, updating the LOFAR software can be annoying. But it is worth ist! Trust me. 8-)
+    * The target data can be processed one time- or frequency- block at a time, e.g. when you have limited disk space. (The processed data is a lot smaller than the input data.)
-  * Get someone to write a better documentation for the genericpipeline.
+  * Don't forget to update the recipe_directories entry in the pipeline.cfg to include the pre-factor plugins, for example: <code> recipe_directories = [%(pythonpath)s/lofarpipe/recipes,/home/rvweeren/software/prefactor]  </code>
 ==== FAQ ====
@@ Line 81: / Line 80: @@
 That was caused by a bug in an old version of the genericpipeline. Update the software, make sure you use the new version of the software when starting the pipeline, and check the pathes in ''pipeline.cfg'' that they use the new version!
+**invalid value for ExecField executable:** Your pipeline run fails like that:
+<code>
+-04-25 15:53:23 ERROR   genericpipeline: *******************************************
+-04-25 15:53:23 ERROR   genericpipeline: Failed pipeline run: Initial-Subtract
+-04-25 15:53:23 ERROR   genericpipeline: Detailed exception information:
+-04-25 15:53:23 ERROR   genericpipeline: <type 'exceptions.TypeError'>
+-04-25 15:53:23 ERROR   genericpipeline: /homea/htb00/htb001/prefactor/bin/InitSubtract_sort_and_compute.py is an invalid value for ExecField executable
+-04-25 15:53:23 ERROR   genericpipeline: *******************************************
+</code>
+The given path points to a file that either doesn't exist or that does not have the execute flag set on the file system ("''chmod +x''"). Usually this affects executables that are defined in the pipeline parset. So make sure that the variables in the pipeline parset point to the right files, and check if the execute flag is set.
 ==== To do list ====
-== Pre factor ==
+For feature requests or bug reports, please open an issue on the GitHub at:
+https://github.com/lofar-astron/prefactor/issues
-  * RMwriter
-  * Write script for automatic flagging on the calibrator and the in the pipeline rerunning of the calibrator after this flagging
-  * TGSS global sky model (not really something to develop)
-  * Collect phase calibration solutions and fit to these and apply the fits (Reinout has script).
-  * Check fix for observations without continuous frequency coverage
-  * Move to NDPPP (waiting for NDPPP developments).
-== Subtract pipeline ==
-  * Create a deep image by stacking 10subband images and use this image or catalog of this image to help create facets and avoid sources at the boundaries.
-== FACTOR ==
-  * Time correlation settings given by user.
-  * Aplpy plots (Tammo has script on his github but change noise to 1mJy (current 5mJy))
-  * All WSclean (any aliasing issues?)
-  * pybdsm boxes from restoring beam size.
-  * outer uvrange option (maybe good for help create models for selfcalbration when working at full resolution)
-  * Check that amplitudes are properly normalised
-  * Second major iteration
-  * High dynamic range option