public:user_software:documentation:ndppp

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revisionBoth sides next revision
public:user_software:documentation:ndppp [2019-10-30 10:30] – [Input] Tammo Jan Dijkemapublic:user_software:documentation:ndppp [2020-05-28 18:52] André Offringa
Line 33: Line 33:
     * **[[#H5ParmPredict]]** to subtract multiple directions of visibilities corrupted by an instrument model (in H5Parm) generated by DDECal.     * **[[#H5ParmPredict]]** to subtract multiple directions of visibilities corrupted by an instrument model (in H5Parm) generated by DDECal.
     * **[[#ApplyBeam]]** to apply the LOFAR beam model, or the inverse of it.     * **[[#ApplyBeam]]** to apply the LOFAR beam model, or the inverse of it.
 +    * **[[#SetBeam]]** to set the beam keywords after prediction.
     * **[[#ScaleData]]** to scale the data with a polynomial in frequency (based on SEFD of LOFAR stations).     * **[[#ScaleData]]** to scale the data with a polynomial in frequency (based on SEFD of LOFAR stations).
     * **[[#Upsample]]** to upsample visibilities in time     * **[[#Upsample]]** to upsample visibilities in time
Line 300: Line 301:
  
 ==== Input ==== ==== Input ====
-| msin \\ msin.name | string | | Name of the input MeasurementSets. If a single name is given, it can be a glob-pattern (like L23456_SAP000_SB*) meaning that all MSs matching the pattern will be used. A glob-pattern can contain *, ?, [], and {} pattern characters (as used in bash). \\ If multiple MSs are to be used, their data are concatenated in frequency, thus multiple subbands are combined to a single band. In principle all MSs should exist, but if 'missingdata=true' and 'orderms=false' flagged zero data will be inserted for missing MS(s) and their frequency info will be deduced from the other MSs. | + 
-| msin.sort | bool | false | Does the MS need to be sorted in TIME order? | +|msin \\ msin.name|string| |Name of the input MeasurementSets. If a single name is given, it can be a glob-pattern (like L23456_SAP000_SB*) meaning that all MSs matching the pattern will be used. A glob-pattern can contain *, ?, [], and {} pattern characters (as used in bash). \\ If multiple MSs are to be used, their data are concatenated in frequency, thus multiple subbands are combined to a single band. In principle all MSs should exist, but if 'missingdata=true' and 'orderms=false' flagged zero data will be inserted for missing MS(s) and their frequency info will be deduced from the other MSs.| 
-| msin.orderms | bool | true | Do the MSs need to be ordered on frequency? If true, all MSs must exist, otherwise they cannot be ordered. If false, the MSs must be given in order of frequency. |  +|msin.sort|bool|false|Does the MS need to be sorted in TIME order?| 
-| msin.missingdata | bool | false | true = it is allowed that a data column in an MS does not exist. In that case its data will be 0 and flagged. It can be useful if the CORRECTED_DATA of subbands are combined, but a BBS run for one of them failed. \\ If 'orderms=false', it also makes it possible that a MS is specified but does not exist.  In such a case flagged data will be used instead. The missing frequency info will be deduced from the other MSs where all MSs have to have the same number of channels and must be defined in order of frequency. | +|msin.orderms|bool|true|Do the MSs need to be ordered on frequency? If true, all MSs must exist, otherwise they cannot be ordered. If false, the MSs must be given in order of frequency.| 
-| msin.baseline | string | | Baselines to be selected (default is all baselines). See [[#Description of baseline selection parameters]]. Only the CASA baseline selection syntax as described in {{msselection.pdf | this note}} can be used. | +|msin.missingdata|bool|false|true = it is allowed that a data column in an MS does not exist. In that case its data will be 0 and flagged. It can be useful if the CORRECTED_DATA of subbands are combined, but a BBS run for one of them failed. \\ If 'orderms=false', it also makes it possible that a MS is specified but does not exist. In such a case flagged data will be used instead. The missing frequency info will be deduced from the other MSs where all MSs have to have the same number of channels and must be defined in order of frequency.| 
-| msin.band | integer | -1 | Band (spectral window) to select (<0 is no selection). This is mainly useful for WSRT data. | +|msin.baseline|string| |Baselines to be selected (default is all baselines). See [[#description_of_baseline_selection_parameters|Description of baseline selection parameters]]. Only the CASA baseline selection syntax as described in {{:public:user_software:documentation:msselection.pdf| this note}}  can be used.| 
-| msin.startchan | integer | 0 | First channel to use from the input MS (channel numbers start counting at 0). Note that skipped channels will not be written into the output MS. It can be an expression with `nchan` (nr of input channels) as parameter. E.g. \\ ''  nchan/32'' \\ will be fine for LOFAR observations with 64 and 256 channels. | +|msin.band|integer|-1|Band (spectral window) to select (<0 is no selection). This is mainly useful for WSRT data.| 
-| msin.nchan | integer | 0 | Number of channels to use from the input MS (0 means till the end). It can be an expression with `nchan` (nr of input channels) as parameter. E.g. \\ ''15*nchan/16''+|msin.startchan|integer|0|First channel to use from the input MS (channel numbers start counting at 0). Note that skipped channels will not be written into the output MS. It can be an expression with `nchan` (nr of input channels) as parameter. E.g. \\  ''nchan/32'' \\ will be fine for LOFAR observations with 64 and 256 channels.| 
-| msin.starttime | string | first time in MS | Center of first time slot to use; if < first time in MS, dummy time slots are inserted. A date/time must be specified in the casacore MVTime format, e.g. 19Feb2010/14:01:23.817 | +|msin.nchan|integer|0|Number of channels to use from the input MS (0 means till the end). It can be an expression with `nchan` (nr of input channels) as parameter. E.g. \\  ''15*nchan/16''
-| msin.starttimeslots | int | 0 | Starting time slot. This can be negative to insert flagged time slots before the beginning of the MS. | +|msin.starttime|string|first time in MS|Center of first time slot to use; if < first time in MS, dummy time slots are inserted. A date/time must be specified in the casacore MVTime format, e.g. 19Feb2010/14:01:23.817| 
-| msin.endtime | string | last time in MS | Center of last time slot to use; if > last time in MS, dummy time slots are inserted. | +|msin.starttimeslot|int|0|Starting time slot. This can be negative to insert flagged time slots before the beginning of the MS.| 
-| msin.ntimes | integer | 0 | Number of time slots to use (0 means till the end). | +|msin.endtime|string|last time in MS|Center of last time slot to use; if > last time in MS, dummy time slots are inserted.| 
-| msin.useflag | bool | true | Use the current flags in the MS? If false, all flags in the MS are ignore and the data (except NaN and infinite values) are assumed to be good and will be used in later steps. | +|msin.ntimes|integer|0|Number of time slots to use (0 means till the end).| 
-| msin.datacolumn | string | DATA | Data column to use, i.e. the name of the column in which the visibilities are written. | +|msin.useflag|bool|true|Use the current flags in the MS? If false, all flags in the MS are ignore and the data (except NaN and infinite values) are assumed to be good and will be used in later steps.| 
-| msin.weightcolumn | string | WEIGHT_SPECTRUM or WEIGHT | Weight column to use. Defaults to WEIGHT_SPECTRUM if this exists, otherwise the WEIGHT column is used. | +|msin.datacolumn|string|DATA|Data column to use, i.e. the name of the column in which the visibilities are written.| 
-| msin.modelcolumn | string | MODEL_DATA | Model data column. Currently only used in gaincal and ddecal.| +|msin.weightcolumn|string|WEIGHT_SPECTRUM or WEIGHT|Weight column to use. Defaults to WEIGHT_SPECTRUM if this exists, otherwise the WEIGHT column is used.| 
-| msin.autoweight | bool | false | Calculate weights using the auto-correlation data? It is meant for setting the proper weights for a raw LOFAR MeasurementSet. | +|msin.modelcolumn|string|MODEL_DATA|Model data column. Currently only used in gaincal and ddecal.| 
-| msin.forceautoweight | bool | false | In principle the calculation of the weights should only be done for the raw LOFAR data. It appeared that sometimes the ''autoweight'' switch was accidently set in a DPPP run on already dppp-ed data. To make it harder to make such mistakes, the ''forceautoweight'' flag has to be set as well for MSs containing dppp-ed data. |+|msin.autoweight|bool|false|Calculate weights using the auto-correlation data? It is meant for setting the proper weights for a raw LOFAR MeasurementSet.| 
 +|msin.forceautoweight|bool|false|In principle the calculation of the weights should only be done for the raw LOFAR data. It appeared that sometimes the ''autoweight'' switch was accidently set in a DPPP run on already dppp-ed data. To make it harder to make such mistakes, the ''forceautoweight'' flag has to be set as well for MSs containing dppp-ed data.| 
 + 
 +\\ 
  
 ==== Output ==== ==== Output ====
Line 526: Line 531:
 | <step>.sources | | | Same as in **Predict** step | | <step>.sources | | | Same as in **Predict** step |
 | <step>.usebeammodel | | | Same as in **Predict** step | | <step>.usebeammodel | | | Same as in **Predict** step |
-| <step>.operation | | | Same as in **Predict** step | 
 | <step>.applycal.* | | | ApplyCal sub-step, same as in **Predict** step | | <step>.applycal.* | | | ApplyCal sub-step, same as in **Predict** step |
 | <step>.onebeamperpatch | | | Same as in **ApplyBeam** step | | <step>.onebeamperpatch | | | Same as in **ApplyBeam** step |
Line 533: Line 537:
  
 ==== DDECal ==== ==== DDECal ====
-| <step>.type | string | | Case-insensitive step type; must be 'ddecal'. | + 
-| <step>.sourcedb | string | | Sourcedb (created with `makesourcedb`) with the sky model to calibrate on. | +|<step>.type|string| |Case-insensitive step type; must be 'ddecal'.| 
-| <step>.directions | list | [] | List of directions to calibrate on. Every element of this list should b a list of facets. Default: every facet is a direction. | +|<step>.sourcedb|string| |Sourcedb (created with `makesourcedb`) with the sky model to calibrate on.| 
-| <step>.usemodelcolumn | bool | false | Use model data from the measurement set. This implies solving for one direction, namely the pointing of the measurement set. If you specify usemodelcolumn to be true, directions and sourcedb are not required | +|<step>.directions|list|[]|List of directions to calibrate on. Every element of this list should b a list of facets. Default: every facet is a direction.| 
-| <step>.maxiter | int | 50 | Maximum number of iterations. | +|<step>.usemodelcolumn|bool|false|Use model data from the measurement set. This implies solving for one direction, namely the pointing of the measurement set. If you specify usemodelcolumn to be true, directions and sourcedb are not required| 
-| <step>.detectstalling | bool | true | Stop iterating when no improvement is measured anymore (after a minimum of 30 iterations). | +|<step>.maxiter|int|50|Maximum number of iterations.| 
-| <step>.stepsize | double | 0.2 | stepsize between iterations. | +|<step>.detectstalling|bool|true|Stop iterating when no improvement is measured anymore (after a minimum of 30 iterations).| 
-| <step>.h5parm | string | | Filename of output H5Parm (to be read by e.g. losoto). If empty, defaults to ''instrument.h5'' within the measurement set. | +|<step>.stepsize|double|0.2|stepsize between iterations.| 
-| <step>.solint | int | 1 | Solution interval in timesteps. | +|<step>.h5parm|string| |Filename of output H5Parm (to be read by e.g. losoto). If empty, defaults to ''instrument.h5'' within the measurement set.| 
-| <step>.usebeammodel | bool | false | use the beam model. All beam-related options of the Predict step are also valid. | +|<step>.solint|int|1|Solution interval in timesteps.| 
-| <step>.mode | string | diagonal | Type of constraint to apply. Options are scalarcomplexgain, scalarphase, scalaramplitude, tec, tecandphase. Modes in development are fulljones, diagonal, phaseonly, amplitudeonly, rotation, rotation+diagonal. | +|<step>.usebeammodel|bool|false|use the beam model. All beam-related options of the Predict step are also valid.| 
-| <step>.tolerance | double | 1e-5 | Controls the accuracy to be reached: when the normalized solutions move less than this value, the solutions are considered to be converged and the algorithm finishes. Lower values will cause more iterations to be performed.| +|<step>.mode|string|diagonal|Type of constraint to apply. Options are scalarcomplexgain, scalarphase, scalaramplitude, tec, tecandphase. Modes in development are fulljones, diagonal, phaseonly, amplitudeonly, rotation, rotation+diagonal.| 
-| <step>.minvisratio | double | 0 | Minimum number of visibilities within a solution interval, e.g. 0.6 for at least 60% unflagged vis. Intervals with fewer vis will be flagged.|  +|<step>.tolerance|double|1e-5|Controls the accuracy to be reached: when the normalized solutions move less than this value, the solutions are considered to be converged and the algorithm finishes. Lower values will cause more iterations to be performed.| 
-| <step>.propagatesolutions | bool | false | Initialize solver with the solutions of the previous time slot. | +|<step>.minvisratio|double|0|Minimum number of visibilities within a solution interval, e.g. 0.6 for at least 60% unflagged vis. Intervals with fewer vis will be flagged.| 
-| <step>.propagateconvergedonly | bool | false | Propagate solutions of the previous time slot only if the solve converged. Only effective when propagatesolutions=true. | +|<step>.propagatesolutions|bool|false|Initialize solver with the solutions of the previous time slot.| 
-| <step>.flagunconverged | bool | false | Flag unconverged solutions (i.e., those from solves that did not converge within maxiter iterations). | +|<step>.propagateconvergedonly|bool|false|Propagate solutions of the previous time slot only if the solve converged. Only effective when propagatesolutions=true.| 
-| <step>.flagdivergedonly | bool | false | Flag only the unconverged solutions for which divergence was detected. At the moment, this option is effective only for rotation+diagonal solves, where divergence is detected when the amplitudes of any station are found to be more than a factor of 5 from the mean amplitude over all stations. If divergence for any one station is detected, all stations are flagged for that solution interval. Only effective when flagunconverged=true and mode=rotation+diagonal. | +|<step>.flagunconverged|bool|false|Flag unconverged solutions (i.e., those from solves that did not converge within maxiter iterations).| 
-| <step>.approximatetec | bool | false | Uses an approximation stage in which the phases are constrained with the piece-wise fitter, to solve local minima problems. Only effective when mode=tec or mode=tecandphase. | +|<step>.flagdivergedonly|bool|false|Flag only the unconverged solutions for which divergence was detected. At the moment, this option is effective only for rotation+diagonal solves, where divergence is detected when the amplitudes of any station are found to be more than a factor of 5 from the mean amplitude over all stations. If divergence for any one station is detected, all stations are flagged for that solution interval. Only effective when flagunconverged=true and mode=rotation+diagonal.| 
-| <step>.maxapproxiter | int | maxiter/2 | Maximum number of iterations during approximating stage. | +|<step>.approximatetec|bool|false|Uses an approximation stage in which the phases are constrained with the piece-wise fitter, to solve local minima problems. Only effective when mode=tec or mode=tecandphase.| 
-| <step>.approxchunksize | int | 0 | Size of fitted chunksize during approximation stage in nr of channels. With approxchunksize=1 the constraint is disabled during the approx stage (so channels are solved for independently). Once converged, the solutions are constrained and more iterations are performed until that has converged too. The default is approxchunksize=0, which calculates the chunksize from the bandwidth (resulting in 10 chunks per octave of bandwidth). | +|<step>.maxapproxiter|int|maxiter/2|Maximum number of iterations during approximating stage.| 
-| <step>.approxtolerance | double | tolerance*10 | Tolerance at which the approximating first stage is considered to be converged and the second full-constraining stage is started. The second stage convergences when the tolerance set by the 'tolerance' keyword is reached. Setting approxtolerance to lower values will cause more approximating iterations. Since tolerance is by default 1e-5, approxtolerance is by default 1e-4. | +|<step>.approxchunksize|int|0|Size of fitted chunksize during approximation stage in nr of channels. With approxchunksize=1 the constraint is disabled during the approx stage (so channels are solved for independently). Once converged, the solutions are constrained and more iterations are performed until that has converged too. The default is approxchunksize=0, which calculates the chunksize from the bandwidth (resulting in 10 chunks per octave of bandwidth).| 
-| <step>.nchan | int | 1 | Number of channels in each channel block, for which the solution is assumed to be constant. The default is 1, meaning one solution per channel (or in the case of constraints, fitting the constraint over all channels individually). 0 means one solution for the whole channel range. If the total number of channels is not divisable by nchan, some channelblocks will become slightly larger. | +|<step>.approxtolerance|double|tolerance*10|Tolerance at which the approximating first stage is considered to be converged and the second full-constraining stage is started. The second stage convergences when the tolerance set by the 'tolerance' keyword is reached. Setting approxtolerance to lower values will cause more approximating iterations. Since tolerance is by default 1e-5, approxtolerance is by default 1e-4.| 
-| <step>.coreconstraint | double | 0 | Distance in meters. When unequal to 0, all stations within the given distance from the reference station (0) will be constraint to have the same solution. | +|<step>.nchan|int|1|Number of channels in each channel block, for which the solution is assumed to be constant. The default is 1, meaning one solution per channel (or in the case of constraints, fitting the constraint over all channels individually). 0 means one solution for the whole channel range. If the total number of channels is not divisable by nchan, some channelblocks will become slightly larger.| 
-| <step>.antennaconstraint | list | [] | A list of lists specifying groups of antennas that are to be constrained to have the same solution. Example: "[ [CS002HBA0,CS002HBA1],[CS003HBA0,CS003HBA1] ]" will keep the solutions of CS002HBA0 and 1 the same, and the same for CS003. | +|<step>.coreconstraint|double|0|Distance in meters. When unequal to 0, all stations within the given distance from the reference station (0) will be constraint to have the same solution.| 
-| <step>.smoothnessconstraint | double | 0 | Kernel size in Hz. When unequal to 0, will constrain the solutions to be smooth over frequency by convolving the solutions with a kernel of the given size (bandwidth). The default kernel is a Gaussian kernel, and the kernel size parameter is the 3 sigma point where the kernel is cut off. | +|<step>.antennaconstraint|list|[]|A list of lists specifying groups of antennas that are to be constrained to have the same solution. Example: "[ [CS002HBA0,CS002HBA1],[CS003HBA0,CS003HBA1] ]" will keep the solutions of CS002HBA0 and 1 the same, and the same for CS003.| 
-| <step>.statfilename | string | "" | File to write the step-sizes to. Form of the file is: "<iterationnr> <normalized-stepsize> <unnormalized-stepsize>", and all solution intervals are concatenated. File is not written when this parameter is empty. | +|<step>.smoothnessconstraint|double|0|Kernel size in Hz. When unequal to 0, will constrain the solutions to be smooth over frequency by convolving the solutions with a kernel of the given size (bandwidth). The default kernel is a Gaussian kernel, and the kernel size parameter is the 3 sigma point where the kernel is cut off.| 
-| <step>.uvlambdamin | double | 0 | Ignore baselines / channels with UV < uvlambdamin wavelengths. **Note**: also all other variants of uv flagging described in [[#UVWFlagger]] (uvmmin, uvmrange, uvlambdarange, etc) are supported (New in 3.1).| +|<step>.statfilename|string| |File to write the step-sizes to. Form of the file is: "<iterationnr> <normalized-stepsize> <unnormalized-stepsize>", and all solution intervals are concatenated. File is not written when this parameter is empty.| 
-| <step>.subtract | bool | false | Subtracts the corrected model from the data. **NOTE** This may not work when you apply a uv-cut. | +|<step>.uvlambdamin|double|0|Ignore baselines / channels with UV < uvlambdamin wavelengths. **Note**: also all other variants of uv flagging described in [[#uvwflagger|UVWFlagger]] (uvmmin, uvmrange, uvlambdarange, etc) are supported (New in 3.1).| 
-| <step>.useidg | bool | false | Do image-based prediction using IDG.| +|<step>.subtract|bool|false|Subtracts the corrected model from the data. **NOTE** This may not work when you apply a uv-cut.| 
-| <step>.idg.image string "" Name of fits file for IDG predictionShould have the phase direction of the measurement set. | +|<step>.useidg|bool|false|Do image-based prediction using IDG.| 
-| <step>.idg.regions | string | "" | DS9 regions file describing the facets for IDG prediction. | +|<step>.idg.images|list|[]|Filename of ''.fits'' model images, one per frequency term. The terms are defined as for a polynomial source spectra (not logarithmic), e.g. see [[https://sourceforge.net/p/wsclean/wiki/ComponentList/|this WSClean page]]. The frequency in the metadata of the fits files is used as nu<sub>0</sub> in the polynomial evaluation.| 
-| <step>.savefacets | bool | false | Write out each facet as a fits file (named facet<N>.fits). Only useful when useidg=true. | +|<step>.idg.regions|string|""|DS9 regions file describing the facets for IDG prediction.
-| <step>.onlypredict | bool | false | Instead of solving, output the predicted visibilities instead. This is useful for testing, although when doing faceted prediction with IDG, it might be fast for certain cases. | +|<step>.idg.buffersize|int|Based on memory|Set the amount of timesteps that are to be used for each IDG buffer
-| <step>.applycal.*| | | ApplyCal sub-step, same as in Predict step. One can pass an h5parm with as many directions as set in "directions" and each direction model is corrupted accordingly. |+|<step>.savefacets|bool|false|Write out each facet as a fits file (named facet<N>.fits). Only useful when useidg=true.| 
 +|<step>.onlypredict|bool|false|Instead of solving, output the predicted visibilities instead. This is useful for testing, although when doing faceted prediction with IDG, it might be fast for certain cases.| 
 +|<step>.applycal.*| | |ApplyCal sub-step, same as in Predict step. One can pass an h5parm with as many directions as set in "directions" and each direction model is corrupted accordingly.| 
 + 
 +\\ 
  
 ==== Predict ==== ==== Predict ====
Line 600: Line 609:
 | <step>.invert | bool | **true** | Invert the beam. When applying the beam to transfer calibration solutions, this should be true. In other words: ''invert=true'' means correcting for the beam, ''invert=false'' means corrupting with the beam. When using the beam in a predict (or gaincal) step, this option defaults to ''false'' (so it will corrupt for the beam). | | <step>.invert | bool | **true** | Invert the beam. When applying the beam to transfer calibration solutions, this should be true. In other words: ''invert=true'' means correcting for the beam, ''invert=false'' means corrupting with the beam. When using the beam in a predict (or gaincal) step, this option defaults to ''false'' (so it will corrupt for the beam). |
 | <step>.beammode | string | "default" | Beam mode to apply, can be "array_factor", "element" or "default". Default is to apply both the element beam and the array factor. | | <step>.beammode | string | "default" | Beam mode to apply, can be "array_factor", "element" or "default". Default is to apply both the element beam and the array factor. |
 +
 +==== SetBeam ====
 +SetBeam is an expert option and should only be used in rare cases. It allows direct manipulation of the beam-keywords for a column in a measurement set. Normally, DP3 registers whether the visibilities in a column are corrected for a beam or not, and if so, in what direction the beam was corrected for. This avoids incorrect corrections / scaling by the beam. However, certain actions can change the scaling of the visibilities without that the beam keywords are changed, in particular when predicting (either with DP3 or with another tool). When predicting a single source and not applying the beam, the visibilities are 'corrected' for the beam in the direction of the source. Under those circumstances, SetBeam can be used to modify the beam keywords. In that case, use direction for the source direction and beammode to default.
 +| <step>.type | string | | Case-insensitive step type; must be 'setbeam' |
 +| <step>.direction | string vector | [] | A RA/Dec value specifying in what direction the beam is corrected. |
 +| <step>.beammode | string | "default" | Beam mode to apply, can be "array_factor", "element" or "default". Default means that sources in the given direction have corrected (intrinsic) flux values, i.e. they are corrected for the full beam. |
  
 ==== UVWFlagger ==== ==== UVWFlagger ====
  • Last modified: 2021-02-26 14:18
  • by Tammo Jan Dijkema