| Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision |
| public:user_software:documentation:ndppp [2019-11-26 14:21] – [Input] André Offringa | public:user_software:documentation:ndppp [2020-05-28 18:52] – André Offringa |
|---|
| * **[[#H5ParmPredict]]** to subtract multiple directions of visibilities corrupted by an instrument model (in H5Parm) generated by DDECal. | * **[[#H5ParmPredict]]** to subtract multiple directions of visibilities corrupted by an instrument model (in H5Parm) generated by DDECal. |
| * **[[#ApplyBeam]]** to apply the LOFAR beam model, or the inverse of it. | * **[[#ApplyBeam]]** to apply the LOFAR beam model, or the inverse of it. |
| | * **[[#SetBeam]]** to set the beam keywords after prediction. |
| * **[[#ScaleData]]** to scale the data with a polynomial in frequency (based on SEFD of LOFAR stations). | * **[[#ScaleData]]** to scale the data with a polynomial in frequency (based on SEFD of LOFAR stations). |
| * **[[#Upsample]]** to upsample visibilities in time | * **[[#Upsample]]** to upsample visibilities in time |
| | <step>.sources | | | Same as in **Predict** step | | | <step>.sources | | | Same as in **Predict** step | |
| | <step>.usebeammodel | | | Same as in **Predict** step | | | <step>.usebeammodel | | | Same as in **Predict** step | |
| | <step>.operation | | | Same as in **Predict** step | | |
| | <step>.applycal.* | | | ApplyCal sub-step, same as in **Predict** step | | | <step>.applycal.* | | | ApplyCal sub-step, same as in **Predict** step | |
| | <step>.onebeamperpatch | | | Same as in **ApplyBeam** step | | | <step>.onebeamperpatch | | | Same as in **ApplyBeam** step | |
| |
| ==== DDECal ==== | ==== DDECal ==== |
| | <step>.type | string | | Case-insensitive step type; must be 'ddecal'. | | |
| | <step>.sourcedb | string | | Sourcedb (created with `makesourcedb`) with the sky model to calibrate on. | | |<step>.type|string| |Case-insensitive step type; must be 'ddecal'.| |
| | <step>.directions | list | [] | List of directions to calibrate on. Every element of this list should b a list of facets. Default: every facet is a direction. | | |<step>.sourcedb|string| |Sourcedb (created with `makesourcedb`) with the sky model to calibrate on.| |
| | <step>.usemodelcolumn | bool | false | Use model data from the measurement set. This implies solving for one direction, namely the pointing of the measurement set. If you specify usemodelcolumn to be true, directions and sourcedb are not required | | |<step>.directions|list|[]|List of directions to calibrate on. Every element of this list should b a list of facets. Default: every facet is a direction.| |
| | <step>.maxiter | int | 50 | Maximum number of iterations. | | |<step>.usemodelcolumn|bool|false|Use model data from the measurement set. This implies solving for one direction, namely the pointing of the measurement set. If you specify usemodelcolumn to be true, directions and sourcedb are not required| |
| | <step>.detectstalling | bool | true | Stop iterating when no improvement is measured anymore (after a minimum of 30 iterations). | | |<step>.maxiter|int|50|Maximum number of iterations.| |
| | <step>.stepsize | double | 0.2 | stepsize between iterations. | | |<step>.detectstalling|bool|true|Stop iterating when no improvement is measured anymore (after a minimum of 30 iterations).| |
| | <step>.h5parm | string | | Filename of output H5Parm (to be read by e.g. losoto). If empty, defaults to ''instrument.h5'' within the measurement set. | | |<step>.stepsize|double|0.2|stepsize between iterations.| |
| | <step>.solint | int | 1 | Solution interval in timesteps. | | |<step>.h5parm|string| |Filename of output H5Parm (to be read by e.g. losoto). If empty, defaults to ''instrument.h5'' within the measurement set.| |
| | <step>.usebeammodel | bool | false | use the beam model. All beam-related options of the Predict step are also valid. | | |<step>.solint|int|1|Solution interval in timesteps.| |
| | <step>.mode | string | diagonal | Type of constraint to apply. Options are scalarcomplexgain, scalarphase, scalaramplitude, tec, tecandphase. Modes in development are fulljones, diagonal, phaseonly, amplitudeonly, rotation, rotation+diagonal. | | |<step>.usebeammodel|bool|false|use the beam model. All beam-related options of the Predict step are also valid.| |
| | <step>.tolerance | double | 1e-5 | Controls the accuracy to be reached: when the normalized solutions move less than this value, the solutions are considered to be converged and the algorithm finishes. Lower values will cause more iterations to be performed.| | |<step>.mode|string|diagonal|Type of constraint to apply. Options are scalarcomplexgain, scalarphase, scalaramplitude, tec, tecandphase. Modes in development are fulljones, diagonal, phaseonly, amplitudeonly, rotation, rotation+diagonal.| |
| | <step>.minvisratio | double | 0 | Minimum number of visibilities within a solution interval, e.g. 0.6 for at least 60% unflagged vis. Intervals with fewer vis will be flagged.| | |<step>.tolerance|double|1e-5|Controls the accuracy to be reached: when the normalized solutions move less than this value, the solutions are considered to be converged and the algorithm finishes. Lower values will cause more iterations to be performed.| |
| | <step>.propagatesolutions | bool | false | Initialize solver with the solutions of the previous time slot. | | |<step>.minvisratio|double|0|Minimum number of visibilities within a solution interval, e.g. 0.6 for at least 60% unflagged vis. Intervals with fewer vis will be flagged.| |
| | <step>.propagateconvergedonly | bool | false | Propagate solutions of the previous time slot only if the solve converged. Only effective when propagatesolutions=true. | | |<step>.propagatesolutions|bool|false|Initialize solver with the solutions of the previous time slot.| |
| | <step>.flagunconverged | bool | false | Flag unconverged solutions (i.e., those from solves that did not converge within maxiter iterations). | | |<step>.propagateconvergedonly|bool|false|Propagate solutions of the previous time slot only if the solve converged. Only effective when propagatesolutions=true.| |
| | <step>.flagdivergedonly | bool | false | Flag only the unconverged solutions for which divergence was detected. At the moment, this option is effective only for rotation+diagonal solves, where divergence is detected when the amplitudes of any station are found to be more than a factor of 5 from the mean amplitude over all stations. If divergence for any one station is detected, all stations are flagged for that solution interval. Only effective when flagunconverged=true and mode=rotation+diagonal. | | |<step>.flagunconverged|bool|false|Flag unconverged solutions (i.e., those from solves that did not converge within maxiter iterations).| |
| | <step>.approximatetec | bool | false | Uses an approximation stage in which the phases are constrained with the piece-wise fitter, to solve local minima problems. Only effective when mode=tec or mode=tecandphase. | | |<step>.flagdivergedonly|bool|false|Flag only the unconverged solutions for which divergence was detected. At the moment, this option is effective only for rotation+diagonal solves, where divergence is detected when the amplitudes of any station are found to be more than a factor of 5 from the mean amplitude over all stations. If divergence for any one station is detected, all stations are flagged for that solution interval. Only effective when flagunconverged=true and mode=rotation+diagonal.| |
| | <step>.maxapproxiter | int | maxiter/2 | Maximum number of iterations during approximating stage. | | |<step>.approximatetec|bool|false|Uses an approximation stage in which the phases are constrained with the piece-wise fitter, to solve local minima problems. Only effective when mode=tec or mode=tecandphase.| |
| | <step>.approxchunksize | int | 0 | Size of fitted chunksize during approximation stage in nr of channels. With approxchunksize=1 the constraint is disabled during the approx stage (so channels are solved for independently). Once converged, the solutions are constrained and more iterations are performed until that has converged too. The default is approxchunksize=0, which calculates the chunksize from the bandwidth (resulting in 10 chunks per octave of bandwidth). | | |<step>.maxapproxiter|int|maxiter/2|Maximum number of iterations during approximating stage.| |
| | <step>.approxtolerance | double | tolerance*10 | Tolerance at which the approximating first stage is considered to be converged and the second full-constraining stage is started. The second stage convergences when the tolerance set by the 'tolerance' keyword is reached. Setting approxtolerance to lower values will cause more approximating iterations. Since tolerance is by default 1e-5, approxtolerance is by default 1e-4. | | |<step>.approxchunksize|int|0|Size of fitted chunksize during approximation stage in nr of channels. With approxchunksize=1 the constraint is disabled during the approx stage (so channels are solved for independently). Once converged, the solutions are constrained and more iterations are performed until that has converged too. The default is approxchunksize=0, which calculates the chunksize from the bandwidth (resulting in 10 chunks per octave of bandwidth).| |
| | <step>.nchan | int | 1 | Number of channels in each channel block, for which the solution is assumed to be constant. The default is 1, meaning one solution per channel (or in the case of constraints, fitting the constraint over all channels individually). 0 means one solution for the whole channel range. If the total number of channels is not divisable by nchan, some channelblocks will become slightly larger. | | |<step>.approxtolerance|double|tolerance*10|Tolerance at which the approximating first stage is considered to be converged and the second full-constraining stage is started. The second stage convergences when the tolerance set by the 'tolerance' keyword is reached. Setting approxtolerance to lower values will cause more approximating iterations. Since tolerance is by default 1e-5, approxtolerance is by default 1e-4.| |
| | <step>.coreconstraint | double | 0 | Distance in meters. When unequal to 0, all stations within the given distance from the reference station (0) will be constraint to have the same solution. | | |<step>.nchan|int|1|Number of channels in each channel block, for which the solution is assumed to be constant. The default is 1, meaning one solution per channel (or in the case of constraints, fitting the constraint over all channels individually). 0 means one solution for the whole channel range. If the total number of channels is not divisable by nchan, some channelblocks will become slightly larger.| |
| | <step>.antennaconstraint | list | [] | A list of lists specifying groups of antennas that are to be constrained to have the same solution. Example: "[ [CS002HBA0,CS002HBA1],[CS003HBA0,CS003HBA1] ]" will keep the solutions of CS002HBA0 and 1 the same, and the same for CS003. | | |<step>.coreconstraint|double|0|Distance in meters. When unequal to 0, all stations within the given distance from the reference station (0) will be constraint to have the same solution.| |
| | <step>.smoothnessconstraint | double | 0 | Kernel size in Hz. When unequal to 0, will constrain the solutions to be smooth over frequency by convolving the solutions with a kernel of the given size (bandwidth). The default kernel is a Gaussian kernel, and the kernel size parameter is the 3 sigma point where the kernel is cut off. | | |<step>.antennaconstraint|list|[]|A list of lists specifying groups of antennas that are to be constrained to have the same solution. Example: "[ [CS002HBA0,CS002HBA1],[CS003HBA0,CS003HBA1] ]" will keep the solutions of CS002HBA0 and 1 the same, and the same for CS003.| |
| | <step>.statfilename | string | "" | File to write the step-sizes to. Form of the file is: "<iterationnr> <normalized-stepsize> <unnormalized-stepsize>", and all solution intervals are concatenated. File is not written when this parameter is empty. | | |<step>.smoothnessconstraint|double|0|Kernel size in Hz. When unequal to 0, will constrain the solutions to be smooth over frequency by convolving the solutions with a kernel of the given size (bandwidth). The default kernel is a Gaussian kernel, and the kernel size parameter is the 3 sigma point where the kernel is cut off.| |
| | <step>.uvlambdamin | double | 0 | Ignore baselines / channels with UV < uvlambdamin wavelengths. **Note**: also all other variants of uv flagging described in [[#UVWFlagger]] (uvmmin, uvmrange, uvlambdarange, etc) are supported (New in 3.1).| | |<step>.statfilename|string| |File to write the step-sizes to. Form of the file is: "<iterationnr> <normalized-stepsize> <unnormalized-stepsize>", and all solution intervals are concatenated. File is not written when this parameter is empty.| |
| | <step>.subtract | bool | false | Subtracts the corrected model from the data. **NOTE** This may not work when you apply a uv-cut. | | |<step>.uvlambdamin|double|0|Ignore baselines / channels with UV < uvlambdamin wavelengths. **Note**: also all other variants of uv flagging described in [[#uvwflagger|UVWFlagger]] (uvmmin, uvmrange, uvlambdarange, etc) are supported (New in 3.1).| |
| | <step>.useidg | bool | false | Do image-based prediction using IDG.| | |<step>.subtract|bool|false|Subtracts the corrected model from the data. **NOTE** This may not work when you apply a uv-cut.| |
| | <step>.idg.image | string | "" | Name of fits file for IDG prediction. Should have the phase direction of the measurement set. | | |<step>.useidg|bool|false|Do image-based prediction using IDG.| |
| | <step>.idg.regions | string | "" | DS9 regions file describing the facets for IDG prediction. | | |<step>.idg.images|list|[]|Filename of ''.fits'' model images, one per frequency term. The terms are defined as for a polynomial source spectra (not logarithmic), e.g. see [[https://sourceforge.net/p/wsclean/wiki/ComponentList/|this WSClean page]]. The frequency in the metadata of the fits files is used as nu<sub>0</sub> in the polynomial evaluation.| |
| | <step>.savefacets | bool | false | Write out each facet as a fits file (named facet<N>.fits). Only useful when useidg=true. | | |<step>.idg.regions|string|""|DS9 regions file describing the facets for IDG prediction.| |
| | <step>.onlypredict | bool | false | Instead of solving, output the predicted visibilities instead. This is useful for testing, although when doing faceted prediction with IDG, it might be fast for certain cases. | | |<step>.idg.buffersize|int|Based on memory|Set the amount of timesteps that are to be used for each IDG buffer| |
| | <step>.applycal.*| | | ApplyCal sub-step, same as in Predict step. One can pass an h5parm with as many directions as set in "directions" and each direction model is corrupted accordingly. | | |<step>.savefacets|bool|false|Write out each facet as a fits file (named facet<N>.fits). Only useful when useidg=true.| |
| | |<step>.onlypredict|bool|false|Instead of solving, output the predicted visibilities instead. This is useful for testing, although when doing faceted prediction with IDG, it might be fast for certain cases.| |
| | |<step>.applycal.*| | |ApplyCal sub-step, same as in Predict step. One can pass an h5parm with as many directions as set in "directions" and each direction model is corrupted accordingly.| |
| | |
| | \\ |
| |
| ==== Predict ==== | ==== Predict ==== |
| | <step>.invert | bool | **true** | Invert the beam. When applying the beam to transfer calibration solutions, this should be true. In other words: ''invert=true'' means correcting for the beam, ''invert=false'' means corrupting with the beam. When using the beam in a predict (or gaincal) step, this option defaults to ''false'' (so it will corrupt for the beam). | | | <step>.invert | bool | **true** | Invert the beam. When applying the beam to transfer calibration solutions, this should be true. In other words: ''invert=true'' means correcting for the beam, ''invert=false'' means corrupting with the beam. When using the beam in a predict (or gaincal) step, this option defaults to ''false'' (so it will corrupt for the beam). | |
| | <step>.beammode | string | "default" | Beam mode to apply, can be "array_factor", "element" or "default". Default is to apply both the element beam and the array factor. | | | <step>.beammode | string | "default" | Beam mode to apply, can be "array_factor", "element" or "default". Default is to apply both the element beam and the array factor. | |
| | |
| | ==== SetBeam ==== |
| | SetBeam is an expert option and should only be used in rare cases. It allows direct manipulation of the beam-keywords for a column in a measurement set. Normally, DP3 registers whether the visibilities in a column are corrected for a beam or not, and if so, in what direction the beam was corrected for. This avoids incorrect corrections / scaling by the beam. However, certain actions can change the scaling of the visibilities without that the beam keywords are changed, in particular when predicting (either with DP3 or with another tool). When predicting a single source and not applying the beam, the visibilities are 'corrected' for the beam in the direction of the source. Under those circumstances, SetBeam can be used to modify the beam keywords. In that case, use direction for the source direction and beammode to default. |
| | | <step>.type | string | | Case-insensitive step type; must be 'setbeam' | |
| | | <step>.direction | string vector | [] | A RA/Dec value specifying in what direction the beam is corrected. | |
| | | <step>.beammode | string | "default" | Beam mode to apply, can be "array_factor", "element" or "default". Default means that sources in the given direction have corrected (intrinsic) flux values, i.e. they are corrected for the full beam. | |
| |
| ==== UVWFlagger ==== | ==== UVWFlagger ==== |