This page describes BBS (BlackBoard Selfcal), a calibration package for LOFAR observations.

BBS has two modes of running. The first is to run as a stand-alone tool to calibrate one subband. If a single subband does not contain enough signal, BBS offers the possibility to use data from multiple subbands together for parameter estimation. This is called /global/ parameter estimation. On this page, we will focus on the stand-alone version; setting up a global solve is described elsewhere.

Currently this page is work in progress, see Configuration Syntax for the old version

To calibrate a single MS, execute the calibrate-stand-alone script on the command line:

calibrate-stand-alone -f <MS> <parset> <source catalog>

The important arguments provided to the calibrate-stand-alone script in this example are:

  • <MS>: name of the MS to process.
  • <parset>: description of the reduction (see below).
  • <source catalog>: A list of sources that can be used for calibration (see below). When the MS already contains a sourcedb, it is not necessary to specify a new source catalog if you're calibrating on the same sourcedb.
  • The -f option, which overwrites stale information from previous runs (remove old instrument table and old source model if they exist with the measurement set).
  • The -t option, which provides some basic multithreading.

You can run the script without arguments for a description of all the options and the mandatory arguments, such as the -v option to get a more verbose output.

A source catalog is a plain text file containing information on your model. An example catalog file is shown below:

# (Name,Type,Ra,Dec,I, ReferenceFrequency='55.468e6', SpectralIndex) = format
3C196, POINT, 08:13:36.062300, +, 153.0, , [-0.56, -0.05212]

More information about the format is on the makesourcedb page. You can use to get a source catalog from an existing catalog.

The parset file describes the steps in the reduction you want to perform. A very basic example parset is:

Strategy.ChunkSize = 100
Strategy.Steps = [predict]

Step.predict.Model.Sources     = []
Step.predict.Operation         = PREDICT
Step.predict.Output.Column     = MODEL_DATA
Step.predict.Model.Beam.Enable = False

An especially important key is Strategy.ChunkSize, which specifies the amount of data (in timeslots) that is read into main memory as a single block. If your measurement set is large, be sure to fill in a sensible value here. It should be at least as large as the largest Solve.CellSize.Time value in your reduction. Also, you do not want to make it too small, because repeatedly reading small chunks hurts performance. It is advisable to ensure all Solve.CellSize.Time values in your reduction are divisors of Strategy.ChunkSize, to avoid truncated solution cells.

  • The tool can be used to plot the calibration solutions.
  • Usually, the corrected data needs another round of flagging before imaging because of bad calibration solutions. Bad solutions may occur for various reasons, e.g. because most of the samples in a solution cell are flagged due to RFI or because some of the calibrators are not above the horizon (when not using a beam model).
Parameter type default description
Strategy.InputColumn string DATA Name of the column that contains the input data
Strategy.Baselines string *& Baselines to read. The CASA baseline selection syntax should be used here. If this key is not specified, all cross-correlations will be selected. Examples:
Strategy.Baselines = *&              # Select all cross-correlations (default).
Strategy.Baselines = CS*&&RS*;CS*&   # Select all cross- and auto-correlations
                                     # between core and remote stations, and all 
                                     # cross-correlations between core stations.
Strategy.TimeRange string vector [] Time range to process, expressed as date/time string(s) (as returned by msinfo). All timeslots will be used if this field is left empty. Either a range or a start time can be provided. Examples:
Strategy.TimeRange = [27-Jul-2007/16:05:04]
Strategy.TimeRange = [27-Jul-2007/16:05:04, 28-Jul-2007/13:05:04]
Strategy.ChunkSize integer Chunk size in timeslots. A chunk represents an amount of input data that is loaded into memory and processed as a whole. This is useful when the amount of visibility data is too large to fit into main memory. A value of zero means all the data will be loaded into memory.
Strategy.UseSolver bool F Use a global solver? (If you specify True here, you should use the calibrate script.)
Strategy.Steps string vector [] The names of steps that compose the strategy. If you leave this empty, BBS will not do anything (and give an error).
Parameter type default description
Step.<StepName>.Operation string Operation to be performed in this step. Possible values are
PREDICT: Simulate visibilities
ADD: Add simulated visibilities to the input visibilities
SUBTRACT: Subtract predicted visibilities from the input visibilities
CORRECT: Correct the input visibilities
SOLVE: Fit model parameters, perform a calibration
Step.<StepName>.Model Description of the model to be used. Only subkeys of this key should be specified. See below
Step.<StepName>.Baselines string *& Baselines to process. See the key Strategy.Baselines
Step.<StepName>.Correlations string [] Correlations to process. If this key is not specified, all correlations will be selected.
Step.simulate.Correlations = [XX,YY]
Step.simulate.Correlations = [RR,RL]
Step.<StepName>.Output.Column string (empty) Column in the observation part where the output values of this step should be written. If left empty, no data will be written.
Step.<StepName>.Output.WriteFlags bool F If set to true the flags in the observation will be updated
Step.<StepName>.Output.WriteCovariance bool F If set to true, covariance will be written to the covariance column linked to the column specified at Output.Column (only for experienced users: the covariance column can be converted to a WEIGHTS column by using an external script)
Parameter type default description
Step.<StepName>.Solve.Parms string vector [] Parameters to fit. Shell style wildcards are recognized (?,*,{}). If specified, this key shall not be empty, or an exception will be thrown.
Solve.Parms = ["Gain:{0:0,1:1}:*"]
Step.<StepName>.Solve.ExclParms string vector [] Subset of the parameters selected by Parms that should not be fitted.
Step.<StepName>.Solve.Mode string COMPLEX Determines how the (complex valued) data and the (complex valued) model are compared. Options are: COMPLEX, PHASE, AMPLITUDE. The COMPLEX mode compares both phase and amplitude, the PHASE mode compares phases and ignores amplitude differences, the AMPLITUDE mode compares amplitudes and ignores phase differences. NB. Comparing only phases is not necessarily equivalent to phase only calibration (and likewise for comparing amplitudes). If you want to do phase only calibration, specify PHASE here.
Step.<StepName>.Solve.CellSize.Freq integer ! Solution grid cell size (frequency): A chunk is divided into solutions cells and a solution is computed for each cell in total independently. This parameter specifies the (nominal) number of channels that are grouped together in a cell. NB. When performing a global solve this parameter is ignored and the frequency range of the calibration group is used instead.
Step.<StepName>.Solve.CellSize.Time integer ! NB. When performing a global solve this parameter is ignored and the frequency range of the calibration group is used instead.
Step.<StepName>.Solve.CellChunkSize integer ! Specifies the number of solution cells along the time axis to process simultaneously. A value of zero (0) selects all solution cells in the current chunk. Can be used to tune memory usage.
Step.<StepName>.Solve.PropagateSolutions bool F If set to true, then the solutions of the previous cell chunk are used as initial values for the next cell chunk. NB. Even if set to true, solutions are not propagated to the next chunk of visibility data. Furthermore, no convergence check is done, so bad solutions may contaminate the solutions in the next cell chunk if this parameter is set to true.
Step.<StepName>.Solve.UVRange float vector [] A list containing either a single number (minimal UV length), or two numbers (minimal, maximal UV length). All UV lengths should be specified in wavelengths. Visibility data from baselines that do not meet the criterion are ignored when estimating model parameters. An empty list means no UV range selection will be performed.
Solve.UVRange = [250.0]         # Use only visibility data from baselines 
                                # with a UV length larger than 250.0 wavelengths.
Solve.UVRange = [250.0, 5000.0] # Use only visibility data from baselines with a 
                                # UV length between 250.0 and 5000.0 wavelengths.
Step.<StepName>.Solve.Options.MaxIter integer 0 Convergence criterion: Iteration is halted if the number of iterations exceeds this value. A value of zero means unlimited. NB. There is a known off by one bug, so in practice a value of x results in x+1 iterations.
Step.<StepName>.Solve.Options.EpsValue float 1e-8 Convergence criterion: Iteration is halted if the minimal relative change in the norm of the solutions is smaller than this value.
Step.<StepName>.Solve.Options.EpsDerivative float 1e-8 Convergence criterion: Iteration is halted if the maximum of the known vector of the equations to be solved is less than this value.
Step.<StepName>.Solve.Options.ColFactor float 1e-6 Colinearity factor. Used to test for solvability of the normal equations. If not solvable an error is returned, unless Options.UseSVD is set to true (in which case Singular Value Decomposition is used to compute a solution). See
Step.<StepName.Solve.Options.LMFactor float 1e-3 Initial Levenberg-Marquardt factor, which controls the balance between gradient descent and Newton-Raphson optimization.
Step.<StepName>.Solve.Options.BalancedEqs bool F If set to true, the Levenberg-Marquardt factor is added (“in some way?”) to the diagonal elements of the normal equation matrix. Otherwise, the diagonal elements are multiplied by (1 + factor). See
Step.<StepName>.Solve.Options.UseSVD bool F If set to true, Singular Value Decomposition is used to compute a solution when the normal equations cannot be triangularized.
Parameter type default description
Step.<stepname>.Model.Sources string vector [] List of sources to include in the model. Shell style wildcards are allowed. An empty list will select all sources in the sky model.
Step.<stepname>.Model.Cache.Enable bool T If set to true, intermediate results will be cached. This is especially useful when solving, because it will prevent unnecessary recomputation at the start of each iteration. This will require a lot of memory, but does speed BBS up.
Step.<stepname>.Model.<effect>.Enable bool F Example: Step.Solve.Model.Clock.Enable=True to enable the clock.

Below are all the effects that can be modeled, in the order they are applied (from sky to ground).

Effect Description Direction dependent? Options
ScalarPhase A phase that is equal for both dipolesDirection dependent -
Rotation Faraday Rotation without frequency dependency (note: parameter name is RotationAngle)Direction dependent -
FaradayRotation Faraday Rotation (note: parameter name is RotationMeasure Direction dependent -
DirectionalTEC TEC (ionosphere) Direction dependent -
Beam The LOFAR beam model Direction dependent See below
DirectionalGain Directional gain Direction dependent Model.DirectionalGain.Phasors (True or False, default is False): express parameters in amplitude and phase instead of real and imaginary part
CommonScalarPhase Scalar Phase Direction independent -
CommonRotation Rotation (note: parameter name is CommonRotationAngle)Direction independent -
TEC TEC (ionosphere) Direction independent 'Model.TEC.Split' (True or False, default is False): have a separate TEC for X and Y dipoles (this is not physical, but used for testing)
Gain Gain Direction independent Model.Gain.Phasors (True or False, default is False): express parameters in amplitude and phase instead of real and imaginary part
Clock Clock Direction independent Model.Clock.Split (True or False, default is False): have a separate clock for X and Y dipoles

The beam model tries to emulate all kinds of distortions to the signal that are caused by the beam of the station. These effects are split into two parts: the element beam, which is the response of a single dipole, and the array factor, which emulates the effect of combining the signal of the many dipoles in a station. In HBA, the array factor model also models the effect of the analog tile beam former.

To have a look at different elements of the beam, you can specifically use only the element beam or only the array factor (if you don't know the details, you need both the element beam and the array factor, which is the default). The options are:

Model.Beam.Mode = ELEMENT       # only element beam
Model.Beam.Mode = ARRAY_FACTOR  # only array factor
Model.Beam.Mode = DEFAULT       # both element beam and array factor (default)

The tile beam former in the HBA tiles forms a beam for a certain reference frequency. When modeling this beam, the beam model should of course do this for the same frequency. Usually, this works automatically: the reference frequency is stored in the measurement set. Things are different when you compress a number of subbands as channels into one measurement set (usually done with NDPPP). Then each `channel' was beamformed at a different reference frequency. In this case, the reference frequency is only right for one of the `channels'. To handle this case well, there is an option that tells the beam model to use the channel frequency (which is usually the center frequency of the compressed subband). This option is:

Step.Solve.Model.Beam.UseChannelFreq = T

Note that the beam model is a direction dependent effect like any other in BBS. That means that over a patch, the beam is assumed to be constant (it is evaluated at the centroid of the patch). This may change in the future.

  • Last modified: 2017-03-08 15:27
  • by