Raw data products

  • Raw observational data
    The raw observational data is recorded in measurement-set (MS) format. A survey field observation consists of forty beams covering the field; with all dishes the size of this dataset is 4.6 TB. A set of calibration scans consists of 40 separate observations taken in succession. For each calibrator scan only one beam contains the calibrator. All other beams not containing the calibrator are discarded. These observations vary in between 3–5 minutes and the total datasize for a complete calibrator scan (all forty beams) is 64-106 GB. The calibrator observations are taken at a higher time resolution than the survey fields (10 vs. 30 seconds) to allow better RFI excision due to their shorter integration time. The raw data are stored in long-term storage at SurfSARA to enable future reprocessing with improvements to the Apercal pipeline.
  • Inspection plots
    Upon ingest to ALTA, inspection plots showing various views of data quality are created. The majority of these plots show the behavior per compound beam, with different slices of the data to highlight different aspects of quality. Example inspection plots can be found here. These plots are:

    • Amplitude(all): Amplitude across all beams, averaged over all parameters
    • Amplitude f(time): Amplitude across all beams, concentric circles as a function of time inwards (start to end)
    • Amplitude f(antenna):  Amplitude across all beams, concentric circles as a function of antenna inwards (RT2 to RTD)
    • Amplitude f(baseline): Amplitude across all beams, concentric circles as a function of baseline inwards (in MS order)
    • Phase (all): Phase across all beams, averaged over all parameters
    • Phase f(time): Phase across all beams, concentric circles as a function of time inwards (start to end)
    • Phase f(antenna):  Phase across all beams, concentric circles as a function of antenna inwards (RT2 to RTD)
    • Phase f(baseline): Phase across all beams, concentric circles as a function of baseline inwards (in MS order)
    • Amplitude vs time: Baseline plot of amplitude as function of time, coloured by MS order of baselines
    • Amplitude vs channel: Baseline plot of amplitude as function of channel, coloured by MS order of baselines
    • Phase vs time: Baseline plot of phase as function of time, coloured by MS order of baselines
    • Phase vs channel: Baseline plot of phase as function of channel, coloured by MS order of baselines
    • Waterfall: scaled amplitude : Waterfall plot of amplitude as a function of channel (x-axis) and time (y-axis),with fixed scaling
    • Waterfall: unscaled amplitude:  Waterfall plot of amplitude as a function of channel (x-axis) and time (y-axis), no scaling
    • Waterfall: scaled phase: Waterfall plot of phase as a function of channel (x-axis) and time (y-axis), fixed scale
    • Waterfall: unscaled phase: Waterfall plot of phase as a function of channel (x-axis) and time (y-axis), no scaling

In addition, there are plots for a single beam, either the central compound beam for a target observation or the compound beam containing a calibrator for calibrator observations showing a full view of the data. These plots are:

    • Beams XX: Waterfall plots of phase (upper right) and amplitude (lower left) for the XX polarization as a function of frequency (x-axis) and time (y-axis) for each baseline pair given by the antenna names (middle diagonal)
    • Beams YY: Waterfall plots of phase (upper right) and amplitude (lower left) for the YY polarization as a function of frequency (x-axis) and time (y-axis) for each baseline pair given by the antenna names (middle diagonal)

 

Processed data products

The Apercal pipeline (see “Apercal overview and structure”) calibrates the data and produces final images and cubes per compound beam. All data products described below are for an individual compound beam, of which there are forty per observation. These data products are not primary-beam corrected; primary-beam images are provided separately (see the "Primary beam response" section, specifically “Released primary beam images”, and User Interfaces”); these may be used for mosaicking or for correction of individual images. 

Processed data products are ingested back into ALTA on a per-beam basis, and in “Released processed data products” we motivate the choice to release data on a per-beam basis. Note that the release of a beam releases all processed data associated with that beam.

An important note is that currently only the upper 150 MHz of the band is processed; thus the processed data products are produced over the range 1280-1430 MHz. Moreover, the first 12.5 MHz of data are flagged due to persistent RFI, therefore the resulting central frequency is 1361.25 MHz. The nominal bandwidth is then 137.5 MHz, but effectively it could be smaller due to additional RFI flagging. The (currently archived) data products are described below:

  • Calibration tables
    The crosscal module portion of Apercal produces calibration tables that can be applied to the original data to reproduce the cross calibration. These tables are in standard CASA/MS format and a summary of all the calibration tables is provided in Table 1, where 3CFFF refers to a flux/bandpass calibrator (namely, 3C147 or 3C196; occasionally 3C295) and 3CPPP to a polarization calibrator (namely, 3C286 or 3C138).
Table description Table name
Global delays 3CFFF.K
Bandpass solutions 3CFFF.Bscan
Intermediary, initial phase solutions 3CFFF.G0ph
Final complex gain solutions 3CFFF.G1ap
Crosshand delay 3CPPP.Kcross
Leakage terms 3CFFF.Df
XY phase offset 3CPPP.Xf

Table 1. Calibration tables and naming scheme.

 

  • Full time, Stokes and spectral-resolution calibrated visibility data
    After the completion of the self-calibration, the final selfcal solutions are applied to the full spectral resolution uv data, and this uv dataset is written out as a UVFITS file as an intermediate data product. Currently, the final calibrated uv-data are stored at full time and spectral resolution for all polarizations; this may change with future updates to the processing pipeline.
  • Continuum images
    A multi-frequency Stokes I image is created over the full processed frequency range (currently 150 MHz; effectively less due to RFI occupancy) and saved as a FITS file for each beam. These images are 3.4°×3.4°(3073×3073 pixels, with 4′′/pixel). This samples well outside the primary beam response but is necessary to account for cases where a strong source is in a side-lobe and needs to be included in the self-calibration model and cleaning. Sources above 5-σ should be identified and cleaned to the 1-σ level.
  • Polarization images and cubes
    A multi-frequency Stokes V image over the full bandwidth is produced. This image matches the continuum image in spatial extent: 3.4°×3.4°(3073×3073 pixels, with 4′′/pixel). In order to prevent bandwidth depolarization and enable rotation measure synthesis studies, Stokes Q and U cubes with a frequency resolution of 6.25 MHz are produced. The cubes have a smaller spatial extent of 2.7°×2.7°(2049×2049 pixels, with 4′′/pixel).
  • Line and dirty beam cubes
    Four line cubes over a set of different frequency ranges are produced. Table 2 summarizes the covered frequency ranges and provides the corresponding redshift range for HI. The lowest redshift cube is produced at full spectral resolution while other cubes are produced with a 3-channel averaging. These cubes have a spatial extent of 1.1°×1.1°(661×661 pixels, with 6′′/pixel). As the Apercal pipeline does not provide source finding or cleaning of the line cubes, corresponding dirty beam cubes, with twice the spatial coverage, are also archived to allow offline cleaning of source emission.
Cube Frequency Range Velocity Range Redshift range Frequency Resolution  Velocity Resolution
 MHz  kms  kHz  kms
Cube0  1292.5 -- 1337.1  18110 -- 28226  0.062 -- 0.099  36.6  8.3
Cube1  1333.1 -- 1377.7  9155 -- 19005  0.031 -- 0.065  36.6  8.1
Cube2  1373.8 -- 1418.4  424 -- 10005 0.001 -- 0.034  36.6  7.9
Cube3  1414.5 -- 1429.3  -1873 -- 1250  0 -- 0.004  12.2  2.6

Table 2. Frequency/velocity coverage and resolution for all line cubes. Velocities use the optical definition and velocity resolutions are for the center frequency/velocity of each cube.

A summary of all the archived data products and their sizes (per beam and for a complete observation) are provided in Table 3:

Data product  Format  Dimensions  Size per beam  Size per observation
Survey field raw visibility data   MS  4 x 24576  117 GB   4.7 TB
Calibrator raw visibility data  MS  4 x 24576  1.6-2.6 GB  64-104 GB
Calibration tables  MS table  --  660 MB  26.3GB
Self-calibrated visibility data  uvfits  4 x 12288  58 GB   2.3TB
Multi-frequency synthesis beam images  fits  3073 x 3073  37 MB  1.5 GB
Stokes Q and U cubes  fits  2049 x 2049 x 24  1.5 GB  62 GB
Stokes V multi-frequency synthesis image  fits  3073 x 3073  37 MB  1.5 GB
Continuum-subtracted dirty line cubes  fits  661 x 661 x 1218  8  320 GB
Restoring beam cubes  fits  1321 x 1321 x 1218  320  1.28 TB

Table 3. Summary of archived data products, including their format and sizes. For visibility data, the dimensions are for polarization and frequency. For images and cubes, the dimensions are for spatial sizes and frequency (when appropriate). The calibrator visibility sizes are after pruning to keep only the Apertif beam that contains the calibrator, and the range of sizes reflects the different calibrator scan lengths.

Latest tweets

Daily Image of the Week: Imaging solar wind scintillation with LOFAR https://bit.ly/3cQfYZl

Daily Image of the Week: Weighing the Milky Way? Yes we CAN! https://bit.ly/3qXGNTY

Daily Image of the Week: New aerial image of the LOFAR superterp. https://bit.ly/3n8prRT

Volcanic ‘activity’ in black holes blows monumental bubbles spanning hundreds of thousands of light years, impacting everything in their way! Study by an international group of astronomers using @LOFAR published in @NatureAstronomy today: https://bit.ly/3aISQuK

searchtwitter-squarelinkedin-squarebarsyoutube-playinstagramfacebook-officialenvelopecrosschevron-right