LOFAR stations are complete, self-contained radio telescopes. Every station is equipped with its own correlator. The stations can operate in 5 different frequency ranges: 10–80, 30–80, 110–190, 170–230, and 210–250 MHz. The first two modes use the LBA antennae, the others the HBA. The antenna voltages can be sampled at either 160 MHz or 200 MHz. The station correlator has the following operating modes:

• Computing dynamic spectra over the full frequency range of a certain

filter/clock combination for all antennae;

• Computing a full array correlation matrix in one sub band.

These data products are extremely useful for diagnosing problems in an observation.

Participants are assumed to have

• basic Python programming knowledge;
• an idea about the signal path within a station.

The goal of this exercise is to familiarize yourself with the dynamic spectra from individual dipoles. After this exercise you should be able to

• spot dead antennae;
• recognize, from the shape of the bandpass, the RCU filter / clock

combination that was used in recording a spectrum;

• recognize the most important sources of radio interference;
• understand the direction of the frequency axis in different nyquist zones.

• `lofarstation.py` is the Python software for reading

and plotting the station correlator data;

• `lofarstation.html` is a nicely rendered version of `lofarstation.py`;

• `cs010.py` contains the positions of the LBA dipoles on CS010;

• `dynamic-spectra` contains the subdirectories with the dynamic

spectra for this exercise;

• `Exercise-01A.txt` is the plain text version of this instruction;

• `Exercise-01A.html` is the HTML version of this instruction.

The first thing to do is changing to the exercise directory and starting the IronPython shell:

$ cd /data/lofarschool/users/station/Exercise-01A-station/
$ ipython -pylab lofarstation.py

the screen should now look something like this:

Python 2.5.1 (r251:54863, Jul 31 2008, 23:17:43) 
Type "copyright", "credits" or "license" for more information.
 
IPython 0.8.1 -- An enhanced Interactive Python.
?       -> Introduction to IPython's features.
%magic  -> Information about IPython's 'magic' % functions.
help    -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.
 
  Welcome to pylab, a matplotlib-based Python environment.
  For more information, type 'help(pylab)'.
 
In [1]: 

You may want to review the contents of [ link:lofarstation.html[lofarstation.py]

while doing this exercise.

The command `dynamic_spectrum_directories()` lists the directories that contain the dynamic spectra:

In [1]: dynamic_spectrum_directories()
Out[1]: 
[u'dynamic-spectra/20090128-school-dynspec-160-3',
 u'dynamic-spectra/20090128-school-dynspec-160-4',
 u'dynamic-spectra/20090128-school-dynspec-160-5',
 u'dynamic-spectra/20090128-school-dynspec-160-5-inv',
 u'dynamic-spectra/20090128-school-dynspec-160-6',
 u'dynamic-spectra/20090128-school-dynspec-160-7',
 u'dynamic-spectra/20090128-school-dynspec-200-3',
 u'dynamic-spectra/20090128-school-dynspec-200-4',
 u'dynamic-spectra/20090128-school-dynspec-200-5',
 u'dynamic-spectra/20090128-school-dynspec-200-5-inv',
 u'dynamic-spectra/20090128-school-dynspec-200-6',
 u'dynamic-spectra/20090128-school-dynspec-200-7',
 u'dynamic-spectra/20090201-school-dynspec-160-5',
 u'dynamic-spectra/20090201-school-dynspec-160-5-inv',
 u'dynamic-spectra/20090201-school-dynspec-160-6',
 u'dynamic-spectra/20090201-school-dynspec-160-7',
 u'dynamic-spectra/20090201-school-dynspec-200-5',
 u'dynamic-spectra/20090201-school-dynspec-200-5-inv',
 u'dynamic-spectra/20090201-school-dynspec-200-6',
 u'dynamic-spectra/20090201-school-dynspec-200-7']

The directory name contains information about the date (e.g. 20090128), clock frequency (160 or 200 MHz), RCU mode (3, 4, 5, 6, or 7), and whether spectral inversion was switched on (`-inv`). The RCU mode indicates which filter and antenna type was used. RCU modes 3 and 4 use the LBA antennae, while RCU modes 5, 6, and 7 use the HBA antennae. The RCU modes correspond to the following input / bandpass filter combinations:

The nyquist zones are labelled as follows:

We did not record spectra in RCU modes 1 and 2 because they are somewhat less interesting.

The contents of a directory can be viewed with e.g.

In [2]: full_path_listdir(dynamic_spectrum_directories()[0])
Out[2]: 
[u'dynamic-spectra/20090128-sch....ec-160-3/20090128_204311_sst_rcu000.dat',
 u'dynamic-spectra/20090128-sch....ec-160-3/20090128_204311_sst_rcu001.dat',
.
.
.
 u'dynamic-spectra/20090128-sch....ec-160-3/20090128_204311_sst_rcu094.dat',
 u'dynamic-spectra/20090128-sch....ec-160-3/20090128_204311_sst_rcu095.dat',
 u'dynamic-spectra/20090128-sch....ec-160-3/summary.txt']

The data file names themselves consist of the UTC date and time of the start of an observation, the letters `_sst_` to indicate that the file contains subband statistics (a dynamic spectrum), and the number of the receiver unit from which the spectrum was acquired.

The file `summary.txt` contains the command used to record the spectrum, which includes information about the total duration of the observation in seconds and the length of a single integration in seconds. Its contents can be viewed as follows:

In [3]: !cat dynamic-spectra/20090128-school-dynspec-160-3/summary.txt
rspctl --stati --duration=3600 --integration=10 \
   --directory=/data/20090128-school-dynspec-160-3

or alternatively:

In [4]: !cat $full_path_listdir(dynamic_spectrum_directories()[0])[96]
In [5]: !cat $full_path_listdir(dynamic_spectrum_directories()[0])[-1]

The `data_files()` function returns a list of all `.dat` files in a directory. The data file belonging to a certain RCU can then be selected by:

In [6]: files = data_files('dynamic-spectra/20090128-school-dynspec-200-3')
In [7]: files[12]
Out[7]: u'dynamic-spectra/20090128-sc...ec-200-3/20090129_014602_sst_rcu012.dat'

One can now display the dynamic spectrum of a single RCU using e.g.

In [8]: plot_dynamic_spectrum(files[0])

If one wants a proper frequency scale instead of a subband scale, one can try

In [9]: plot_dynamic_spectrum(files[12],
          frequencies_mhz=subband_frequencies_mhz(clock_mhz=200,
                nyquist_zone=1, spectral_inversion=False))

One can easily inspect all spectra in a directory by plotting some statistic of the spectra as a function of frequency/subband for every RCU. These statistics may be \'max', \'mean', \'median', or \'min'.

In [10]: plot_dynamic_spectrum_stat_all_rcus(
    'dynamic-spectra/20090128-school-dynspec-200-3/',
    statistic='median')
dynamic-spectra/20090128-school-dynspec-200-3//20090129_014602_sst_rcu000.dat
dynamic-spectra/20090128-school-dynspec-200-3//20090129_014602_sst_rcu001.dat
.
.
.

The remainder of this exercise consists of analysing all directories listed by `dynamic_spectrum_directories()` using the tools described above. Begin with the LBA datasets using the 200 MHz clock, followed by the 160 MHz clock. Do the same in the high band and work your way up in frequency. Attempt to answer the following questions for each data set:

• Is the frequency scale correct? This can be verified with the help of a couple of strong interference sources. In the LBA band there is an amateur HAM band at 27 MHz, as well as a strong FM transmitter at 88 MHz. In the HBA band there is a very strong narrow band signal at approximately 169–170 MHz. The highest HBA band contains a forest of military narrow band transmissions upwards of 240 MHz as well as a broad band digital signal between 223 and 225 MHz.

• Does the clock / RCU mode combination make sense? Why (not)?

• Which dipoles/receivers work correctly? Remember this: different is wrong. There are a couple of problematic datasets too. Which ones?

In [11]: plot_dynamic_spectrum_stat_all_rcus(
            'dynamic-spectra/20090201-school-dynspec-200-5/',
            statistic='mean',
            frequencies_mhz=subband_frequencies_mhz(clock_mhz=200,
                                                    spectral_inversion=False,
                                                    nyquist_zone=2))

In [12]: plot_dynamic_spectrum(
     data_files('dynamic-spectra/20090128-school-dynspec-200-3')[12],
     frequencies_mhz=subband_frequencies_mhz(200,
                                             nyquist_zone=1,
                                             spectral_inversion=False))

← Data Processing School