===== Data Processing Workshop :: Exercise 01A =====
^Source directory| /data/lofarschool/users/station/Exercise-01A-station|
^Contact person | Michiel Brentjens |
==== Context ====
LOFAR stations are complete, self-contained radio telescopes. Every
station is equipped with its own correlator. The stations can operate
in 5 different frequency ranges: 10--80, 30--80, 110--190,
170--230, and 210--250 MHz. The first two modes use the LBA
antennae, the others the HBA. The antenna voltages can be sampled at
either 160 MHz or 200 MHz. The station correlator has the
following operating modes:
* Computing dynamic spectra over the full frequency range of a certain
filter/clock combination for all antennae;
* Computing a full array correlation matrix in one sub band.
These data products are extremely useful for diagnosing problems in an
observation.
====Prerequisites====
Participants are assumed to have
* basic Python programming knowledge;
* an idea about the signal path within a station.
====Description====
The goal of this exercise is to familiarize yourself with the dynamic
spectra from individual dipoles. After this exercise you should be
able to
* spot dead antennae;
* recognize, from the shape of the bandpass, the RCU filter / clock
combination that was used in recording a spectrum;
* recognize the most important sources of radio interference;
* understand the direction of the frequency axis in different nyquist zones.
====Files & Directories====
* `lofarstation.py` is the Python software for reading
and plotting the station correlator data;
* `lofarstation.html` is a nicely rendered version of `lofarstation.py`;
* `cs010.py` contains the positions of the LBA dipoles on CS010;
* `dynamic-spectra` contains the subdirectories with the dynamic
spectra for this exercise;
* `Exercise-01A.txt` is the plain text version of this instruction;
* `Exercise-01A.html` is the HTML version of this instruction.
====Step-by-step instructions====
The first thing to do is changing to the exercise directory and
starting the IronPython shell:
$ cd /data/lofarschool/users/station/Exercise-01A-station/
$ ipython -pylab lofarstation.py
the screen should now look something like this:
Python 2.5.1 (r251:54863, Jul 31 2008, 23:17:43)
Type "copyright", "credits" or "license" for more information.
IPython 0.8.1 -- An enhanced Interactive Python.
? -> Introduction to IPython's features.
%magic -> Information about IPython's 'magic' % functions.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.
Welcome to pylab, a matplotlib-based Python environment.
For more information, type 'help(pylab)'.
In [1]:
You may want to review the contents of
[
link:lofarstation.html[lofarstation.py]
while doing this exercise.
The command `dynamic_spectrum_directories()` lists the directories
that contain the dynamic spectra:
In [1]: dynamic_spectrum_directories()
Out[1]:
[u'dynamic-spectra/20090128-school-dynspec-160-3',
u'dynamic-spectra/20090128-school-dynspec-160-4',
u'dynamic-spectra/20090128-school-dynspec-160-5',
u'dynamic-spectra/20090128-school-dynspec-160-5-inv',
u'dynamic-spectra/20090128-school-dynspec-160-6',
u'dynamic-spectra/20090128-school-dynspec-160-7',
u'dynamic-spectra/20090128-school-dynspec-200-3',
u'dynamic-spectra/20090128-school-dynspec-200-4',
u'dynamic-spectra/20090128-school-dynspec-200-5',
u'dynamic-spectra/20090128-school-dynspec-200-5-inv',
u'dynamic-spectra/20090128-school-dynspec-200-6',
u'dynamic-spectra/20090128-school-dynspec-200-7',
u'dynamic-spectra/20090201-school-dynspec-160-5',
u'dynamic-spectra/20090201-school-dynspec-160-5-inv',
u'dynamic-spectra/20090201-school-dynspec-160-6',
u'dynamic-spectra/20090201-school-dynspec-160-7',
u'dynamic-spectra/20090201-school-dynspec-200-5',
u'dynamic-spectra/20090201-school-dynspec-200-5-inv',
u'dynamic-spectra/20090201-school-dynspec-200-6',
u'dynamic-spectra/20090201-school-dynspec-200-7']
The directory name contains information about the date
(e.g. 20090128), clock frequency (160 or 200 MHz), RCU mode (3,
4, 5, 6, or 7), and whether spectral inversion was switched on
(`-inv`). The RCU mode indicates which filter and antenna type was
used. RCU modes 3 and 4 use the LBA antennae, while RCU modes 5, 6,
and 7 use the HBA antennae. The RCU modes correspond to the following
input / bandpass filter combinations:
^1 | 50 Ohm load| 10--80 MHz |
^2 | 50 Ohm load| 30--80 MHz |
^3 | LBA | 10--80 MHz |
^4 | LBA | 30--80 MHz |
^5 | HBA | 110--190 MHz|
^6 | HBA | 170--230 MHz|
^7 | HBA | 210--250 MHz|
The nyquist zones are labelled as follows:
^ Clock ^ Freq. range ^ Nyquist zone|
^ 160 MHz | 00--80 MHz | 1|
^ | 80--160 MHz | 2|
^ | 160--240 MHz | 3|
^ | 240--320 MHz | 4|
^ 200 MHz | 000--100 MHz | 1|
^ | 100--200 MHz | 2|
^ | 200--300 MHz | 3|
We did not record spectra in RCU modes 1 and 2 because they are
somewhat less interesting.
The contents of a directory can be viewed with e.g.
In [2]: full_path_listdir(dynamic_spectrum_directories()[0])
Out[2]:
[u'dynamic-spectra/20090128-sch....ec-160-3/20090128_204311_sst_rcu000.dat',
u'dynamic-spectra/20090128-sch....ec-160-3/20090128_204311_sst_rcu001.dat',
.
.
.
u'dynamic-spectra/20090128-sch....ec-160-3/20090128_204311_sst_rcu094.dat',
u'dynamic-spectra/20090128-sch....ec-160-3/20090128_204311_sst_rcu095.dat',
u'dynamic-spectra/20090128-sch....ec-160-3/summary.txt']
The data file names themselves consist of the UTC date and time of the
start of an observation, the letters `_sst_` to indicate that the file
contains subband statistics (a dynamic spectrum), and the number of
the receiver unit from which the spectrum was acquired.
The file `summary.txt` contains the command used to record the
spectrum, which includes information about the total duration of the
observation in seconds and the length of a single integration in
seconds. Its contents can be viewed as follows:
In [3]: !cat dynamic-spectra/20090128-school-dynspec-160-3/summary.txt
rspctl --stati --duration=3600 --integration=10 \
--directory=/data/20090128-school-dynspec-160-3
or alternatively:
In [4]: !cat $full_path_listdir(dynamic_spectrum_directories()[0])[96]
In [5]: !cat $full_path_listdir(dynamic_spectrum_directories()[0])[-1]
The `data_files()` function returns a list of all `.dat` files in a
directory. The data file belonging to a certain RCU can then be
selected by:
In [6]: files = data_files('dynamic-spectra/20090128-school-dynspec-200-3')
In [7]: files[12]
Out[7]: u'dynamic-spectra/20090128-sc...ec-200-3/20090129_014602_sst_rcu012.dat'
One can now display the dynamic spectrum of a single RCU using e.g.
In [8]: plot_dynamic_spectrum(files[0])
If one wants a proper frequency scale instead of a subband scale, one
can try
In [9]: plot_dynamic_spectrum(files[12],
frequencies_mhz=subband_frequencies_mhz(clock_mhz=200,
nyquist_zone=1, spectral_inversion=False))
One can easily inspect all spectra in a directory by plotting some
statistic of the spectra as a function of frequency/subband for every
RCU. These statistics may be \'max', \'mean', \'median', or \'min'.
In [10]: plot_dynamic_spectrum_stat_all_rcus(
'dynamic-spectra/20090128-school-dynspec-200-3/',
statistic='median')
dynamic-spectra/20090128-school-dynspec-200-3//20090129_014602_sst_rcu000.dat
dynamic-spectra/20090128-school-dynspec-200-3//20090129_014602_sst_rcu001.dat
.
.
.
The remainder of this exercise consists of analysing all directories
listed by `dynamic_spectrum_directories()` using the tools described
above. Begin with the LBA datasets using the 200 MHz clock,
followed by the 160 MHz clock. Do the same in the high band and
work your way up in frequency. Attempt to answer the following
questions for each data set:
* Is the frequency scale correct? This can be verified with the help of a couple of strong interference sources. In the LBA band there is an amateur HAM band at 27 MHz, as well as a strong FM transmitter at 88 MHz. In the HBA band there is a very strong narrow band signal at approximately 169--170 MHz. The highest HBA band contains a forest of military narrow band transmissions upwards of 240 MHz as well as a broad band digital signal between 223 and 225 MHz.
* Does the clock / RCU mode combination make sense? Why (not)?
* Which dipoles/receivers work correctly? Remember this: different is wrong. There are a couple of problematic datasets too. Which ones?
====Example outputs====
{{exercise-01a-stats-mean-200-5.png?600}}
In [11]: plot_dynamic_spectrum_stat_all_rcus(
'dynamic-spectra/20090201-school-dynspec-200-5/',
statistic='mean',
frequencies_mhz=subband_frequencies_mhz(clock_mhz=200,
spectral_inversion=False,
nyquist_zone=2))
{{exercise-01a-dynspec-rcu12-200-3.png?600}}
In [12]: plot_dynamic_spectrum(
data_files('dynamic-spectra/20090128-school-dynspec-200-3')[12],
frequencies_mhz=subband_frequencies_mhz(200,
nyquist_zone=1,
spectral_inversion=False))
\\
----
<- [[public:meetings:2009-02_data_processing_school|Data Processing School]]