Workshops :: LOFAR Data Processing School (10.-12. Feb 2009, ASTRON, Dwingeloo)
Who should attend?
The aim of this workshop is to introduce new members of the collaboration, who will be involved in the commissioning effort, to LOFAR. Students, postdocs, and staff are all encouraged to apply. The school will cover the many aspects of the LOFAR system from the operation of the basic station hardware to the software pipelines and science products they produce. Lectures and tutorials will be presented by members of the LOFAR project team as well as staff from the many institutions involved in the collaboration. Hands-on sessions will be provided to give attendees an opportunity to gain experience with real LOFAR data.
Presentations will be given at a level appropriate for someone new to the LOFAR system. Familiarity with the concepts of radio interferometry and standard data processing software such as CASA, AIPS, or Myriad will be useful but not required. Minimum requirements should include familiarity with scripting languages and in particular Python. Parallel sessions for more expert students are also planned.
Attendance will be limited to 35 people, so initial preference will be given to registrations from members of the LOFAR Key Science Project teams and other international partners contributing to the commissioning effort. Registrations from other applicants will be supported on a best-effort basis.
Schedule
Presentations
Tuesday, 10. Feb
- A. Gunst: System overview
- M. Brentjens: Station data
- R. Nijboer: Imaging pipeline
- V.N. Pandey: Data preparation
- M. Wise: LOFAR Data Products
- G. van Diepen: Data Handling Tools (pyrap)
Wednesday, 11. Feb
- G. de Bryun: Calibration
- B. Stappers: Pulsar pipeline
- J. Hessels: Pulsar data
- O. Smirnov: MeqTrees Practical Session
Thursday, 12. Feb
- A. Horneffer: CR pipelines
- L. Bähren: CR Data
- H. Holties: LOFAR Archive
Connecting to the cluster nodes
Logging into the cluster nodes requires passing through at least one – often two – front-end node. So in order to access the cluster nodes from a machine located at ASTRON you will have to go through the following chain of connections:
ASTRON -> listfen (129.125.99.50) -> lioff{021}-{029}
While this is the default sequence to connect to the reserved cluster nodes, the intermediate step going through the front-end-node (lioffen) can be spared.
Here is the procedure for connecting to the cluster nodes set aside for the data processing school:
- @ ASTRON :
ssh -X <login>@129.125.99.50
This will establish a connection to the front-end node (listfen)
- @ listfen :
ssh -X lioff021
This will establish a connection to one of the I/O cluster nodes set aside for actual processing.
To initialize your environment for a package (e.g. AIPS):
source /app/scripts/do<package>
For an overview of available initialization scripts:
ls /app/scripts
Hands-on exercises
Directories & file locations
The workshop will mainly utilize some of the LOFAR offline cluster nodes to run the exercises. These machines are named “lioff0XX” where X goes from 21-29. To reach these machines, you have to first login to listfen. So ssh into listfen and then ssh into one of the working nodes.
On these machines, there is an area set aside named “/data/lofarschool” where all the exercises will be stored. Within that directory there will be two sub-directories, one called “users” which the students will use as working areas and one called “data” which will contain subdirectories for each of the exercises:
/data/lofarschool |-- data | |-- Exercise_01 | |-- Exercise_02 | |-- Exercise_03 | ` `-- users
Each directory subdirectory will contain all the inputs, expected outputs, scripts, logfiles, instructions, etc associated with a given exercise, i.e. :
lioff023:/data/lofarschool/data/Exercise_01> ll total 8 -rw-r--r-- 1 wise wise 0 2009-02-04 19:26 example_01.log -rw-r--r-- 1 wise wise 0 2009-02-04 19:26 example_01.py drwxr-xr-x 2 wise wise 4096 2009-02-04 19:25 inputs/ -rw-r--r-- 1 wise wise 0 2009-02-04 19:27 instructions.txt drwxr-xr-x 2 wise wise 4096 2009-02-04 19:25 outputs/ -rw-r--r-- 1 wise wise 0 2009-02-04 19:26 README lioff023:/data/lofarschool/data/Exercise_01>
Preparation
Everything in “/data/lofarschool/data” will be write protected. If users want to modify any of the input files or scripts, they can copy them to them working areas:
/data/lofarschool/users
and there create a new directory as your working areacd /data/lofarschool/users mkdir myname cd myname
- Create a working copy of the exercise within your directory, so e.g.
cp -r /data/lofarschool/data/Exercise-50 .
Exercise instructions
- Exercise 01A – Station Data
- Exercise 02B – Station Data
- Exercise 05 – Flagging, Compression along Freq, Time with DPPP
- Exercise 08 – Calibration with BBS
- Exercise 10 – AIPS and ParselTongue
- Exercise 45 – Visualizing the LOFAR LBA Beam
- Exercise 50 – Working with the Data Access Layer
- Exercise 51 – Working with the Data Access Layer
- Exercise 52 – Working with the Data Access Layer
- Exercise 53 – Working with the Data Access Layer
- Exercise 60 – Inspecting, flagging and imaging with CASA
- Exercise 61 – Inspecting, flagging and imaging with CASA
- Exercise 62 – Inspecting, flagging and imaging with CASA
- Exercise 63 – Inspecting and plotting visibility data with pyrap
- Exercise 64 – Image manipulation with pyrap
- Exercise PSR 01 – Pulsars/Beam-formed Data: Inspecting Timeries and Power Spectra
- Exercise PSR 02 – Pulsars/Beam-formed Data: Folding and Dedispersing to Detect a Pulsar
- Exercise PSR 03 – Pulsars/Beam-formed Data: Creating and Examining a Dynamic Spectrum
- Exercise CR0 – Setup, building of executables
- Exercise CR1 – DataReader
- Exercise CR2 – Dynamic spectrum from LOFAR data
- Exercise CR3 – LOPES CR-pipeline