public:meetings:2009-02_data_processing_school

The aim of this workshop is to introduce new members of the collaboration, who will be involved in the commissioning effort, to LOFAR. Students, postdocs, and staff are all encouraged to apply. The school will cover the many aspects of the LOFAR system from the operation of the basic station hardware to the software pipelines and science products they produce. Lectures and tutorials will be presented by members of the LOFAR project team as well as staff from the many institutions involved in the collaboration. Hands-on sessions will be provided to give attendees an opportunity to gain experience with real LOFAR data.

Presentations will be given at a level appropriate for someone new to the LOFAR system. Familiarity with the concepts of radio interferometry and standard data processing software such as CASA, AIPS, or Myriad will be useful but not required. Minimum requirements should include familiarity with scripting languages and in particular Python. Parallel sessions for more expert students are also planned.

Attendance will be limited to 35 people, so initial preference will be given to registrations from members of the LOFAR Key Science Project teams and other international partners contributing to the commissioning effort. Registrations from other applicants will be supported on a best-effort basis.

Tuesday, 10. Feb

Wednesday, 11. Feb

Thursday, 12. Feb

Logging into the cluster nodes requires passing through at least one – often two – front-end node. So in order to access the cluster nodes from a machine located at ASTRON you will have to go through the following chain of connections:

ASTRON  -> listfen (129.125.99.50)  -> lioff{021}-{029}

While this is the default sequence to connect to the reserved cluster nodes, the intermediate step going through the front-end-node (lioffen) can be spared.

Here is the procedure for connecting to the cluster nodes set aside for the data processing school:

  1. @ ASTRON :
    ssh -X <login>@129.125.99.50

    This will establish a connection to the front-end node (listfen)

  2. @ listfen :
    ssh -X lioff021

    This will establish a connection to one of the I/O cluster nodes set aside for actual processing.

To initialize your environment for a package (e.g. AIPS):

source /app/scripts/do<package> 

For an overview of available initialization scripts:

ls /app/scripts

Directories & file locations

The workshop will mainly utilize some of the LOFAR offline cluster nodes to run the exercises. These machines are named “lioff0XX” where X goes from 21-29. To reach these machines, you have to first login to listfen. So ssh into listfen and then ssh into one of the working nodes.

On these machines, there is an area set aside named “/data/lofarschool” where all the exercises will be stored. Within that directory there will be two sub-directories, one called “users” which the students will use as working areas and one called “data” which will contain subdirectories for each of the exercises:

/data/lofarschool
      |-- data
      |   |-- Exercise_01
      |   |-- Exercise_02
      |   |-- Exercise_03
      |   `
      `-- users

Each directory subdirectory will contain all the inputs, expected outputs, scripts, logfiles, instructions, etc associated with a given exercise, i.e. :

lioff023:/data/lofarschool/data/Exercise_01> ll
total 8
-rw-r--r-- 1 wise wise    0 2009-02-04 19:26 example_01.log
-rw-r--r-- 1 wise wise    0 2009-02-04 19:26 example_01.py
drwxr-xr-x 2 wise wise 4096 2009-02-04 19:25 inputs/
-rw-r--r-- 1 wise wise    0 2009-02-04 19:27 instructions.txt
drwxr-xr-x 2 wise wise 4096 2009-02-04 19:25 outputs/
-rw-r--r-- 1 wise wise    0 2009-02-04 19:26 README
lioff023:/data/lofarschool/data/Exercise_01>

Preparation

Everything in “/data/lofarschool/data” will be write protected. If users want to modify any of the input files or scripts, they can copy them to them working areas:

  • /data/lofarschool/users and there create a new directory as your working area
    cd /data/lofarschool/users
    mkdir myname
    cd myname
  • Create a working copy of the exercise within your directory, so e.g.
    cp -r /data/lofarschool/data/Exercise-50 .

Exercise instructions

  1. Exercise 01A – Station Data
  2. Exercise 02B – Station Data
  3. Exercise 05 – Flagging, Compression along Freq, Time with DPPP
  4. Exercise 08 – Calibration with BBS
  5. Exercise 10 – AIPS and ParselTongue
  6. Exercise 45 – Visualizing the LOFAR LBA Beam
  7. Exercise 50 – Working with the Data Access Layer
  8. Exercise 51 – Working with the Data Access Layer
  9. Exercise 52 – Working with the Data Access Layer
  10. Exercise 53 – Working with the Data Access Layer
  11. Exercise 60 – Inspecting, flagging and imaging with CASA
  12. Exercise 61 – Inspecting, flagging and imaging with CASA
  13. Exercise 62 – Inspecting, flagging and imaging with CASA
  14. Exercise 63 – Inspecting and plotting visibility data with pyrap
  15. Exercise 64 – Image manipulation with pyrap
  16. Exercise PSR 01 – Pulsars/Beam-formed Data: Inspecting Timeries and Power Spectra
  17. Exercise PSR 02 – Pulsars/Beam-formed Data: Folding and Dedispersing to Detect a Pulsar
  18. Exercise PSR 03 – Pulsars/Beam-formed Data: Creating and Examining a Dynamic Spectrum
  19. Exercise CR0 – Setup, building of executables
  20. Exercise CR1 – DataReader
  21. Exercise CR2 – Dynamic spectrum from LOFAR data
  22. Exercise CR3 – LOPES CR-pipeline
  • Last modified: 2017-03-08 15:27
  • (external edit)