~~NOTOC~~

===== Workshops :: LOFAR Data Processing School (10.-12. Feb 2009, ASTRON, Dwingeloo) =====

  - [[#Schedule]]
  - [[#Presentations]]
  - [[#Connecting to the cluster nodes]]
  - [[#Hands-on exercises]]
    * [[#Directories & file locations]]
    * [[#Preparation]]
    * [[#Exercise instructions]]

==== Who should attend? ====

The aim of this workshop is to introduce new members of the collaboration, who will be involved in the commissioning effort, to LOFAR. Students, postdocs, and staff are all encouraged to apply. The school will cover the many aspects of the LOFAR system from the operation of the basic station hardware to the software pipelines and science products they produce. Lectures and tutorials will be presented by members of the LOFAR project team as well as staff from the many institutions involved in the collaboration. Hands-on sessions will be provided to give attendees an opportunity to gain experience with real LOFAR data.

Presentations will be given at a level appropriate for someone new to the LOFAR system. Familiarity with the concepts of radio interferometry and standard data processing software such as CASA, AIPS, or Myriad will be useful but not required. Minimum requirements should include familiarity with scripting languages and in particular Python. Parallel sessions for more expert students are also planned.

Attendance will be limited to 35 people, so initial preference will be given to registrations from members of the LOFAR Key Science Project teams and other international partners contributing to the commissioning effort. Registrations from other applicants will be supported on a best-effort basis.

==== Schedule ====

{{.2009-02_processing_school:schedule.png?600}}

==== Presentations ====

__Tuesday, 10. Feb__

  * A. Gunst: {{.2009-02_processing_school:gunst_-_lofar_system_overview.pdf|System overview}}
  * M. Brentjens: {{.2009-02_processing_school:brentjens_-_station_data.pdf|Station data}}
  * R. Nijboer: {{.2009-02_processing_school:Nijboer - Imaging Pipeline.pdf|Imaging pipeline}}
  * V.N. Pandey: {{.2009-02_processing_school:Pandey - Data preparation.pdf|Data preparation}}
  * M. Wise: {{.2009-02_processing_school:Wise - Data Products.pdf|LOFAR Data Products}}
  * J. Swinbank: {{.2009-02_processing_school:Swinbank - Software Tools.pdf|Software  Tools I (Python, data access, plotting, etc)}}
  * G. van Diepen: {{.2009-02_processing_school:Diepen - Data Handling Tools.pdf|Data Handling Tools (pyrap)}}

__Wednesday, 11. Feb__

  * G. de Bryun: {{.2009-02_processing_school:de_bruyn_-_calibration.pdf|Calibration}}
  * B. Stappers: {{.2009-02_processing_school:stappers_-_pulsar_pipeline.pdf|Pulsar pipeline}}
  * J. Hessels: {{.2009-02_processing_school:hessels_-_pulsar_data.pdf|Pulsar data}}
  * O. Smirnov: {{.2009-02_processing_school:smirnov_-_meqtrees_practical_session.pdf|MeqTrees Practical Session}}

__Thursday, 12. Feb__

  * A. Horneffer: {{.2009-02_processing_school:horneffer - CR pipelines.pdf|CR pipelines}}
  * L. Bähren: {{.2009-02_processing_school:baehren_-_cr_data.pdf|CR Data}}
  * H. Holties: {{.2009-02_processing_school:holties_-_lofar_archive.pdf|LOFAR Archive}}

==== Connecting to the cluster nodes ====

Logging into the cluster nodes requires passing through at least one -- often two -- front-end node. So in order to access the cluster nodes from a machine located at ASTRON you will have to go through the following chain of connections:
<code>
ASTRON  -> listfen (129.125.99.50)  -> lioff{021}-{029}
</code>
While this is the default sequence to connect to the reserved cluster nodes, the intermediate step going through the front-end-node (//lioffen//) can be spared.

Here is the procedure for connecting to the cluster nodes set aside for the data processing school:
  - @ ASTRON : <code bash>
ssh -X <login>@129.125.99.50
</code> This will establish a connection to the front-end node (**listfen**)
  - @ listfen : <code bash>
ssh -X lioff021
</code> This will establish a connection to one of the I/O cluster nodes set aside for actual processing.

To initialize your environment for a package (e.g. AIPS):
<code csh>
source /app/scripts/do<package> 
</code>
For an overview of available initialization scripts:
<code csh>
ls /app/scripts
</code>

==== Hands-on exercises ====

=== Directories & file locations ===

The workshop will mainly utilize some of the LOFAR offline cluster nodes to run the exercises. These machines are named "lioff0XX" where X goes from 21-29. To reach these machines, you have to first login to listfen. So ssh into listfen and then ssh into one of the working nodes.

On these machines, there is an area set aside named "/data/lofarschool" where all the exercises will be stored. Within that directory there will be two sub-directories, one called "users" which the students will use as working areas and one called "data" which will contain subdirectories for each of the exercises:

<code>
/data/lofarschool
      |-- data
      |   |-- Exercise_01
      |   |-- Exercise_02
      |   |-- Exercise_03
      |   `
      `-- users
</code>

Each directory subdirectory will contain all the inputs, expected outputs, scripts, logfiles, instructions, etc associated with a  given exercise, i.e. :

<code>
lioff023:/data/lofarschool/data/Exercise_01> ll
total 8
-rw-r--r-- 1 wise wise    0 2009-02-04 19:26 example_01.log
-rw-r--r-- 1 wise wise    0 2009-02-04 19:26 example_01.py
drwxr-xr-x 2 wise wise 4096 2009-02-04 19:25 inputs/
-rw-r--r-- 1 wise wise    0 2009-02-04 19:27 instructions.txt
drwxr-xr-x 2 wise wise 4096 2009-02-04 19:25 outputs/
-rw-r--r-- 1 wise wise    0 2009-02-04 19:26 README
lioff023:/data/lofarschool/data/Exercise_01>
</code>

=== Preparation ===

Everything in "/data/lofarschool/data" will be write protected. If users want to modify any of the input files or scripts, they can copy them to them working areas: 
  * ''/data/lofarschool/users'' and there create a new directory as your working area <code bash>
cd /data/lofarschool/users
mkdir myname
cd myname
</code>
  * Create a working copy of the exercise within your directory, so e.g. <code bash>
cp -r /data/lofarschool/data/Exercise-50 .
</code>

=== Exercise instructions ===

  - [[.2009-02_processing_school:Exercise 01A]] -- Station Data
  - [[.2009-02_processing_school:Exercise 02B]] -- Station Data
  - [[.2009-02_processing_school:Exercise 05]] -- Flagging, Compression along Freq, Time with DPPP
  - [[.2009-02_processing_school:Exercise 08]] -- Calibration with BBS
  - [[.2009-02_processing_school:Exercise 10]] -- AIPS and ParselTongue
  - [[.2009-02_processing_school:Exercise 45]] -- Visualizing the LOFAR LBA Beam
  - [[.2009-02_processing_school:Exercise 50]] -- Working with the Data Access Layer
  - [[.2009-02_processing_school:Exercise 51]] -- Working with the Data Access Layer
  - [[.2009-02_processing_school:Exercise 52]] -- Working with the Data Access Layer
  - [[.2009-02_processing_school:Exercise 53]] -- Working with the Data Access Layer
  - [[.2009-02_processing_school:Exercise 60]] -- Inspecting, flagging and imaging with CASA
  - [[.2009-02_processing_school:Exercise 61]] -- Inspecting, flagging and imaging with CASA
  - [[.2009-02_processing_school:Exercise 62]] -- Inspecting, flagging and imaging with CASA
  - [[.2009-02_processing_school:Exercise 63]] -- Inspecting and plotting visibility data with pyrap
  - [[.2009-02_processing_school:Exercise 64]] -- Image manipulation with pyrap
  - [[.2009-02_processing_school:Exercise PSR 01]] -- Pulsars/Beam-formed Data: Inspecting Timeries and Power Spectra
  - [[.2009-02_processing_school:Exercise PSR 02]] -- Pulsars/Beam-formed Data: Folding and Dedispersing to Detect a Pulsar
  - [[.2009-02_processing_school:Exercise PSR 03]] -- Pulsars/Beam-formed Data: Creating and Examining a Dynamic Spectrum
  - [[.2009-02_processing_school:Exercise CR0]] -- Setup, building of executables
  - [[.2009-02_processing_school:Exercise CR1]] -- DataReader 
  - [[.2009-02_processing_school:Exercise CR2]] -- Dynamic spectrum from LOFAR data
  - [[.2009-02_processing_school:Exercise CR3]] -- LOPES CR-pipeline