Differences

This shows you the differences between two versions of the page.

--- public:meetings:2009-02_data_processing_school [2011-10-25 12:13] – created Lars Bahren
+++ public:meetings:2009-02_data_processing_school [2017-03-08 15:27] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
 ~~NOTOC~~
-===== Workshops :: LOFAR Data Processing School (10.--12. Feb 2009, ASTRON, Dwingeloo) =====
+===== Workshops :: LOFAR Data Processing School (10.-12. Feb 2009, ASTRON, Dwingeloo) =====
-  * [[#Schedule]]
+  - [[#Schedule]]
-  * [[#Presentations]]
+  - [[#Presentations]]
-  * [[#Hands-on exercises]]
+  - [[#Connecting to the cluster nodes]]
+  - [[#Hands-on exercises]]
+    * [[#Directories & file locations]]
+    * [[#Preparation]]
+    * [[#Exercise instructions]]
 ==== Who should attend? ====
@@ Line 23: / Line 27: @@
 __Tuesday, 10. Feb__
-  * A. Gunst: {{.2009-02_processing_school:gunst_-_lofar_system_overview.ppt|System overview}} ({{.2009-02_processing_school:gunst_-_lofar_system_overview.pdf|PDF}})
+  * A. Gunst: {{.2009-02_processing_school:gunst_-_lofar_system_overview.pdf|System overview}}
   * M. Brentjens: {{.2009-02_processing_school:brentjens_-_station_data.pdf|Station data}}
-  * R. Nijboer: {{.2009-02_processing_school:SIP_Nijboer.pdf|Imaging pipeline}}
+  * R. Nijboer: {{.2009-02_processing_school:Nijboer - Imaging Pipeline.pdf|Imaging pipeline}}
-  * V.N. Pandey: {{.2009-02_processing_school:pandey_lofar_dppp_lofarschool_2009_feb10_09.pdf|Data preparation}}
+  * V.N. Pandey: {{.2009-02_processing_school:Pandey - Data preparation.pdf|Data preparation}}
-  * M. Wise: {{.2009-02_processing_school:wise_products.pdf|LOFAR Data Products}}
+  * M. Wise: {{.2009-02_processing_school:Wise - Data Products.pdf|LOFAR Data Products}}
-  * J. Swinbank: {{.2009-02_processing_school:swinbank-software_tools_1.pdf|Software  Tools I (Python, data access, plotting, etc)}}
+  * J. Swinbank: {{.2009-02_processing_school:Swinbank - Software Tools.pdf|Software  Tools I (Python, data access, plotting, etc)}}
-  * G. van Diepen: {{.2009-02_processing_school:data-handling-pyrap.pdf|Data Handling Tools (pyrap)}}
+  * G. van Diepen: {{.2009-02_processing_school:Diepen - Data Handling Tools.pdf|Data Handling Tools (pyrap)}}
 __Wednesday, 11. Feb__
@@ Line 40: / Line 44: @@
 __Thursday, 12. Feb__
-  * A. Horneffer: {{.2009-02_processing_school:horneffer-lofar-data-school-2009_02_12.pdf|CR pipelines}}
+  * A. Horneffer: {{.2009-02_processing_school:horneffer - CR pipelines.pdf|CR pipelines}}
   * L. Bähren: {{.2009-02_processing_school:baehren_-_cr_data.pdf|CR Data}}
   * H. Holties: {{.2009-02_processing_school:holties_-_lofar_archive.pdf|LOFAR Archive}}
+==== Connecting to the cluster nodes ====
+Logging into the cluster nodes requires passing through at least one -- often two -- front-end node. So in order to access the cluster nodes from a machine located at ASTRON you will have to go through the following chain of connections:
+<code>
+ASTRON  -> listfen (129.125.99.50)  -> lioff{021}-{029}
+</code>
+While this is the default sequence to connect to the reserved cluster nodes, the intermediate step going through the front-end-node (//lioffen//) can be spared.
+Here is the procedure for connecting to the cluster nodes set aside for the data processing school:
+  - @ ASTRON : <code bash>
+ssh -X <login>@129.125.99.50
+</code> This will establish a connection to the front-end node (**listfen**)
+  - @ listfen : <code bash>
+ssh -X lioff021
+</code> This will establish a connection to one of the I/O cluster nodes set aside for actual processing.
+To initialize your environment for a package (e.g. AIPS):
+<code csh>
+source /app/scripts/do<package>
+</code>
+For an overview of available initialization scripts:
+<code csh>
+ls /app/scripts
+</code>
 ==== Hands-on exercises ====
-== Connecting to the cluster nodes ==
+=== Directories & file locations ===
+The workshop will mainly utilize some of the LOFAR offline cluster nodes to run the exercises. These machines are named "lioff0XX" where X goes from 21-29. To reach these machines, you have to first login to listfen. So ssh into listfen and then ssh into one of the working nodes.
+On these machines, there is an area set aside named "/data/lofarschool" where all the exercises will be stored. Within that directory there will be two sub-directories, one called "users" which the students will use as working areas and one called "data" which will contain subdirectories for each of the exercises:
+<code>
+/data/lofarschool
+      |-- data
+      |   |-- Exercise_01
+      |   |-- Exercise_02
+      |   |-- Exercise_03
+      |   `
+      `-- users
+</code>
+Each directory subdirectory will contain all the inputs, expected outputs, scripts, logfiles, instructions, etc associated with a  given exercise, i.e. :
+<code>
+lioff023:/data/lofarschool/data/Exercise_01> ll
+total 8
+-rw-r--r-- 1 wise wise    0 2009-02-04 19:26 example_01.log
+-rw-r--r-- 1 wise wise    0 2009-02-04 19:26 example_01.py
+drwxr-xr-x 2 wise wise 4096 2009-02-04 19:25 inputs/
+-rw-r--r-- 1 wise wise    0 2009-02-04 19:27 instructions.txt
+drwxr-xr-x 2 wise wise 4096 2009-02-04 19:25 outputs/
+-rw-r--r-- 1 wise wise    0 2009-02-04 19:26 README
+lioff023:/data/lofarschool/data/Exercise_01>
+</code>
+=== Preparation ===
+Everything in "/data/lofarschool/data" will be write protected. If users want to modify any of the input files or scripts, they can copy them to them working areas:
+  * ''/data/lofarschool/users'' and there create a new directory as your working area <code bash>
+cd /data/lofarschool/users
+mkdir myname
+cd myname
+</code>
+  * Create a working copy of the exercise within your directory, so e.g. <code bash>
+cp -r /data/lofarschool/data/Exercise-50 .
+</code>
+=== Exercise instructions ===
+  - [[.2009-02_processing_school:Exercise 01A]] -- Station Data
+  - [[.2009-02_processing_school:Exercise 02B]] -- Station Data
+  - [[.2009-02_processing_school:Exercise 05]] -- Flagging, Compression along Freq, Time with DPPP
+  - [[.2009-02_processing_school:Exercise 08]] -- Calibration with BBS
+  - [[.2009-02_processing_school:Exercise 10]] -- AIPS and ParselTongue
+  - [[.2009-02_processing_school:Exercise 45]] -- Visualizing the LOFAR LBA Beam
+  - [[.2009-02_processing_school:Exercise 50]] -- Working with the Data Access Layer
+  - [[.2009-02_processing_school:Exercise 51]] -- Working with the Data Access Layer
+  - [[.2009-02_processing_school:Exercise 52]] -- Working with the Data Access Layer
+  - [[.2009-02_processing_school:Exercise 53]] -- Working with the Data Access Layer
+  - [[.2009-02_processing_school:Exercise 60]] -- Inspecting, flagging and imaging with CASA
+  - [[.2009-02_processing_school:Exercise 61]] -- Inspecting, flagging and imaging with CASA
+  - [[.2009-02_processing_school:Exercise 62]] -- Inspecting, flagging and imaging with CASA
+  - [[.2009-02_processing_school:Exercise 63]] -- Inspecting and plotting visibility data with pyrap
+  - [[.2009-02_processing_school:Exercise 64]] -- Image manipulation with pyrap
+  - [[.2009-02_processing_school:Exercise PSR 01]] -- Pulsars/Beam-formed Data: Inspecting Timeries and Power Spectra
+  - [[.2009-02_processing_school:Exercise PSR 02]] -- Pulsars/Beam-formed Data: Folding and Dedispersing to Detect a Pulsar
+  - [[.2009-02_processing_school:Exercise PSR 03]] -- Pulsars/Beam-formed Data: Creating and Examining a Dynamic Spectrum
+  - [[.2009-02_processing_school:Exercise CR0]] -- Setup, building of executables
+  - [[.2009-02_processing_school:Exercise CR1]] -- DataReader
+  - [[.2009-02_processing_school:Exercise CR2]] -- Dynamic spectrum from LOFAR data
+  - [[.2009-02_processing_school:Exercise CR3]] -- LOPES CR-pipeline