public:grid:srmcp_pipes

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
public:grid:srmcp_pipes [2011-02-01 20:33] – Joeri van Leeuwenpublic:grid:srmcp_pipes [2017-03-08 15:27] (current) – external edit 127.0.0.1
Line 8: Line 8:
 For speed, we would like to do this tar on the machine that holds the original data (an lse node), and send this tar file through a lexar to the grid light path, without touching the (relatively slow) lexar disk.  For speed, we would like to do this tar on the machine that holds the original data (an lse node), and send this tar file through a lexar to the grid light path, without touching the (relatively slow) lexar disk. 
  
-To write to the Grid, use globus-url-copy, not srmcp which fails as it tries a seek() on the pipe, in a setup like this (after all Grid proxies etc have been set):+To write to the Grid, use globus-url-copy, not srmcp which fails as it tries a seek() on the pipe, in a setup like this (after all Grid proxies etc have been set). The below would run on data (lse) node:
  
   # To store to the Grid one need to use globus-url-copy   # To store to the Grid one need to use globus-url-copy
Line 14: Line 14:
   set GSIFTPCP="globus-url-copy -vb"   set GSIFTPCP="globus-url-copy -vb"
   set SRMSSH="ssh lexar001 -c arcfour"   set SRMSSH="ssh lexar001 -c arcfour"
 +  
   # Our data directory on the Grid:   # Our data directory on the Grid:
   set GSIFTPDIR=gsiftp://gridftp.grid.sara.nl//pnfs/grid.sara.nl/data/lofar/pulsar/   set GSIFTPDIR=gsiftp://gridftp.grid.sara.nl//pnfs/grid.sara.nl/data/lofar/pulsar/
 +  
   # In this case, read OBSID from the script input   # In this case, read OBSID from the script input
   set OBSID=$1   set OBSID=$1
   set FILE=/tmp/psr/${OBSID}.raw.tar   set FILE=/tmp/psr/${OBSID}.raw.tar
- +   
-  # Make named pipe, wait 2 sec, then start transfer +  # Tar data dir (named $OBSID), pipe to ssh and make named pipe on lexar 
-  tar -cp ${ID} | ${SRMSSH} "(mkfifo $FILE; cat - > $FILE)" &+  tar -cp ${OBSID} | ${SRMSSH} "(mkfifo $FILE; cat - > $FILE)" & 
 +  # wait 2 sec, then start transfer
   sleep 2   sleep 2
   ${SRMSSH} "${GSIFTPCP} file://$FILE $GSIFTPDIR/tape/; rm $FILE"   ${SRMSSH} "${GSIFTPCP} file://$FILE $GSIFTPDIR/tape/; rm $FILE"
  • Last modified: 2011-02-01 20:33
  • by Joeri van Leeuwen