How to use srmcp with named pipes

Entire subdirectories cannot be recursively copied over to the grid with the tools currently installed on the lexar (2011-1-1).

The solution is to tar entire subdirectory trees, and move these to the archive. For speed, we would like to do this tar on the machine that holds the original data (an lse node), and send this tar file through a lexar to the grid light path, without touching the (relatively slow) lexar disk.

To write to the Grid, use globus-url-copy, not srmcp which fails as it tries a seek() on the pipe, in a setup like this (after all Grid proxies etc have been set). The below would run on data (lse) node:

# To store to the Grid one need to use globus-url-copy
# which does not do a seek() in the stream
set GSIFTPCP="globus-url-copy -vb"
set SRMSSH="ssh lexar001 -c arcfour"

# Our data directory on the Grid:
set GSIFTPDIR=gsiftp://gridftp.grid.sara.nl//pnfs/grid.sara.nl/data/lofar/pulsar/

# In this case, read OBSID from the script input
set OBSID=$1
set FILE=/tmp/psr/${OBSID}.raw.tar

# Tar data dir (named $OBSID), pipe to ssh and make named pipe on lexar
tar -cp ${OBSID} | ${SRMSSH} "(mkfifo $FILE; cat - > $FILE)" &
# wait 2 sec, then start transfer
sleep 2
${SRMSSH} "${GSIFTPCP} file://$FILE $GSIFTPDIR/tape/; rm $FILE"

To read back from the Grid one can use the same as below but inverted, or use srmcp: output to the named pipe and untar.