Differences

This shows you the differences between two versions of the page.

--- public:grid:srmcp_pipes [2011-02-01 19:49] – created Adriaan Renting
+++ public:grid:srmcp_pipes [2017-03-08 15:27] (current) – external edit 127.0.0.1
@@ Line 2: / Line 2: @@
 ===== Problem =====
+Entire subdirectories cannot be recursively copied over to the grid with the tools currently installed on the lexar (2011-1-1).
 ===== Solution =====
-The solution found needs the following commands:
+The solution is to tar entire subdirectory trees, and move these to the archive.
-  writing text like this with two spaces makes it fixed width
+For speed, we would like to do this tar on the machine that holds the original data (an lse node), and send this tar file through a lexar to the grid light path, without touching the (relatively slow) lexar disk.
+To write to the Grid, use globus-url-copy, not srmcp which fails as it tries a seek() on the pipe, in a setup like this (after all Grid proxies etc have been set). The below would run on data (lse) node:
+  # To store to the Grid one need to use globus-url-copy
+  # which does not do a seek() in the stream
+  set GSIFTPCP="globus-url-copy -vb"
+  set SRMSSH="ssh lexar001 -c arcfour"
+  # Our data directory on the Grid:
+  set GSIFTPDIR=gsiftp://gridftp.grid.sara.nl//pnfs/grid.sara.nl/data/lofar/pulsar/
+  # In this case, read OBSID from the script input
+  set OBSID=$1
+  set FILE=/tmp/psr/${OBSID}.raw.tar
+  # Tar data dir (named $OBSID), pipe to ssh and make named pipe on lexar
+  tar -cp ${OBSID} | ${SRMSSH} "(mkfifo $FILE; cat - > $FILE)" &
+  # wait 2 sec, then start transfer
+  sleep 2
+  ${SRMSSH} "${GSIFTPCP} file://$FILE $GSIFTPDIR/tape/; rm $FILE"
+To read back from the Grid one can use the same as below but inverted, or use srmcp: output to the named pipe and untar.