public:docker

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
public:docker [2017-02-08 15:36] – [LOFAR Pipeline Software] Jan David Molpublic:docker [2018-06-07 06:52] (current) – [Development (extending the software)] Jan David Mol
Line 3: Line 3:
 We offer the LOFAR software as Docker images, allowing anyone to run our software using the same configuration, without having to build it. Our images can be browsed at https://hub.docker.com/r/lofar/. We offer the LOFAR software as Docker images, allowing anyone to run our software using the same configuration, without having to build it. Our images can be browsed at https://hub.docker.com/r/lofar/.
  
-====== LOFAR Pipeline Software ======+====== LOFAR Interferometry Post-Processing Software ======
  
 To run the lofar pipeline software, you need to: To run the lofar pipeline software, you need to:
  
-  * Install and configure [[https://docs.docker.com/engine/getstarted/|Docker]] on your computer (Mac/Linux/Windows) +  - Install and configure [[https://docs.docker.com/engine/getstarted/|Docker]] on your computer (Mac/Linux/Windows) 
-  * Run ''docker run -it --rm -u $UID -e USER -e HOME -v $HOME:$HOME lofar/lofar-pipeline:LOFAR-Release-2_19'' to download and run the LOFAR pipeline software version 2.19. To list all available versions, go to https://hub.docker.com/r/lofar/lofar-pipeline/tags/. +  - Download and start the LOFAR image by running: <file> 
-  * You now have the LOFAR software available at your finger tips, and your home directory available. You can run for example: <file>+docker run -it --rm -u $UID -e USER -e HOME -v $HOME:$HOME lofar/lofar-pipeline:LOFAR-Release-2_19 
 +</file> To list all available versions, go to https://hub.docker.com/r/lofar/lofar-pipeline/tags/. 
 +  - You now have the LOFAR software available at your finger tips, and your home directory available. You can run for example: <file>
 you@3617438dfc63:/$ NDPPP   you@3617438dfc63:/$ NDPPP  
 Usage: DPPP [-v] [parsetfile] [parsetkeys...] Usage: DPPP [-v] [parsetfile] [parsetkeys...]
   parsetfile: a file containing one parset key=value pair per line   parsetfile: a file containing one parset key=value pair per line
   parsetkeys: any number of parset key=value pairs, e.g. msin=my.MS   parsetkeys: any number of parset key=value pairs, e.g. msin=my.MS
-</file> or <file>+</file> or: <file> 
 +you@3617438dfc63:/$ long_baseline_pipeline.py  
 +/opt/lofar/lib/python2.7/site-packages/lofarpipe/support/utilities.pyc : Using default subprocess module! 
 +Usage: /opt/lofar/bin/long_baseline_pipeline.py <parset-file>  [options] 
 +Results: 
 +</file> or one of our dependencies: <file>
 you@3617438dfc63:/$ /opt/aoflagger/bin/aoflagger you@3617438dfc63:/$ /opt/aoflagger/bin/aoflagger
 AOFlagger 2.8.0 (2016-06-21) command line application AOFlagger 2.8.0 (2016-06-21) command line application
Line 26: Line 33:
 </file> </file>
  
 +====== LOFAR Pulsar Post-Processing Software ======
  
-Once you have the above up and running, you will need some data to work on, and likely a parset with configuration settings. If you've put both in your home directory, they're immediately available when running the above command. If not, you can use additional [[https://docs.docker.com/engine/reference/run/#/volume-shared-filesystems|''-v'']] parameters to make more directories available in your Docker container.+To run the lofar pipeline software, you need to: 
 +  - Install and configure [[https://docs.docker.com/engine/getstarted/|Docker]] on your computer (Mac/Linux/Windows) 
 +  - Download and start the LOFAR image by running: <file> 
 +docker run -it --rm -u $UID -e USER -e HOME -v $HOME:$HOME lofar/lofar-pulp:LOFAR-Release-2_19 
 +</file> To list all available versions, go to https://hub.docker.com/r/lofar/lofar-pulp/tags/. 
 +  - You now have the LOFAR pulp software available at your finger tips, and your home directory available. You can run for example: <file> 
 +you@3617438dfc63:/$ pulp.py  
 +Usage: pulp.py <--id ObsID> [-h|--help] [OPTIONS] 
 +</file> 
 + 
 +====== Q&A ====== 
 + 
 +===== Installation ===== 
 + 
 +=== Q: Where can I find a list of available LOFAR software versions? === 
 + 
 +A: For ''lofar-pipeline'', surf to [[https://hub.docker.com/r/lofar/lofar-pipeline/tags/]]. For ''lofar-pulp'', surf to [[https://hub.docker.com/r/lofar/lofar-pulp/tags/]]. 
 + 
 +=== Q: What is included in the image? === 
 + 
 +We put the following in our ''lofar-pipeline'' image: 
 + 
 +  * LOFAR Pipeline Framework & Recipes 
 +  * NDPPP 
 +  * AWImager 
 +  * AOFlagger 
 +  * DAL2 
 +  * Casacore + casarest + python-casacore 
 + 
 +===== Running the software ===== 
 + 
 +=== Q: I can't see my data or parset? === 
 + 
 +Once you have the Docker image up and running, you will need some data to work on, and likely a parset with configuration settings. If you've put both in your home directory, they're immediately available when running with the commands described on this page. If not, you can use additional [[https://docs.docker.com/engine/reference/run/#/volume-shared-filesystems|''-v'']] parameters to make more directories available in your Docker container. 
 + 
 +=== Q: I get "Illegal instruction" when running some of the software? === 
 + 
 +A: The LOFAR software is compiled for a 2015-era processor (Intel Xeon E5-2603v3 to be exact) for performance reasons, and uses CPU instructions not available on older machines. Please run our images on a newer machine, or [[#development_extending_the_software|rebuild the Docker images from scratch]]. 
 + 
 +===== Development (extending the software) ===== 
 + 
 +=== Q: How do I obtain your Dockerfiles? === 
 + 
 +A: By running: 
 + 
 +<file> 
 +svn co -N https://svn.astron.nl/LOFAR/branches/LOFAR-Release-2_19/ 
 +cd LOFAR-Release-2_19 
 +svn up CMake 
 +mkdir -p build/gnu_opt && cd build/gnu_opt 
 + 
 +cmake ../.. -DBUILD_PACKAGES=Docker -DUSE_LOG4CPLUS=OFF -DUSE_CASACORE=OFF && make -j 16 install 
 +</file> 
 + 
 +which will put the Dockerfiles in subdirectories in ''LOFAR-Release-2_19/build/gnu_opt/installed/share/docker''. 
 + 
 +=== Q: How do I build your Docker images from scratch? === 
 + 
 +A: By obtaining the Dockerfiles (see above), and running: 
 + 
 +<file> 
 +# load LOFAR environment variables, used to determine tags etc 
 +source installed/lofarinit.sh 
 +# build all LOFAR images, in the order required by their interdependencies 
 +docker-build-all.sh 
 +</file> 
 + 
 +=== Q: How do I add files/modifications permanently? === 
 + 
 +A: The best way to add stuff to our images is by creating a new image based on ours. To do so, create a directory containing the files you want to add. Add to this directory a ''Dockerfile'' that describes your extension, for example: 
 + 
 +<file> 
 +FROM lofar-pipeline:LOFAR-Release-2_19 
 + 
 +COPY my_extension /opt/my_extension 
 +</file> 
 + 
 +Finally, you run <file>docker build -t myimage:mytag directory/</file> where ''directory'' is the directory containing the Dockerfile, and ''myimage:mytag'' is the name and tag of your new image (instead of ''lofar-pipeline:LOFAR-Release-2_19''). See also the [[https://docs.docker.com/engine/reference/builder/|Docker manual]] on how to make and use Dockerfiles. 
 + 
 +=== Q: How do I add initialisation stuff? (PYTHONPATH, etc) === 
 + 
 +A: Our Docker images execute ''/opt/bashrc'' upon entry, which in turn reads and sources all the files in ''/opt/bashrc.d/'' in order. You can thus add bash files to this directory that contain your initialisation statements, f.e. a file called ''20-myextension'' containing ''export PYTHONPATH=$PYTHONPATH:/opt/my_extension''. 
 + 
 +Note: if you add eggs to the image, you need to put the egg //filename// to the PYTHONPATH. It is not enough to add the directory containing the egg.
  • Last modified: 2017-02-08 15:36
  • by Jan David Mol