How to build a beowulf cluster for astronomical image and data processing?

A detailed document on how to go about building your own Beowulf class computer cluster using commercial off the self hardware and open source software. It is targeted towards astronomical image and data processing and uses standard Message Passing Interface (MPI) but can be used for any computationally intensive workflow. Click on the following link to access the PDF document - Beowulf Cluster.

Installing ESO Scisoft on Ubuntu 14.04 64-bit

European Southern Observatory's Scisoft is a collection of astronomical software applications. It includes IRAF, PyRAF, MIDAS, CASA, DS9 among other data analysis, image display, and scientific libraries.

The latest version of Scisoft is 7.7. It is a 32 bit package and therefore need some external dependencies to be satisfied before it can be used on 64 bit Ubuntu.

Here are the steps I followed to install it on Ubuntu 14.04 -

1. Downloaded the latest version in tar format from ESO Scisoft website.
2. As root, uncompress the downloaded file :

   cd /
   sudo tar xvfz path-to-scisoft-save-dir/scisoft-7.7.0.tar.gz
   

3. As Scisoft is 32 bit software, we need to install 32 bit external packages to resolve dependency issues. I followed the suggestion on this askubuntu.com post to install 32 bit libraries. Ran the following commands in terminal :

   sudo dpkg --add-architecture i386
   sudo apt-get update
   sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386
   

4. Following suggestion on this post, I installed the dependencies using following commands : 

   sudo apt-get install tcsh:i386 libgfortran3:i386 libreadline5:i386 
   libsdl-image1.2:i386 libsdl-ttf2.0-0:i386 unixodbc:i386
   

5. Downloaded packages libg2c0_3.4.6-8ubuntu2_i386.deb and gcc-3.4-base_3.4.6-8ubuntu2_i386.deb from Ubuntu archive. Install both :

   sudo dpkg -i libg2c0_3.4.6-8ubuntu2_i386.deb
   sudo dpkg -i gcc-3.4-base_3.4.6-8ubuntu2_i386.deb
   

6. I had to install libxft2, libxrandr2, and libxmu6 for xgterm and graphic windows to work in IRAF and PyRAF :

   sudo apt-get install libxft2:i386 libxrandr2:i386 libxmu6:i386
   

7. We are all set to initialize IRAF. For the first run, run following command to set paths and environment variables :

   . /scisoft/bin/Setup.bash
   

   There is a space between dot(.) and first slash(/).
8. Make a new directory and run mkiraf to generate login.cl file. You can customize the login file as per your requirement.
9. To start IRAF, open a xgterm window by running xgterm & command in terminal. Type ecl in xgterm window to start IRAF session. The command should be issued in the same directory where the login.cl file is lying. Otherwise the command will fail.
10. To start PyRAF, just type pyraf command in terminal.
11. Test if DS9 graphic window is able to display image and you can examine the image using imexam.

Installing S2PLOT on Ubuntu/Linux Mint

S2PLOT is a 3D plotting library written in C and developed at Swinburne University of Technology in Australia. The generated plots can be saved in VRML format and embedded in PDF documents. 

While installing S2PLOT version 3.2.1 on Linux Mint 16.0 (32 bit) and Ubuntu 14.04 (64 bit), I needed to make few minor tweaks for it to work. Below are the steps I followed to install it - 

1. Download the latest version of S2PLOT from official S2PLOT website.
2. Uncompress the downloaded file to a directory where you want to install it. I downloaded the file in Downloads directory and installed it in /usr/local using the following commands -

   cd /usr/local
   sudo tar xvjf ~/Downloads/s2plot-3.2.1.tar.bz2

   It will uncompress the contents of the file into s2plot-3.2.1 folder.
3. The main script to build S2PLOT is build.csh in scripts directory. This script in turn executes four other build scripts : build-apps.csh, build-viewer.csh, build-examples.csh, and build-prc-module.csh.
4. Before running build.sh, we need to set some environment variables. Run the following commands in terminal :

   export S2PATH=/usr/local/s2plot-3.2.1
   export S2ARCH=linux-gnu-i386 or linux-gnu-x86_64 (for 64 bit)
   export S2SHARED=yes                   
   export S2FREEGLUT=yes                 
   export S2FREETYPE=yes                 
   export S2PLOT_PRCDRIVER=s2prcwriter     
   

   More details about environment variables in ENVIRONMENT.TXT file.
5. Open file s2plot.csh in scripts directory for editing. Set the value of variable S2X11PATH to point to directory where X11 header and library files are lying. On both Linux Mint and Ubuntu, they are in /usr directory. Therefore change 

   set S2X11PATH=/usr/X11R6
   

   to

   set S2X11PATH=/usr
   

6. Update COMPPATH variable in either linux-gnu-i386.in or linux-gnu-x86_64.in (depending on OS) in scripts folder to point to directory having gcc and g++ binaries. On Linux Mint and Ubuntu, they are in /usr/bin. Therefore change 

   set COMPPATH="/usr/local/gcc/4.4.5/bin"
   

   to

   set COMPPATH="/usr/bin"
   

7. Install following packages from the repository :

   sudo apt-get install libhpdf-2.2.1 libhpdf-dev freeglut3 
   freeglut3-dev libfreetype6 libfreetype6-dev
   

8. Now run the build.csh script to compile and link programs and examples. Build script should be run from /usr/local/s2plot-3.2.1 directory.

   cd /usr/local/s2plot-3.2.1
   ./scripts/build.csh 
   OR
   ./scripts/build.csh >& build.txt  (save messages to a file)
   

9. The build process may fail while linking S2view program. In my case it failed as it was not able to locate libdl and libXpm. They may be installed but not in system path. In file build-viewer.csh in scripts directory, change following line :

   $S2CLINKER -o s2view geomviewer.o -L. ${S2LINKS} ${MLLINKS} 
   ${SWLINKS} ${GLLINKS} -L${S2X11PATH}/lib${S2LBITS} 
   ${S2FORMSLINK} -lXpm -lX11 ${IMATH} -lm ${XLINKPATH}
   

   to

   $S2CLINKER -o s2view geomviewer.o -L. ${S2LINKS} ${MLLINKS} 
   ${SWLINKS} ${GLLINKS} -L${S2X11PATH}/lib${S2LBITS} 
   ${S2FORMSLINK} -L/usr/lib/x86_64-linux-gnu -ldl -lXpm -lX11 
   ${IMATH} -lm ${XLINKPATH}
   
   OR
   
   $S2CLINKER -o s2view geomviewer.o -L. ${S2LINKS} ${MLLINKS} 
   ${SWLINKS} ${GLLINKS} -L${S2X11PATH}/lib${S2LBITS} 
   ${S2FORMSLINK} -L/usr/lib/i386-linux-gnu -ldl -lXpm -lX11 
   ${IMATH} -lm ${XLINKPATH}
   

10. The linker may still give error that it is not able to find libXpm on your machine. It may be the case that libXpm.so.4 and not libXpm.so is present. We can create a symbolic link libXpm.so to point to libXpm.so.4 to circumvent this error. Run the following command in terminal :

    sudo ln -s /usr/lib/x86_64-linux-gnu/libXpm.so.4 
    /usr/lib/x86_64-linux-gnu/libXpm.so
    
    OR
    
    sudo ln -s /usr/lib/i386-linux-gnu/libXpm.so.4 
    /usr/lib/i386-linux-gnu/libXpm.so
    

11. Re-run build.sh script to create executables. Executables are created in either /usr/local/s2plot-3.2.1/linux-gnu-i386 or /usr/local/s2plot-3.2.1/linux-gnu-x86_64 (on 64 bit linux) directory.
12. To test if everything works fine, try some of the examples. Run following commands in terminal from /usr/local/s2plot-3.2.1 directory :

    ./linux-gnu-i386/s2view      or ./linux-gnu-x86_64/s2view

    ./linux-gnu-i386/ns2thwcube  or ./linux-gnu-x86_64/ns2thwcube

    ./linux-gnu-i386/ns2vplanett or ./linux-gnu-x86_64/ns2vplanett 

    
    
    
13. Check out S2PLOT website for examples and documentation on how to generate various types of 3D plots.
14. Happy plotting!