Adenine User Guide

Plese check the Adenine Administrator Guide if you are looking for information on how to configure or repair Adenine (only accessible from clemson.edu domain).

System Information

Adenine.parl.clemson.edu is a 66 node linux cluster (one head node and 65 slaves), located in 352 EIB. Each node is a dual 1GHz Pentium III with 1 Gbyte of RAM. The nodes are connected by channel bonded fast ethernet, providing a theoretical bandwith of 200 Mbit/sec between each node. Each node has two IDE hard disk drives. The current kernel version can be determined by running "uname -a" on the head node. The operating system is based on Debian.

USAGE POLICY

All jobs should be submitted to the batch scheduler for execution.

Users are not permitted to run mpirun, rsh, rlogin directly; the only way to access compute nodes is through the scheduler.

There are no regular backups of Adenine at this time. It is the reponsibility of the user and only the user to maintain backups of any data stored on Adenine.

Preparing your environment (new users)

Your account should already be configured to allow you to access the parallel programming tools on Adenine. You may verify this with the following commands:
  • "which mpicc": This command should return the full path to mpicc, which will most likely look something like this: "/opt/mpich2/bin/mpicc", though it may differ slightly.
If the above commands does not produce the expected results, then it is most likely because you are using an unusual shell, or else are using login scripts that clear certain key environment variables.

Compiling and submitting MPI jobs

This is best explained with a tutorial. Please follow the steps below to run an example program, then modify the steps as needed to reflect the needs of your own job.

Tutorial

  1. Create a directory to store the tutorial files: "mkdir tutorial; cd tutorial".
  2. Download this example program. It is a trivial MPI program that computes the value of pi.
  3. Compile the example program: "mpicc cpi.c -o cpi".
  4. Download the submission script cpi_job.pbs. It is a typical job submission script for the batch scheduler.
  5. Edit the job submission script to include the correct executable path.
  6. Submit the job to the batch scheduler: "qsub cpi_job.pbs".
  7. If the job is submitted successfully, it should yield output like this:

  8. When the job finishes, you will be able to find the output of the job in your home directory with files named cpi_job.eXXX and cpi_job.oXXX where XXX is the job number.
  9. If the job does not complete immediately, that simply means that it has been queued up to execute later, when more processors are free.
  10. You can check the status of jobs in the queue using the "showq" command. This shows all jobs that are currently in the queue, including those that must wait until more resources are available. Currently running jobs will have an "R" status field, queued jobs will have a "Q" status field.
  11. If you change your mind and decide not to run the submitted job, it can be removed at any time with the "qdel" command, which takes as its argument the job ID as shown in the output of the mpisubmit script, or the first column of the qstat output.
To submit a different program, simply change the arguments to mpisubmit to reflect the number of processors that you wish to use and the executable that you wish to run.

Advanced batch scheduler usage

PBS includes many advanced features beyond those used in this tutorial.
  • send email when the job starts or stops
  • send stdout and stderr to alternative locations
  • run arbitrary shell commands before or after your program
To do some of these things, you need to create your own pbs script, and modify it appropriately. Most casual users will not need to do this, but if you are interested, please visit the Custom PBS Script Guide.

Using alternative versions of MPI (including MPICH2)

The default MPI installation is sufficient for most users. However, if you have special needs (such as support for MPI-IO, or features only included in MPICH2), then you may need to select from one of the other MPI packages available on adenine.

To see a list of available MPI packages, run this command: "use -l". There should be multiple versions of both mpich and mpich2 listed, among other software packages. To select one, simply run "use [package name]" (for example: "use mpich2"). This will update your environment (including the path and man page locations) to point to this version of MPI. If you intend to always use this same version of MPI, then you may want to add the appropriate use command(s) to your .cshrc or .bashrc file.

Important note!

If you switch to a different version of MPI, then you must recompile your program before submitting it to the queue. The mpisubmit program will automatically perform different actions depending on the MPI package that you have selected.

Running interactive jobs

It is possible to submit a job to the scheduler that will result in the creation of an interactive session, where you can manually run any commands that you wish. This may be useful for debugging sessions, or for manually sweeping through various parameters on the command line of your program.

To ask for an interactive session, run this command: "qsub -l nodes=4:ppn=2 -I". Substitute the value of nodes (number of nodes to use) and processors per node as necessary for your desired usage. When this job is successfully executed, it will present you with a login shell on one of the nodes that you were assigned. You can run any commands that you wish at this point. Simply type "exit" when you are done in order to release the nodes.

You can find a list of nodes that have been assigned (for example, as input to an mpirun command, or a script) by looking at the PBS_NODEFILE environment variable.

Currently Available Nodes

Most of the slave nodes are named a1, a2, a3, a4, etc. Occsionally some of theme will be offline for maintenance.

To see a list of nodes that are currently operational on the system, run this command: "pbsnodes -a", or check the contents of the /etc/hosts-cluster file.

Available File Systems

Your home directory, /home/"username", is visible (mounted via NFS) from every node in the cluster. /tmp may be used for local temporary storage on each node. PVFS (a high performance parallel file system) will be added at a later date.