BBQ Howto
By Tim Shelling and Dan Stanzione
December 2002

INTRODUCTION

The Clemson mini-grid was designed to give you, the scientist and 
researcher, the ability to solve large computational problems in a 
highly scalable and highly functional computing environment. In aiming 
to make that environment fmore functional, we have implemented a BATCH 
QUEUING SYSTEM. The goals of this system are to keep the cluster busy by 
ensuring that more jobs run simultaneously and to completion and without 
interference from each other.  In short, the aim of the queuing system, 
is to make the environment more functional for you.

BBQ is the Queuing mechanism we have implemented. BBQ is provided by the 
makers of the OS, and is appropriately called Beowulf Batch Queue. BBQ 
is based on the standard unix facility AT and the corresponding daemons 
ATD and CRON. BBQ also integrates functions native to Beowulf, to allow 
it to determine when a particular node is permitted and idle enough to 
run a particular request.  BBQ also integrates very tightly with MPI, 
and you can take advantage of this by specifying the number of MPI tasks 
or even the "mapping" of those tasks to nodes. For those non-MPI aware 
programs, BBQ can take advantage of mpirun.

All this and more will be detailed below.


We have chosen, from this point forward for users of THYMINE, to ENFORCE 
your using BBQ to run jobs. We do this by restricting all nodes to run 
ONLY jobs owned by the "daemon" user/group. Obviously, BBQ is set up to 
take advantage of this.  For those of you who want to test your code 
without waiting for nodes to become available, we will allow some nodes 
to be run by anyone. Note, however, that we will assume these jobs are 
undergoing testing and/or debugging, and we reserve the right to 
terminate processes.
[Dan, is this too harsh?]

[Dan, I copied the *administrator* portion instead of the *user* manual 
for BBQ. So I've basically winged it here.]


STARTING JOBS

You have an MPI program named "hello_comm" that you would like to run on 
the cluster.  What you will do is submit a job to BBQ via the command 
BATCH. This command takes a SHELL SCRIPT as input. You can specify this 
script as a named script, or as piped input, or as prompted input from 
the terminal, such as this:

$ batch
 > NP=10
  > $HOME/bin/hello_comm
   > 

	At the last line, you would have hit CTRL-D to terminate the script 
	input.

	As you can see from above, you must specify hello_comm in the script. It 
	is always advisable to specify the path like this, but it will work if 
	you know the program in question is a directory specified by your 
	exported PATH variable.

The line NP=10 controls MPI, as you probably already know, to run the 
program on 10 *CPU's* (not nodes).  BBQ will allocate 1 process per 
processor (not "node"). In this case, BBQ will find 10 processors which 
are mostly idle and run this job on those processors, as long as these 
processors are on nodes that are part of the BBQ cluster. The head node 
will *never* be a part of the BBQ cluster. [Dan, I need help with this: 
How do users specify a root node to run on the head? Like a collator 
process?]

But what if 10 processors are NOT available? BBQ will *wait* until 10 
are sufficiently idle to start your job. Correspondingly, if you specify 
BEOWULF_JOB_MAP so that your job runs on X specific processors, it may 
take a long time for those *Specific* nodes to be available.  Unless 
topology is important to you, it is probably best to specify how many 
processes you want to run, rather than which nodes will run (or not run, 
via EXCLUDE) that process.

The line NP=10 controls MPI, as you probably already know, to run the 
program on 10 *CPU's* (not nodes).  BBQ will allocate 1 process per 
processor (not "node"). In this case, BBQ will find 10 processors which 
are mostly idle and run this job on those processors, as long as these 
processors are on nodes that are part of the BBQ cluster. The head node 
will *never* be a part of the BBQ cluster. [Dan, I need help with this: 
How do users specify a root node to run on the head? Like a collator 
process?]

But what if 10 processors are NOT available? BBQ will *wait* until 10 
are sufficiently idle to start your job. Correspondingly, if you specify 
BEOWULF_JOB_MAP so that your job runs on X specific processors, it may 
take a long time for those *Specific* nodes to be available.  Unless 
topology is important to you, it is probably best to specify how many 
processes you want to run, rather than which nodes will run (or not run, 
via EXCLUDE) that process.


The "batch" command has the same syntax as the unix command "at". 
However, in at, there is a requirement that a start-time be specified. 
In BATCH, the time specification is optional and defaults to "NOW". Of 
course, NOW really doesn't mean "now", rather it means "as soon as there 
are enough resources to run me".  There may be other jobs in the queue 
that are run before yours, so NOW, might in fact, be a very long time 
from "now".

VIEWING THE QUEUE

You can view the queue in two ways. On the command line, you may run 
"bbq", which outputs the pending and running jobs. To see the same 
information, updated once every 5 seconds, run "beoqstat" in an X 
display. [Future: We will ultimnately provide this interface via HTTP 
and you can see the information in a browser.]

REMOVING JOBS

You can use the command "atrm" to remove one of your jobs in the BBQ. 
  "atrm" takes a job number as a parameter [need to check this]. The job 
  number is part of the output from bbq. So first, run "bbq", write down 
  the job number, then run "atrm" to remove it.

  MORE INFORMATION

  For more information, we suggest you read the on-line man pages.
  	man batch
	man at
	man atrm
	man bbq
	man beoqstat
	man atd