Information on Triton Cluster Usage

Triton Cluster

Triton Cluster contains 256 blade nodes. Each node has 2 quad-core Intel Nehalem 2.4 GHz processors with 24 GB memory and 20 TeraFlops peak performance. Addition information can be found:

Login
To use Triton, log in to the front-end machine: triton-login.sdsc.edu

The files you see from the login node are typically shared among all nodes in the cluster. But use this login machine only for compiling and general tasks. For running MPI or other parallel jobs, use the queuing system discussed below to access the computer cluster.

Account balance

Every class user is allocated with a certain amount of service units. Use the gbalance -u username command to check the amount of time in your account.

Your account is billed according to the formula:

#CPUs x #nodes x wall-clock-time.

SU charge is rounded to the nearest CPU node hour. For example, if you run with 8 CPUs (cores) on 1 node for 12 minutes, your account will be charged 2 SUs.

It is easy to accidentally chew through your time allocation by running a program with dead lock or leaving an interactive session open. One thing you can do is to always specify a maximum time the job can execute. For example, 10-20 minutes.

Compile MPI programs
By default, you will be using the Portland Group compiler. It gets called from the OpenMP compiler wrapper.

mpicc -o test1 test1.c

Running MPI programs in an interactive mode
Triton uses the TORQUE scheduler for running jobs with four different queues available. The only two we will probably want are small and batch. Job execution can be conducted in two modes: batch and interactive. Command qsub -I allows you to submit interactive jobs, where you are given direct access to a set of resources.

Use the command
```
qsub -I -l nodes=1:ppn=8 -l walltime=00:15:00 
```
to start an interactive job on a single node with 8 cores for at most 15 minutes.
Once you run it, it will wait until a node is available, and then it will create an SSH session to that node. Your job will last until you exit that SSH session.
Then run an MPI job interactively as:
```
 mpirun -np  8 test1 
```
When running MPI jobs, keep the "ppn" parameter set to 8. If running hybrid MPI-OpenMP jobs, set ppn=1. Note that your job will be charged for all cores on a node no matter how many cores you are using. As an that interactive jobs deplete your bank account quickly, it is good to use 1 node.
Caveats: interactive jobs will be scheduled just like script-based jobs, so they'll take as long to execute as the queue is. Right now, this doesn't seem to be a problem, so running with the batch queue is fine. If the queue is slow, remember to stop it (control-C works) so that you don't waste a bunch of time doing nothing.

Compile MPI programs in a batch mode
Job submission is done with a job script file with a number of options. These include the queue, the number of nodes, the title of your job, the maximum time it can take, where to put console output, and email notifications at job start and end, and Linux commands that run your programs.

To run an MPI job, so your script can include the two lines in the following format:

cd ~/path-to-program
mpirun -v -machinefile $PBS_NODEFILE -np somenumber ./program

Command
```
qsub jobfile 
```
will submit your job to the scheduler and assign it a job number. Be careful when you are submitting multiple jobs. The output files will get overwritten by subsequent or concurrent jobs writing to the same file.
qstat -a and showq -u $USER are useful for examining the current state of your jobs. In qstat, Q means queued, R means running, and C means complete. showq will give you slightly more detailed information about your jobs.
Sometimes, one queue will be more full than the other. If jobs are taking too long to run in the batch queue, try the small queue instead, or vice-versa. To do this, you should first delete the jobs you created previously. Since they haven't run yet, you can delete them with qdel jobnumber, where jobnumber was assigned by qsub.

Compile and Run OpenMP programs
The OpenMP compiler in this cluster is from Intel and you need to add these lines in your .bashrc:

module load intel

To compile, use icpc or icc compiler with the following sample command:

icpc -openmp test.c -o test

To run, you can use the interactive or batch mode as MPI to allocate one node with upto 8 cores. Set up the actual number of openMP threads using.

export OMP_NUM_THREADS=8

At most you can use 8 openMP threads since openMP assumes a shared memory machine setting.

Sample code and makefile in Triton

	~tyang-ucsb/mpiex     ------------MPI sample code
	~tyang-ucsb/openmpex     ------------OpenMP sample code
	~tyang-ucsb/pthreads     ------------pthreads code
	~tyang-ucsb/MKL     ------------Use Intel math library including BLAS.