Short Instructions on Using the DataStar at SDSC

This is a short guide which will get you started on using the IBM DataStar supercomputing cluster at SDSC. For more detailed instructions, please refer to the excellent documentation on SDSC's website: http://www.npaci.edu/DataStar

1. Introduction

DataStar consists of nodes of two types: 8-way P655+ and 32-way P690 nodes (see the table below). The use of 8-way nodes is exclusive: users will have exclusive access to any requested 8-way node(s) during job execution, and they will be charged for using 8 cpus per node (whether or not they use 8 cpus per node). The use of 32-way nodes is shared among users, but subject to CPU and memory constraints specified in the batch script. Performance of your job on these nodes may vary depending on the other users' jobs competition over memory and I/O bandwidth. The nodes are suitable for both shared-memory (e.g.OpenMP or Pthreads) and message-passing (e.g. MPI) programming models, as well as the mixture of the two.
# Nodes Node Type Memory per Node CPUs per Node CPU Speed CPU Peak Performance
6
690+
128 GB
32
1.7 GHz
6.8 GFlops
1
690+
256 GB
32
1.7 GHz
6.8 GFlops
176
655+
16 GB
8
1.5 GHz
6.0 GFlops
In addition to the above nodes for running batch-scheduled jobs, DataStar also provides one interactive node and one login node (dslogin). The interactive and login nodes are p690 (32-way) with 64 GB of memory. We discuss how to use these nodes in later sections.

2. Logging in

To access your DataStar account, login to the frontend/login node:
ssh yourusername@dslogin.sdsc.edu
Please do not use this node for direct program execution. Just use it for program development/debugging, batch job submission and to transfer files (via scp) to or from DataStar.

3. Compiling Programs

The relevant sections in the User Guide explain the process of compiling Serial, MPI and OpenMP programs very clearly:
· Compiling Serial Programs
· Compiling MPI Programs
· Compiling OpenMP Programs

4. Executing Programs

· Interactive Execution

There is ONE 32-way node available on DataStar where you can run your programs interactively. You can access it using:
ssh yourusername@dspoe.sdsc.edu
However, remember this is a shared node and hence the performance of your program might highly vary for each run. You use the IBM Parallel Operating Environment (POE) to submit interactive jobs. This is explained in the User Guide in the following section:
Running: Interactive

· Batch Execution

The best way to execute your programs is through the Batch Scheduling system. DataStar uses the LoadLeveler batch scheduler to execute your programs. You provide a script to LoadLever for your program and it will enqueue/schedule it for execution. The script specifies various resource requirements for your program, environment variables, input/error log files, etc. Again, please read the following section in the User Guide:
Running: Batch Jobs - LoadLeveler
In particular, read the sub-section titled "8-way nodes" which explains everything with a relevant example. Again, please remember there are different queues for different machine requirements. You have to specify the appropriate batch queue in your LoadLeveler script file. One more thing to remember: 8-way nodes are always exclusively owned by your program once it gets batch-scheduled on them. On the other hand, 32-way nodes are shared even in batch-mode.

Here's a working script-file for an OpenMP job: Script File
I recommend you work with this file for building your own scripts. A short explaination of the sample script is in order here: Most of the syntax is explained well enough in the SDSC User Guide. However, please note some of the important directives:
#@ initialdir: This specifies what will be your "current working directory" when your program executes. The output, error and executable files are relative to this directory. If omitted, your "current working directory" for the job will be set to the directory from which you submitted the job using llsubmit.
#@ output and #@ error: This is where your output and error will be stored after the job is done.


5. Important: Reservation!!

 To use the "reserved" nodes for class on 05/10 and 05/11 for your programs, please specify the "normal" queue in your LoadLeveler script. The system will automatically schedule your program on the nodes exclusively reserved for the class.