IBM Power4

Introduction

The Operating System on the Power4 systems is AIX 5.3.

Hardware and Configuration

• There are four p690 (Regatta) nodes.
  • One 1.3 Ghz 32-processor p690 (Regatta) sharing 64 GB of memory
  • One 1.3 Ghz 24-processor p690 (Regatta) sharing 24 GB of memory
  • One 1.7 Ghz 32-processor p690 (Regatta) sharing 128 GB of memory
  • One 1.7 Ghz 32-processor p690 (Regatta) sharing 64 GB of memory
• There are 12 1.5 GHz 8-processor p655 nodes. Each node contains 16 GB of memory.
• There are 11 1.7 GHz 8-processor p655 nodes. Each node contains 16 GB of memory. One of these nodes is the interactive node.
• In addition, there are four 4-processor nodes that are used as file servers.

If desired, you can specify the type of nodes that you want your job to run on. Specifying the type of node is optional. If you do not specify the type of node, it will use whatever processors are available. This will not present a problem for serial jobs, but may be an inefficient use of the system for parallel jobs. Not specifying a feature may mean a parallel jobs will run on nodes with different processor speeds leading to faster processors waiting for results from slower processors. For benchmarking codes, we recommend you specify the type of node. Nodes are specified through features. The following features are available:

The IBM Power4 system has separate home directories and scratch spaces from the IBM SP.

Memory available

* This includes memory for large pages. If you are not using large pages, subtract the amount of memory reserved for large pages from this number.

Wall clock limits:

Operations

• Login Procedure

  You can only login to the interactive node, which is one of 8-processor p655 nodes. This node is used for compiling and job submission. To run on the nodes of the Power4 system, you have to submit your job to the queue.

  To login:

   ssh -X -l username regatta.msi.umn.edu 

  You only need to use -l if your login name is different than the login name you use on your own workstation.

• Interactive Use

  There is one node for interactive jobs. Interactive use should be restricted to compiling codes and code testing (both serial and parallel), and for submitting jobs to the queue. For parallel jobs, you may use up to 4 processors. Larger jobs and all productions jobs must be submitted to the queues.

• The Scratch Space

  There are four scratch spaces installed on the Power4 system:

             /scratch1 : 800 GB
             /scratch2 : 800 GB
             /scratch3 : 800 GB
             /scratch4 : 800 GB
            

• How to compile

  FORTRAN:

            Fortran compiler level 7.1.1.2 is installed and fortran codes can be compiled
            using xlf or xlf90 for serial codes and mpxlf and mpxlf90 for MPI parallel
            codes. Please look at the man page for compiler options.
  
             Examples:
               Fortran 77: xlf -O3 -qtune=pwr4 -qarch=pwr4 prog.f
               Fortran 90: xlf90 -O3 -qtune=pwr4 -qarch=pwr4 prog.f
               Fortran 95: xlf95 -O3 -qtune=pwr4 -qarch=pwr4 prog.f
               Thread-safe: xlf_r, xlf90_r and xlf95_r
               MPI       : mpxlf
               MPI Thread-safe: mpxlf_r
            

  C:

            C compiler level 5.0.2.2 is installed and C codes can be compiled using xlc
            for serial codes and mpcc for mpi parallel code.
  
             Examples:
               C: xlc
               Thread-safe C: xlc_r
               MPI C: mpcc
               MPI Thread-safe C: mpcc_r
            

• Submitting batch jobs to the queue using Loadleveler

  We have only one queue for the entire system. You do not need to specify a queue class in your LoadLeveler script. As mentioned above, you can specify a feature to specify the type of node that your job will use.

  Maximum number of jobs: The maximum number of jobs that a person can have in the queue on the IBM Power4 is 32.

  Example of Loadleveler script to submit a serial batch job

          #!/bin/csh
          #@ initialdir = /homes/r30/runesha/TESTS
          #@ output     = mm_serial.output
          #@ error      = mm_serial.error
          #@ wall_clock_limit = 10:00
          #@ resources =  ConsumableMemory(500)
          #@ queue
          ./a.out
          

  No feature is specified in the above example, so the job will run an any available processor on the system.

  Example of Loadleveler script to submit a parallel MPI batch job

          #!/bin/csh
          #@ initialdir   = /homes/r30/runesha/TESTS
          #@ output       = mpi_$(jobid).out
          #@ error        = mpi_$(jobid).err
          #@ job_type     = parallel
          #@ wall_clock_limit = 20:00
          #@ resources    = ConsumableMemory(700)
          #@ network.MPI = css0,shared,IP
          #@ blocking = unlimited
          #@ total_tasks = 4
          #@ requirements = (Feature == "Regatta")
          #@ node_usage   = shared
          #@ queue
          poe ./a.out
          

  In this example the feature "Regatta" is specified so the parallel job will run only on the 1.3 Ghz Regatta nodes. Here is an example on using the Loadleveler feature "FastRegatta" in a script.

  Example of a script to submit a MPI batch job with the Loadleveler feature "FastRegatta"

          #!/bin/csh
          #@ initialdir   = /homes/r30/runesha/TESTS
          #@ output       = mpi_$(jobid).out
          #@ error        = mpi_$(jobid).err
          #@ job_type     = parallel
          #@ wall_clock_limit = 20:00
          #@ resources    = ConsumableMemory(700)
          #@ blocking = unlimited
          #@ total_tasks = 4
          #@ requirements = (Feature == "FastRegatta")
          #@ node_usage   = shared
          #@ queue
          poe ./a.out
          

  A similar modification in the Loadleveler script is needed in order to use the "p655" feature.

  Example of Loadleveler script to submit an OpenMP batch job

          #!/bin/csh
          #@ initialdir   = /homes/r30/runesha/TESTS
          #@ output       = OpenMP_test.out
          #@ error        = OpenMp_test.err
          #@ job_type     = parallel
          #@ wall_clock_limit = 30:00
          #@ resources    = ConsumableMemory(500)
          #@ node         = 1
          #@ tasks_per_node = 4
          #@ node_usage   = shared
          #@ queue
          setenv OMP_NUM_THREADS 4
          ./a.out
          

  Loadleveler commands

           - Submitting a job:  llsubmit test.ll 
           - Monitoring job's status:  llq 
           - Canceling a job:   llcancel job_id 
  
           where test.ll is your LoadLeveler script and job_id is the job id
           that is returned when you submit a job to the queue.
  
           

Documentation