Slurm Partitions
MSI uses the Slurm scheduler to fairly allocate compute resources to users of our systems. Slurm uses different partitions to organize jobs with similar characteristics. The job partitions on our systems manage different sets of hardware, and have different limits for quantities such as wall clock time (walltime), available processors, and available memory. When submitting a calculation it is important to choose a partition where the job is suited to the hardware and resource limitations.
Selecting a Partition
Each MSI system contains job partitions managing sets of hardware with different resource and policy limitations. MSI currently has three primary systems: the Agate cluster, the Mesabi cluster, and the Mesabi expansion Mangi. These systems have high-performance hardware and a wide variety of partitions suitable for many different job types. Agate is the latest cluster with the best hardware, and should be your first choice for submitting jobs. Which system to choose depends highly on which system has partitions appropriate for your software/script. More information about selecting a partitions and the different partition parameters can be found on the Choosing A Partition (Slurm) page.
Below is a summary of the available partitions organized by system, and the associated limitations. The quantities listed are totals or upper limits.
Federated Partitions
Paritions for jobs that can run on either Agate or Mesabi/Mangi.
Job arrays will only run on the cluster you submit the job on.
| Partition name | Node sharing? | Cores per node | Walltime limit | Total node memory | Advised memory per core | Local scratch per node | Maximum nodes per job |
|---|---|---|---|---|---|---|---|
| msismall | Yes | 24-128 | 96:00:00 | 40-499 GB | 1900 MB | 380-850 GB | 1 |
| msilarge | No | 24-128 | 24:00:00 | 40-499 GB | 1900 MB | 380-850 GB | 32 |
| msibigmem | Yes | 24-128 |
24:00:00 |
499-1995 GB |
4000 MB | 380-850 GB | 1 |
| msigpu | Yes | 24-128 | 24:00:00 | 40-499 GB | 1900 MB | 380-850 GB | 1 |
| interactive | Yes | 24-128 | 24:00:00 | 60-499 GB | 2 GB | 228-850 GB | 2 |
| interactive-gpu | Yes | 24-128 | 24:00:00 | 60-499 GB | 2 GB | 228-850 GB | 2 |
| preempt | Yes | 24-64 | 24:00:00 | 60-499 GB | 2 GB | 228-850 GB | 1-2 |
| preempt-gpu | Yes | 24-64 | 24:00:00 | 60-499 GB | 2 GB | 228-850 GB | 1-2 |
Note: Partitions with the same name across clusters are federated (interactive, interactive-gpu, preempt, preempt-gpu).
Agate Partitions
| Partition name | Node sharing? | Cores per node | Walltime limit | Total node memory | Advised memory per core | Local scratch per node | Maximum nodes per job |
|---|---|---|---|---|---|---|---|
| agsmall | Yes | 128 | 96:00:00 | 499 GB | 4000 MB | 850 GB | 1 |
| aglarge | No | 128 | 24:00:00 | 499 GB | 4000 MB | 850 GB | 32 |
| ag2tb | Yes | 128 | 96:00:00 | 1995 GB | 15.5 GB | 850 GB | 1 |
|
a100-4(1) |
Yes | 64 | 24:00:00 | 499 GB | 4000 MB | 850 GB | 4 |
| a100-8(1) | Yes | 128 | 24:00:00 | 1002 GB | 7.5 GB | 850 GB | 1 |
| interactive | Yes | 128 | 24:00:00 | 499 GB | 4000 MB | 850 GB | 2 |
| interactive-gpu(2) | Yes | 64 | 24:00:00 | 499 GB | 8000 MB | 850 GB | 2 |
| preempt | Yes | 128 | 24:00:00 | 499 GB | 4000 MB | 850 GB | 1 |
| preempt-gpu(2) | Yes | 64 | 24:00:00 | 499 GB |
8000 MB |
850 GB | 1 |
(1)Note: Jobs using an Agate partition must be submitted from an Agate host (ex. Agate login node).
(2)Note: In addition to selecting a100-4 or a100-8 GPU partitions, GPUs need to be requested for all GPU jobs.
One or more A100 GPU can be requested by including the following two lines in your submission script:
This example asks for a single A100 GPU, using the a100-4 partition
#SBATCH -p a100-4
#SBATCH --gres=gpu:a100:1
(3)Note: The interactive-gpu and preempt-gpu partitions contain A40 GPUs, so include the following two lines in your submission script:
This example asks for a single A40 GPU, using the interactive-gpu partition
#SBATCH -p interactive-gpu #SBATCH --gres=gpu:a40:1
Mangi and Mesabi
| Partition name | Node sharing? | Cores per node | Walltime limit | Total node memory | Advised memory per core | Local scratch per node | Maximum nodes per job |
|---|---|---|---|---|---|---|---|
| amdsmall | Yes | 128 | 96:00:00 | 248 GB | 1900 MB | 415 GB | 1 |
| amdlarge | No | 128 | 24:00:00 | 248 GB | 1900 MB | 415 GB | 32 |
| amd512 | Yes | 128 | 96:00:00 | 499 GB | 4000 MB | 415 GB | 1 |
| amd2tb | Yes | 128 | 96:00:00 | 1995 GB | 15 GB | 415 GB | 1 |
| v100 (1) | Yes | 24 | 24:00:00 | 374 GB | 15 GB | 859 GB | 1 |
| small | Yes | 24 | 96:00:00 | 60 GB | 2500 MB | 429 GB | 10 |
| large | No | 24 | 24:00:00 | 60 GB | 2500 MB | 429 GB | 48 |
| max | Yes | 24 | 696:00:00 | 60 GB | 2500 MB | 429 GB | 1 |
| ram256g | Yes | 24 | 96:00:00 | 248 GB | 10 GB |
429 GB |
2 |
| ram1t (2) | Yes | 32 | 96:00:00 | 1002 GB | 31 GB | 380 GB | 2 |
| k40 (1) | Yes | 24 | 24:00:00 | 123 GB | 5 GB | 429 GB | 40 |
| interactive (3) | Yes | 24 | 24:00:00 | 60 GB | 2 GB | 228 GB | 2 |
| interactive-gpu (3) | Yes | 24 | 24:00:00 | 60 GB | 2 GB | 228 GB | 2 |
| preempt (4) | Yes | 24 | 24:00:00 | 60 GB | 2 GB | 228 GB | 2 |
| preempt-gpu (4) | Yes | 24 | 24:00:00 | 60 GB | 2 GB | 228 GB | 2 |
(1)Note: In addition to selecting a GPU partition, GPUs need to be requested for all GPU jobs. A k40 GPU can be requested by including the following two lines in your submission script:
#SBATCH -p k40
#SBATCH --gres=gpu:k40:1
A V100 GPU can be requested by including the following two lines in your submission script:
#SBATCH -p v100
#SBATCH --gres=gpu:v100:1
(2)Note: The ram1t nodes contain Intel Ivy Bridge processors, which do not support all of the optimized instructions of the Haswell processors. Programs compiled using the Haswell instructions will only run on the Haswell processors.
(3)Note: Users are limited to 2 jobs in the interactive and interactive-gpu partitions.
(3)Note: Jobs in the preempt and preempt-gpu partitions may be killed at any time to make room for jobs in the interactive or interactive-gpu partitions.