Introduction to Using SLURM for Job Scheduling
SLURM (Simple Linux Utility for Resource Management) is a widely-used workload manager for clusters, designed to allocate resources efficiently and manage job queues. As a cluster workload manager, Slurm has three key functions:
- It allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work.
- It provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes.
- Finally, it arbitrates contention for resources by managing a queue of pending work.
SchedMD provides a full set of documentation on SLURM. Please check out their quick start guide and download their cheat sheet before you begin. This page is only a short tutorial and provides an introductory guide on using SLURM for submitting and managing jobs on the ReD Environment.
We also suggest that you browse FASRC's detailed documentation on using SLURM: Running Jobs, Convenient SLURM Commands, and Cluster Customs and Responsibilities.
Understanding Basic SLURM Concepts
Node: A virtual server (an AWS EC2 instance) within the cluster.
Partition: A logical grouping of nodes of the same EC2 instance type, functioning similarly to queues.
Job: A user-submitted task to execute code that has specific requests for compute resources (e.g. RAM, CPU cores, etc.).
Fairshare A scoring system that prioritizes users who have not used computing resources over those that have already done so, allowing one's 'fair share' use of the cluster.
ReD Environment Specific Setup
- Unlike many SLURM environments, which are on-prem physical nodes, in the ReD Environment all AWS nodes are VM instances that are dynamically provisioned and destroyed.
- Partitions, and thus nodes, are not shared between labs. This means that the typical Fairshare sorting of jobs does not apply, and you are not in competition with other labs for resources.
- The number and type of nodes are not fixed. Upon request and approval by your PI Sponsor, we can add additional partitions and instance types for your lab.
- By default, no nodes are allocated as standby nodes, and thus one must wait for a node to be provisioned for all job types, including interactive jobs. This can be changed via approval from the lab PI/sponsor.
- If a compute node is running, you're paying for it, regardless of the percent utilization of the on-machine resources.
Attention! It is important to understand that the ReD environment is configured differently that FASRC's Cannon or FASSE clusters. Please don't assume that everything works the same way. Please approach using this resource understanding its default configuration and reexaminging all script paramaters and options. Failure to do so could result, in the worst case, in high costs for your compute usage.
Interacting with SLURM
One can typically interacts with the SLURM scheduler to submit jobs, monitoring their execution, and for troubleshooting:
- For submitting GUI interactive applications and GUI job monitoring, use the OOD Interactive Apps menu and the Jobs > Active Jobs menu items.
- For submitting interactive command-line jobs, batch jobs, or more thorough job monitoring and troubleshooting, open a terminal window via selecting the Dashboard menu Clusters >
_urcds-red-pc-prod Shell Access. - For more complex tasks, including submitting job arrays, one can use OOD's Job Composer via the Dashboard menu Jobs > Job Composer. Please see Open OnDemand's Job Composer documentation.
Due to security requirements, the typical ssh into a login node from your local computer is not availble for the ParallelCluster.
Submitting Jobs
Via the command line, one can run interactive command-line jobs or submit batch jobs via job scripts.
Interactive Command-Line Jobs
One can get an interactive session on a compute node via the command line using the salloc command. For example, to allocate two CPUs in the urcdtest-small partition for 2 hrs, execute:
SLURM works with ParallelCluster to provision the compute node and allocate the resources to your job:
salloc: Pending job allocation 43398
salloc: job 43398 queued and waiting for resources
salloc: job 43398 has been allocated resources
salloc: Granted job allocation 43398
salloc: Waiting for resource configuration
...
salloc: Nodes urcdtest-small-st-urcdtest-small-cr-0-1 are ready for job
It may take up to 10 minutes to provision the node. Once done, use the srun command to get a shell so that you can run your scripts or execute your commands:
[jharvard@ip-10-3-2-176 ~]$ srun --pty /bin/bash
Warning: Permanently added 'urcdtest-small-st-urcdtest-small-cr-0-1,10.3.3.33' (ECDSA) to the list of known hosts.
Authorized uses only. All activity may be monitored and reported.
Last login: Thu Oct 9 00:06:49 2025
, #_
~\_ ####_ Amazon Linux 2
~~ \_#####\
~~ \###| AL2 End of Life is 2026-06-30.
~~ \#/ ___
~~ V~' '->
~~~ / A newer version of Amazon Linux is available!
~~._. _/
_/ _/ Amazon Linux 2023, GA and supported until 2028-03-15.
_/m/' https://aws.amazon.com/linux/amazon-linux-2023/
[jharvard@ip-10-3-2-176 ~]$$
Note that one can also use the SSH (ssh urcdtest-small-st-urcdtest-small-cr-0-1) command to drop into the compute node session as well. Although srun is preferred, SSH is useful as a way to peek on what is going on in the job.
Note: There is a 15 minute idle timeout on OOD terminal windows. If the session times out, you will see the message Your connection to the remote server has been terminated. One can always open a new terminal session and connect to the existing job/resources with one of two options:
- Use
srun -j JOBID, or - SSH once again into the compute node allocated to you
Batch Jobs
The example job script and instructions below demonstrate how to run a python script in one's project. This is an example pattern and will not work as written, as most ReD programs are requiremed to run inside containers. Also note the scripts brought from FASRC's Cannon or FASSE clusters will not work without modification.
Please see the doc pages for your preferred scripting language for the most up-to-date instructions.
Create a SLURM batch script (my_slurm.job):
#!/bin/bash
#SBATCH --job-name=my_job # Job name
#SBATCH --output=%x_%j.out # Output file (%x expands to Job name, %j expands to job ID)
#SBATCH --error=%x_%j.err # Error file
#SBATCH --ntasks=1 # Number of tasks (processes)
#SBATCH --cpus-per-task=1 # Number of CPU cores per task
#SBATCH --nodes=1 # Use only one node (== not an MPI job)
#SBATCH --mem=4G # Total memory per node
#SBATCH --time=00:30:00 # Time limit (hh:mm:ss)
#SBATCH --partition=urcdtest-med # Partition name
#SBATCH --open-mode=append # Ensure that log files are appended (vs truncate) on job restarts
# Set up global, project variables
export LAB_DIR=/path/to/harvardj_lab
export PROJ_ROOT=${LAB_DIR}/projects/my_project
# ensure we use our project modules first instead of ones in my home folder
export PYTHONPATH="${PROJ_ROOT}/lib/python3.11/site-packages:$PYTHONPATH"
# Working from the project root
cd ${PROJ_ROOT}
# Run my application
python code/my_script.py
# Report SLURM job efficiency when exiting
seff $SLURM_JOBID
Submit the job via the command sbatch my_slurm.job.
Note: Email notifications for job state changes are not available in ReD, as we have to limit the likelihood that confidential info or regulated data might be accidentally communicated in the message body.
Monitoring Jobs
To check the status of your active jobs, use the squeue command:
[jharvard@ip-10-3-2-176 ~]$ *squeue*
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
43403 urcdtest- sys/dash jharvard R 0:08 1 urcdtest-small-st-urcdtest-small-cr-0-1
43409 urcdtest- interact jharvard CF 0:10 1 urcdtest-serial-dy-urcdtest-serial-cr-0-1
[jharvardt@ip-10-3-2-176 ~]$
This command lists your active, pending, or recently completed jobs. Important fields include:
- JOBID: Unique job identifier.
- PARTITION: Specific partition the job is running in
- NAME: Name of the job
- ST: Job state (e.g., CF, configuring, PD for pending, R for running).
- TIME: Runtime duration.
- NODES: Number of nodes job is occupying
You can customize the output of the squeue command by setting the environment variable SQUEUE_FORMAT. For example, adding the following line to your ~/.bashrc:
results in the following output with squeue:
[jharvard@ip-10-3-2-176 dd-test]$ squeue
JOBID PARTITION NAME USER STATE TIME TIME_LIMI CPUS NODES NODELIST(REASON)
216 urcdtest-med dd.job jharvard CONFIGUR 4:02 UNLIMITED 16 1 urcdtest-med-dy-urcdtest-med-cr-0-1
214 urcdtest-med dd.job jharvard RUNNING 5:01 UNLIMITED 16 1 urcdtest-med-st-urcdtest-med-cr-0-1
See the SchedMD docs on squeue for a full set of options and field definitions.
Viewing Job Details
For more detailed job information use the scontrol command:
scontrol show job JOBID
Canceling Jobs
To cancel a job, supply the JOBID to scancel:
scancel JOBID
or kill all your jobs using ID 0:
scancel 0
Troubleshooting Jobs
- I've submitted a job, but it hasn't started and more than 10 minutes have elapsed. What do I do?
There's a very good likelihood that a problem with the compute node has prevented the job from starting, and the SLURM controller/scheduler hasn't dispatched the job yet. Let's peek at what SLURM knows about the job via sacct and having SLURM show duplicate (-D) job info:
[jharvard@ip-10-3-2-176 ~]$ sacct -j 43553 -D
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
43553 sys/dashb+ dominici-+ 1 NODE_FAIL 0:0
43553 sys/dashb+ dominici-+ 1 RUNNING 0:0
Checking Job Utilization
sacct Details
SLRUM has a rich accounting database of prior run jobs. This is a great way to check how long your job ran, how much memory, ... Like squeue, many different fields can be displayed with sacct. By default sacct provides only limited information:
[jharvard@ip-10-3-2-176 ~]$ sacct
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
214 dd.job urcdtest-+ 16 COMPLETED 0:0
214.batch batch 16 COMPLETED 0:0
214.extern extern 16 COMPLETED 0:0
215 dd.job urcdtest-+ 0 FAILED 1:0
Similar to squeue, you can update your ~/.bashrc with an environment varible for sacct to enhance your output. For example adding:
results in the following output:
[jharvard@ip-10-3-2-176 ~]$ sacct
JobID User AllocCPUS ReqMem MaxRSS TotalCPU Start End Elapsed
------------ --------- ---------- ---------- ---------- ---------- ------------------- ------------------- ----------
214 jharvard+ 16 31129M 07:08.715 2025-01-15T21:20:10 2025-01-15T21:27:22 00:07:12
214.batch 16 197124K 07:08.713 2025-01-15T21:20:10 2025-01-15T21:27:22 00:07:12
214.extern 16 764K 00:00.001 2025-01-15T21:20:10 2025-01-15T21:27:22 00:07:12
215 jharvard+ 0 3891M 00:00:00 2025-01-15T21:20:25 2025-01-15T21:20:25 00:00:00
MaxRSS is the max memory used. In job 214, 31G was requested, but only 197M was consumed. This means I could have used a much smaller instance type to conserve on resources (and cost).
There are multiple ways that one can use this information, including checking the % efficienty of parallelized code. Please reach out to the ReD Team if you have additional questions.
See the manual on sacct for a full set of options and field definitions.
seff Command for SLURM Job Efficiency
SLURM now includes a command to help one understand the efficiency of your jobs: seff takes a jobid and reports on that job's cpu and memory utilization. This allows users to become aware if they are wasting resources.
$ seff
Usage: seff [Options] <Jobid>
Options:
-h Help menu
-v Version
-d Debug mode: display raw Slurm data
The seff output is mostly self-explanatory:
$ seff 3485050
Job ID: 3485050
Cluster: della
User/Group: dmcr/cses
State: COMPLETED (exit code 0)
Nodes: 2
Cores per node: 2
CPU Utilized: 00:00:01
CPU Efficiency: 0.40% of 00:04:08 core-walltime
Memory Utilized: 2.04 GB (estimated maximum)
Memory Efficiency: 86.89% of 2.34 GB (1.17 GB/node)
While the command may be applied to running or interactive jobs, the statistics may be misleading:
- As statistics are taken at a regular interval, not continuously, job stats early on may be relatively meaningless.
- CPU stats for interactive sessions may not accurately reflect CPU usage, unless efforts are made to minimize idle time. Memory stats, however, should be fairly accurate.
If used in SLRUM job scripts as the last line of the script file, the job statistics will be appended to your job output file:
seff is built to report on one job only. To check the resource efficient of multiple jobs:
seff does not handle job arrays, nor can it give more broad statistics about usage. Yale's Center for Research Computing has published extensions to this command for job arrays and accounts; we are working to deploy these codes.
Job Dependencies
Submit a job that starts only after another completes:
sbatch --dependency=afterok:JOBID my_dependent_job.slurm
This line schedules my_dependent_job.slurm to start only if JOBID finishes successfully.
Advanced Tips
Resource Optimization: Adjust --cpus-per-task and --mem according to your job's requirements for optimal resource use. Array Jobs: Easily submit multiple similar jobs using job arrays with sbatch --array=0-9 my_slurm.job
Conclusion
SLURM is a robust and flexible tool that helps efficiently manage compute resources in cluster environments. By mastering the basic commands and concepts outlined above, you can effectively run your computational tasks. For more information, explore the SLURM documentation for detailed guidance and advanced functionalities.
Last Updated: Oct 10, 2025