Atomic and Laser Linux Compute Cluster

For support, please email '' which will create a ticket and will be seen by the whole team.

The Basics

The new Atomic and Laser Cluster is running CentOS 7 with approximately 70Tbytes of raw network-attached storage and is accessible through the "interactive" login node

At present, the CentOS 7 cluster consists of one interactive node, allxfs1 file server (aka allxint1) and two worker nodes: allxwn01 and allxwn02. Slurm is the job scheduler for the new cluster.

Please note that interactive logins to the worker nodes is disabled for all users.


File/Interactive Server: allxfs1 aka allxint1

Dell PowerEdge R730xd; 16 logical CPUs (Intel Xeon CPU E5-2623 v4 @ 2.60GHz); 64Gb Memory

Compute Nodes: allxwn01, allxwn02

Dell PowerEdge R940; 144 Logical CPUs (Intel Xeon Gold 6154 CPU @ 3.00GHz); 192Gb Memory

NFS Mounted Data Partitions

  • /home
  • /software
  • /data/vinko
  • /data/wark
  • /data/gregori
  • /data/common

Slurm Quick Start Guide

This is an introduction to the Slurm (Simple Linux Utility for Resource Management) job scheduler. See for more details.


Jobs on the Atom and Laser Cluster run as batch jobs. In order to submit jobs to the cluster, log on to the interactive server, allxint1 (aka allxfs1). allxint1 is also the Slurm controller, which controls:

  • the computer resources allocation requested by the job
  • running the job
  • reporting the outcome of the user jobs

Running a Simple Job on Cluster

Simplest way to run a job on a cluster is to use srun command, which defaults to ask for one task, on a single core, on just one node.

$ srun hostname

Running in an Interactive Mode

Use the salloc command to run jobs in an interactive mode. Just like srun, salloc defaults to one node on the default queue.

$ salloc salloc: Granted job allocation 8 # run the hostname command $ srun hostname # run another command $ srun date Thu 22 Mar 09:34:01 GMT 2018 # run the hostname command $ srun hostname # Now exit the job and allocation $ exit exit salloc: Relinquishing job allocation 8

Create a Batch Job Script

A submission script is a shell script that describes the processing to be carry out and it requests resources such as number of CPUs, amount of memory etc for the processing job.

In our example below, a batch job script,, asks to run the program no more than 10:00 hours, with 10Mb memory per CPUs, two task per CPUs and run the task hundred time across two nodes.

#!/bin/bash #SBATCH --job-name=test #SBATCH --output=test-%j.out #SBATCH --ntasks=100 #SBATCH --time=10:00 #SBATCH --mem-per-cpu=10 #SBATCH --cpus-per-task=2 $ srun /home/davda/test/

The script starts with #!/bin/bash, denoting that it is a bash script. The lines starting with #SBATCH are directives that request resources from the Slurm controller. Note that all the batch directives must be at the top of a script, before any other commands. Any #SBATCH directives after a bash command are ignored.

The resource requests:

#SBATCH –job-name=test specifies name of the job allocation which appear along with job is number.
#SBATCH –output=test-j%.out specifies name of the output file with the job.
#SBATCH --ntasks=100 run the same task 100 times.
#SBATCH --time=10:00:00 sets maximum wall time for the job
#SBATCH –mem-per-cpu=10 sets minimum memory required per CPU in megabytes.
#SBATCH –cpus-per-task=2 advise the Slurm controller that job will requires 2 cpu per task.

How to Submit a Job

Use sbatch command to submit a job to a batch queue

$ sbatch
Submitted batch job 10

How to View the Jobs in the Queue

$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 10 compute test davda R 0:09 2 allxwn[01-02]

After the job finishes, the output will be written to the named output file with the job id: test-10.out

$ ls -l test-10.out -rw-r--r-- 1 davda Physics IT Support 1200 Mar 22 10:24 test-10.out

How to List Only Your Jobs in the Queue

$ squeue -u <user name>

Display the Pending Jobs Ordered by Decreasing Priority

$ squeue -t pd -S-p

Display Details about a Specific Job

$ scontrol show job <jobID>

$ scontrol show job 11 JobId=11 JobName=test UserId=davda(23027) GroupId=Physics IT Support(10016) MCS_label=N/A Priority=4294901750 Nice=0 Account=(null) QOS=(null) JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:04 TimeLimit=00:10:00 TimeMin=N/A SubmitTime=2018-03-22T10:42:07 EligibleTime=2018-03-22T10:42:07 StartTime=2018-03-22T10:42:08 EndTime=2018-03-22T10:52:08 Deadline=N/A PreemptTime=None SuspendTime=None SecsPreSuspend=0 LastSchedEval=2018-03-22T10:42:08 Partition=compute AllocNode:Sid=allxfs1:103719 ReqNodeList=(null) ExcNodeList=(null) NodeList=allxwn[01-02] BatchHost=allxwn01 NumNodes=2 NumCPUs=288 NumTasks=100 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=288,mem=2880M,node=2,billing=288 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=10M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 Gres=(null) Reservation=(null) OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null) Command=/home/davda/test/ WorkDir=/home/davda/test StdErr=/home/davda/test/test-11.out StdIn=/dev/null StdOut=/home/davda/test/test-11.out Power=

How to Cancel a Job

$ scancel <job_ID>

How to Send a Signal to a Running Job

$ scancel -s USR1 <job_ID>

Display the Worker Nodes Allocated to the Partitions

$ sinfo

$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST compute* up infinite 2 idle allxwn[01-02]

Useful Slurm Commands

A short description of useful Slurm commands

sacct Displays job accounting information about active or completed jobs salloc Allocate resources for a job in real time (typically used to allocate resources and spawn a shell, in which the srun command is used to launch parallel tasks) sbatch Submit a job script for later execution. The script can contain one or more srun commands to launch parallel tasks scancel Cancel a pending or running job sinfo Displays the state of partitions and nodes managed squeue Displays the state of jobs. By default, reports the running jobs in priority order followed by the pending jobs in priority order srun Submit a job in real time

Linux man pages have extensive help on Slurm commands.

Using Environment Modules

Environment Modules allows user to use the multiple versions of the software packages on the cluster. The variables can by dynamically loaded to use a particular software package, including finding all the software's dependencies.

To list all available modules on the cluster, run the command module avail.

$ module avail ------------------------------ /software/modules ------------------------------- gcc/7.3.0 openmpi/3.0.0 ------------------------ /usr/share/Modules/modulefiles ------------------------ dot module-git module-info modules null use.own ------------------------------- /etc/modulefiles ------------------------------- mpi/mpich-3.2-x86_64

To use a particular module, run module load modulename.

$ module load openmpi/3.0.0
$ module load gcc/7.3.0

To list what modules are loaded, run module list.

$ module list Currently Loaded Modulefiles: 1) openmpi/3.0.0 2) gcc/7.3.0 $ gcc --version gcc (GCC) 7.3.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

To unload a module, run module unload modulename.

$ module unload gcc openmpi $ module list No Modulefiles Currently Loaded.

Using Environment Modules with Slurm

You can use the module command in your Slurm submission job script, and it will automatically load the module when the job is run.

In the example below we are going to load openmpi under slurm and run it under batch.
(The mpi code is from

#include <mpi.h> #include <stdio.h> int main(int argc, char** argv) { // Initialize the MPI environment MPI_Init(NULL, NULL); // Get the number of processes. int world_size; MPI_Comm_size(MPI_COMM_WORLD, &world_size); // Get the rank of the process. int world_rank; MPI_Comm_rank(MPI_COMM_WORLD, &world_rank); // Get the name of the processor. char processor_name[MPI_MAX_PROCESSOR_NAME]; int name_len; MPI_Get_processor_name(processor_name, &name_len); //Print off a hello world message. printf("Hello world from processor %s, rank %d" " out of %d processors\n", processor_name, world_rank, world_size); // Finalize the MPI environment. MPI_Finalize(); }

Load openmpi

$ module load openmpi/3.0.0

Compile the code

$ mpic++ mpi.c -o hello_mpi

Create a batch job script to load the module and run the mpi code

#!/bin/bash #SBATCH --job-name=mpi_test #SBATCH --output=mpi_test-%j.out #SBATCH --ntasks=100 #SBATCH --time=10:00 #SBATCH --mem-per-cpu=10 #SBATCH --cpus-per-task=1 module load openmpi/3.0.0 mpirun /home/davda/mpi/hello_mpi

Submit the batch script

$ sbatch Submitted batch job 183

When the job finishes, the output will be written to mpi_test-183.out

$ less mpi_test-183.out

Hello world from processor, rank 4 out of 100 processors Hello world from processor, rank 12 out of 100 processors Hello world from processor, rank 36 out of 100 processors Hello world from processor, rank 8 out of 100 processors Hello world from processor, rank 20 out of 100 processors Hello world from processor, rank 31 out of 100 processors Hello world from processor, rank 39 out of 100 processors Hello world from processor, rank 47 out of 100 processors Hello world from processor, rank 0 out of 100 processors Hello world from processor, rank 6 out of 100 processors Hello world from processor, rank 10 out of 100 processors Hello world from processor, rank 13 out of 100 processors Hello world from processor, rank 19 out of 100 processors Hello world from processor, rank 38 out of 100 processors Hello world from processor, rank 43 out of 100 processors Hello world from processor, rank 74 out of 100 processors Hello world from processor, rank 2 out of 100 processors

Categories: Unix