For support, please email 'itsupport@physics.ox.ac.uk' which will create a ticket and will be seen by the whole team.
The Basics
The new Atomic and Laser Cluster is running CentOS 7 with approximately 70Tbytes of raw network-attached storage and is accessible through the "interactive" login node allxint1.physics.ox.ac.uk.
At present, the CentOS 7 cluster consists of one interactive node, allxfs1 file server (aka allxint1) and two worker nodes: allxwn01 and allxwn02. Slurm is the job scheduler for the new cluster.
Please note that interactive logins to the worker nodes is disabled for all users.
Hardware
File/Interactive Server: allxfs1 aka allxint1
Dell PowerEdge R730xd; 16 logical CPUs (Intel Xeon CPU E5-2623 v4 @ 2.60GHz); 64Gb Memory
Compute Nodes: allxwn01, allxwn02
Dell PowerEdge R940; 144 Logical CPUs (Intel Xeon Gold 6154 CPU @ 3.00GHz); 192Gb Memory
NFS Mounted Data Partitions
- /home
- /software
- /data/vinko
- /data/wark
- /data/gregori
- /data/common
Slurm Quick Start Guide
This is an introduction to the Slurm (Simple Linux Utility for Resource Management) job scheduler. See https://slurm.schedmd.com/ for more details.
Introduction
Jobs on the Atom and Laser Cluster run as batch jobs. In order to submit jobs to the cluster, log on to the interactive server, allxint1 (aka allxfs1). allxint1 is also the Slurm controller, which controls:
- the computer resources allocation requested by the job
- running the job
- reporting the outcome of the user jobs
Running a Simple Job on Cluster
Simplest way to run a job on a cluster is to use srun command, which defaults to ask for one task, on a single core, on just one node.
$ srun hostname
allxwn01.physics.ox.ac.uk
Running in an Interactive Mode
Use the salloc command to run jobs in an interactive mode. Just like srun, salloc defaults to one node on the default queue.
$ salloc
salloc: Granted job allocation 8
# run the hostname command
$ srun hostname
allxwn02.physics.ox.ac.uk
# run another command
$ srun date
Thu 22 Mar 09:34:01 GMT 2018
# run the hostname command
$ srun hostname
allxwn02.physics.ox.ac.uk
# Now exit the job and allocation
$ exit
exit
salloc: Relinquishing job allocation 8
Create a Batch Job Script
A submission script is a shell script that describes the processing to be carry out and it requests resources such as number of CPUs, amount of memory etc for the processing job.
In our example below, a batch job script, slurm_job_script.sh, asks to run the test.sh program no more than 10:00 hours, with 10Mb memory per CPUs, two task per CPUs and run the task hundred time across two nodes.
#!/bin/bash
#SBATCH --job-name=test
#SBATCH --output=test-%j.out
#SBATCH --ntasks=100
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=10
#SBATCH --cpus-per-task=2
$ srun /home/davda/test/test.sh
The script starts with #!/bin/bash, denoting that it is a bash script. The lines starting with #SBATCH are directives that request resources from the Slurm controller. Note that all the batch directives must be at the top of a script, before any other commands. Any #SBATCH directives after a bash command are ignored.
The resource requests:
#SBATCH –job-name=test specifies name of the job allocation which appear along with job is number.
#SBATCH –output=test-j%.out specifies name of the output file with the job.
#SBATCH --ntasks=100 run the same task 100 times.
#SBATCH --time=10:00:00 sets maximum wall time for the job
#SBATCH –mem-per-cpu=10 sets minimum memory required per CPU in megabytes.
#SBATCH –cpus-per-task=2 advise the Slurm controller that job will requires 2 cpu per task.
How to Submit a Job
Use sbatch command to submit a job to a batch queue
$ sbatch slurm_job_script.sh
Submitted batch job 10
How to View the Jobs in the Queue
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
10 compute test davda R 0:09 2 allxwn[01-02]
After the job finishes, the output will be written to the named output file with the job id: test-10.out
$ ls -l test-10.out
-rw-r--r-- 1 davda Physics IT Support 1200 Mar 22 10:24 test-10.out
How to List Only Your Jobs in the Queue
$ squeue -u <user name>
Display the Pending Jobs Ordered by Decreasing Priority
$ squeue -t pd -S-p
Display Details about a Specific Job
$ scontrol show job <jobID>
$ scontrol show job 11
JobId=11 JobName=test
UserId=davda(23027) GroupId=Physics IT Support(10016) MCS_label=N/A
Priority=4294901750 Nice=0 Account=(null) QOS=(null)
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:04 TimeLimit=00:10:00 TimeMin=N/A
SubmitTime=2018-03-22T10:42:07 EligibleTime=2018-03-22T10:42:07
StartTime=2018-03-22T10:42:08 EndTime=2018-03-22T10:52:08 Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
LastSchedEval=2018-03-22T10:42:08
Partition=compute AllocNode:Sid=allxfs1:103719
ReqNodeList=(null) ExcNodeList=(null)
NodeList=allxwn[01-02]
BatchHost=allxwn01
NumNodes=2 NumCPUs=288 NumTasks=100 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=288,mem=2880M,node=2,billing=288
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryCPU=10M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
Gres=(null) Reservation=(null)
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/home/davda/test/slurm_job_script.sh
WorkDir=/home/davda/test
StdErr=/home/davda/test/test-11.out
StdIn=/dev/null
StdOut=/home/davda/test/test-11.out
Power=
How to Cancel a Job
$ scancel <job_ID>
How to Send a Signal to a Running Job
$ scancel -s USR1 <job_ID>
Display the Worker Nodes Allocated to the Partitions
$ sinfo
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
compute* up infinite 2 idle allxwn[01-02]
Useful Slurm Commands
A short description of useful Slurm commands
sacct Displays job accounting information about active or completed jobs
salloc Allocate resources for a job in real time (typically used to
allocate resources and spawn a shell, in which the srun command
is used to launch parallel tasks)
sbatch Submit a job script for later execution. The script can contain one
or more srun commands to launch parallel tasks
scancel Cancel a pending or running job
sinfo Displays the state of partitions and nodes managed
squeue Displays the state of jobs. By default, reports the running jobs in
priority order followed by the pending jobs in priority order
srun Submit a job in real time
Linux man pages have extensive help on Slurm commands.
Using Environment Modules
Environment Modules allows user to use the multiple versions of the software packages on the cluster. The variables can by dynamically loaded to use a particular software package, including finding all the software's dependencies.
To list all available modules on the cluster, run the command module avail.
$ module avail
------------------------------ /software/modules -------------------------------
gcc/7.3.0 openmpi/3.0.0
------------------------ /usr/share/Modules/modulefiles ------------------------
dot module-git module-info modules null use.own
------------------------------- /etc/modulefiles -------------------------------
mpi/mpich-3.2-x86_64
To use a particular module, run module load modulename.
$ module load openmpi/3.0.0
$ module load gcc/7.3.0
To list what modules are loaded, run module list.
$ module list
Currently Loaded Modulefiles:
1) openmpi/3.0.0 2) gcc/7.3.0
$ gcc --version
gcc (GCC) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
To unload a module, run module unload modulename.
$ module unload gcc openmpi
$ module list
No Modulefiles Currently Loaded.
Using Environment Modules with Slurm
You can use the module command in your Slurm submission job script, and it will automatically load the module when the job is run.
In the example below we are going to load openmpi under slurm and run it under batch.
(The mpi code is from http://15418.courses.cs.cmu.edu/spring2013/article/22#fn:1)
#include <mpi.h>
#include <stdio.h>
int main(int argc, char** argv) {
// Initialize the MPI environment
MPI_Init(NULL, NULL);
// Get the number of processes.
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process.
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
// Get the name of the processor.
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
//Print off a hello world message.
printf("Hello world from processor %s, rank %d"
" out of %d processors\n",
processor_name, world_rank, world_size);
// Finalize the MPI environment.
MPI_Finalize();
}
Load openmpi
$ module load openmpi/3.0.0
Compile the code
$ mpic++ mpi.c -o hello_mpi
Create a batch job script to load the module and run the mpi code
#!/bin/bash
#SBATCH --job-name=mpi_test
#SBATCH --output=mpi_test-%j.out
#SBATCH --ntasks=100
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=10
#SBATCH --cpus-per-task=1
module load openmpi/3.0.0
mpirun /home/davda/mpi/hello_mpi
Submit the batch script
$ sbatch slurm_mpi_job_script.sh
Submitted batch job 183
When the job finishes, the output will be written to mpi_test-183.out
$ less mpi_test-183.out
Hello world from processor allxwn01.physics.ox.ac.uk, rank 4 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 12 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 36 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 8 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 20 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 31 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 39 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 47 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 0 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 6 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 10 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 13 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 19 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 38 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 43 out of 100 processors
Hello world from processor allxwn02.physics.ox.ac.uk, rank 74 out of 100 processors
Hello world from processor allxwn01.physics.ox.ac.uk, rank 2 out of 100 processors
