History

Slurm » History » Version 54

« Previous - Version 54/136 (diff) - Next » - Current version
Sebastian Bocquet, 12/01/2015 02:37 PM

Hardware overview¶

You access the Euclid cluster through alexandria@usm.uni-muenchen.de

alexandria is the file server and should not be used for computing
There are 12 compute nodes named euclides1--euclides12
euclides8 hosts a virtual machine and is not available for computing
euclides12 is only available for debugging, see below
each node has 32 logical CPUs and 64GB of RAM

How to run jobs on the euclides nodes (using Slurm)¶

Use slurm to submit jobs or login to the euclides nodes (euclides1-12).

Please read through this entire wikipage so everyone can make efficient use of this cluster

alexandria¶

Please do not use alexandria as a compute node - it's hardware is different from the nodes. It hosts our file server and other services that are important to us.

You should use alexandria to

transfer files
compile your code
submit jobs to the nodes

If you need to debug and would like to login to a node, please start an interactive job to one of the nodes using slurm. For instructions see below.

euclides nodes¶

Job submission to the euclides nodes is handled by the slurm jobmanager (see http://slurm.schedmd.com and https://computing.llnl.gov/linux/slurm/).
Important: In order to run jobs, you need to be added to the slurm accounting system - please contact the admin

All slurm commands listed below have very helpful man pages (e.g. man slurm, man squeue, ...).

If you are already familiar with another jobmanager the following information may be helpful to you http://slurm.schedmd.com/rosetta.pdf‎.

Scheduling of Jobs¶

At this point there are two queues, called partitions in slurm:

normal which is the default partition your jobs will be sent to if you do not specify it otherwise. At this point there is a time limit of
two days. Jobs at this point can only run on 1 node.
debug which is meant for debugging, you can only run one job at a time, other jobs submitted will remain in the queue. Time limit is
12 hours.

The default memory per core used is 2GB, if you need more or less, please specify with the --mem or --mem-per-cpu option.

We have also set up a scheduler that goes beyond the first come first serve - some jobs will be favoured over others depending
on how much you or your group have been using euclides in the past 2 weeks, how long the job has been queued and how much
resources it will consume.

This is serves as a starting point, we may have to adjust parameters once the slurm jobmanager is used. Job scheduling is a complex
issue and we still need to build expertise and gain experience what are the user needs in our groups. Please feel free to speak out if
there is something that can be improved without creating an unfair disadvantage for other users.

You can run interactive jobs on both partitions.

Running an interactive job with slurm (a.k.a. logging in)¶

To run an interactive job with slurm in the default partition, use

srun -u --pty bash

If you want to use tcsh use

srun -u --pty tcsh

If you want to use a larger memory per job do

srun -u --mem-per-cpu=8000 --pty tcsh

In case you want to open x11 applications, use the --x11=first option, e.g.

srun --x11=first -u   --pty  bash

In case the 'normal' partition is overcrowded, to use the 'debug' partition, use:

srun --account cosmo_debug -p debug -u --pty bash # if you are part of the Cosmology group
srun --account euclid_debug -p debug -u --pty bash  # if you are part of the EuclidDM group

As soon as a slot is open, slurm will log you in to an interactive session on one of the nodes.

limited ssh access¶

If you have an active job (batch or interactive), you can login to the node the job is running on. Your ssh session will be killed if the job terminates. Your ssh session will be restricted to the same resources as your job (so you cannot accidentally bypass the job scheduler and harm other user's jobs).

Running a simple once core batch job with slurm using the default partition¶

To see what queues are available to you (called partitions in slurm), run:
```
sinfo
```

To run slurm, create a myjob.slurm containing the following information:

#!/bin/bash
#SBATCH --output=slurm.out
#SBATCH --error=slurm.err
#SBATCH --mail-user <put your email address here>
#SBATCH --mail-type=BEGIN
#SBATCH -p normal

/bin/hostname

To submit a batch job use:
```
sbatch myjob.slurm
```

To see the status of you job, use
```
squeue
```

To kill a job use:
```
scancel <jobid>
```
the <jobid> you can get from using squeue.

For some more information on your job use
```
scontrol show job <jobid>
```
the <jobid> you can get from using squeue.

Running a simple once core batch job with slurm using the debug partition¶

Change the partition to debug and add the appropriate account depending if you're part of
the euclid or cosmology group.

#!/bin/bash
#SBATCH --output=slurm.out
#SBATCH --error=slurm.err
#SBATCH --mail-user <put your email address here>
#SBATCH --mail-type=BEGIN
#SBATCH -p debug
#SBATCH -account [cosmo_debug/euclid_debug]

/bin/hostname

Accessing a node where a job is running or starting additional processes on a node¶

You can attach an srun command to an already existing job (batch or interactive). This
means you can start an interactive session on a node where a job of yours is running
or start an additional process.

First determine the jobid of the desired job using squeue, then use

srun  --jobid <jobid> [options] <executable>

Or more concrete

srun  --jobid <jobid> -u --pty  bash # to start an interactive session
srun  --jobid <jobid> ps -eaFAl  # to start get detailed process information

The processes will only run on cores that have been allocated to you. This works
for batch as well as interactive jobs.
Important: If the original job that was submitted is finished, any process
attached in this fashion will be killed.

Batch script for running a multi-core job¶

mpi is installed on alexandria.

To run a 4 core job for an executable compiled with mpi you can use

#!/bin/bash
#SBATCH --output=slurm.out
#SBATCH --error=slurm.err
#SBATCH --mail-user <put your email address here>
#SBATCH --mail-type=BEGIN
#SBATCH -n 4

mpirun <programname>

and it will automatically start on the number of nodes specified.

To ensure that the job is being executed on only one node, add

#SBATCH -n 4

to the job script.

If you would like to run a program that itself starts processes, you can use the
environment variable $SLURM_NPROCS that is automatically defined for slurm
jobs to explicitly pass the number of cores the program can run on.

To check if your job is acutally running on the specified number of cores, you can check
the PSR column of

ps -eaFAl
# or ps -eaFAl | egrep "<yourusername>|UID" if you just want to see your jobs

environment for jobs¶

By default, slurm does not initialize the environment (using .bashrc, .profile, .tcshrc, ...)

To use your usual system environment, add the following line in the submission script:

#SBATCH --get-user-env

Software specific setup¶

Python environment¶

You can use the python 2.7.3 installed on the euclides cluster by using

source /data2/users/ccsoft/etc/setup_all
source  /data2/users/ccsoft/etc/setup_python2.7.3

Notes For Euclid users¶

For those submitting jobs to euclides* nodes through Cosmo DM pipeline here are some things which need to be specified for customized job submissions,
since a different interface to slurm is used.

To use larger memory per block , specify max_memory = 6000 (for 6G) and so on. inside block definition or in the submit file (in
case you want to use it for all blocks)

If you want to run on multiple cores/cores then use
nodes='<number of nodes>:ppn=<number of cores> inside the block definition of a particular block or in the submit file in case you want
to use it for all blocks.

If you want to use a larger wall time then specify wall_mod=<wall time in minutes> inside the module definition

note that queue=serial does not work on alexandria(we usually use it for c2pap)

Admin¶

There is a user "slurm" which however is not really necessary for the administration work. The slurm administrator needs sudo access. Some script for adding a user and similar things are in "/data1/users/slurm". With the sudo access the admin can execute those scripts. In the mysql database there is the username "slurmdb" with password.

Overview over users, accounts, etc.¶

No sudo access needed:

/usr/local/bin/sacctmgr show account withassoc

Adding a new user¶

As root on alexandria,

cd /data1/users/slurm/
./add_user.sh UserName account(cosmo or debug)
/usr/local/bin/.scontrol reconfigure

To increase memory, cores etc for a user¶

Inside script above, various commands for changing user settings, e.g.

/usr/local/bin/sacctmgr -i modify user  name=$1 set GrpCPUs=32
/usr/local/bin/sacctmgr -i modify user  name=$1 set GrpMem=128000

Node state "drain"¶

When a node is in "drain" state when calling

sinfo

run

/usr/local/bin/scontrol update nodename=NODE_NAME state=resume

to put it back to operation.

Nodes down¶

Sometimes nodes are reported as "down". This seems to happen as a result of network problems. Here is some troubleshooting for this situation. Also after a re-boot of alexandria some manual work on slurm might be necessary to get going again.

Project

General

Profile

Cluster Cosmology

Wiki