Skip to content
Contact Support

NUMA Domains

Inside a Mahuika node, some of the memory is closer to some of the cores than the rest. If a job's work and its data end up far apart, the job can run slower than it should, and its runtime can change from one run to the next.

Near and far memory

A node's cores do the calculations, and its memory holds the data they work on. A core has to fetch each number from memory before it can use it, and how long that takes depends on where the data is. A program that uses several cores runs one thread or MPI task on each (see Parallel Computing), and Slurm decides which cores and memory your job gets.

Picture a node as two buildings joined by a footbridge. Each building has four labs, and each lab has a few scientists and a fridge full of samples. The scientists are the cores, and the fridges are the memory.

A scientist who needs a sample gets it fastest from the fridge in their own lab. Fetching one from a lab down the corridor takes longer, and fetching one from the other building, across the footbridge, takes longest of all.

Two buildings joined by a footbridge, each with four labs. Every lab has a few scientists and a fridge. One scientist in lab 0 needs a sample. Path 1 goes to the fridge in their own lab and is the quickest. Path 2 goes to the fridge in lab 1, down the corridor, and is slower. Path 3 crosses the footbridge to the fridge in lab 4, in the other building, and is the slowest.

In a node, each lab is a NUMA domain: a group of cores along with the part of the memory closest to them. NUMA is short for non-uniform memory access, which means that some memory takes longer to reach than other memory.

NUMA domains on Mahuika

A real node is laid out the same way. The two buildings are the node's two sockets, each holding one processor chip, and the link between the sockets does the job of the footbridge.

A Genoa node laid out like the labs. Two sockets of 84 cores are joined by a socket link. Each socket is split into four NUMA domains, numbered 0 to 7, and each NUMA domain has 21 cores and its own memory.

Milan nodes are split the same way, with fewer cores in each NUMA domain.

A Milan node laid out the same way. Two sockets of 64 cores are joined by a socket link. Each socket is split into four NUMA domains, numbered 0 to 7, and each NUMA domain has 16 cores and its own memory.

The memory is shared out evenly between a node's NUMA domains. The table below lists every type of node. Hardware has more about each one.

PartitionGPUsCoresMemoryNUMA domainsCores per domainMemory per domain
genoa-168 (166)358 GB821 (19)45 GB
genoa2 x RTX PRO 6000168 (166)717 GB821 (20)90 GB
genoa-168 (166)1.4 TB821 (19)179 GB
genoa-168 (166)1.4 TB821 (20)179 GB
genoa2 x H100168 (166)1.4 TB821 (19)179 GB
genoa4 x L4168 (166)1.4 TB821 (19)179 GB
milan3 x A100 (a100_80gb)64 (62)480 GB416 (15)120 GB
milan4 x A100 (a100_80gb)64 (62)480 GB416 (15)120 GB
milan-128 (126)480 GB816 (15)60 GB
milan3 x A100 (a100_40gb)48 (46)480 GB86 (4)60 GB
milan-128 (126)992 GB816 (15)124 GB
hugemem-40 (38)1.4 TB220 (18)717 GB
hugemem-88 (86)5.8 TB422 (20)1.5 TB

Memory is what jobs can use, which is a little less than the memory installed, as some is kept back for the operating system. Slurm also keeps a few cores on each node for the system. They come out of one or two NUMA domains, which then have fewer cores for jobs. The numbers in brackets are the cores jobs can use: in the whole node, and in the NUMA domain with the fewest.

Your desktop and Mahuika

You've probably never had to think about NUMA domains on your own desktop or laptop. Most of them have one processor, and all of their memory is the same distance from every core.

A desktop processor and a Genoa node, drawn to the same scale. The desktop has one processor, about 4 cm across, with four memory slots on one side, all wired to the same part of the processor, so every core reaches the memory the same way and there is one NUMA domain. The Genoa node has two processors, each about 7.5 cm across and joined by a link, with six memory slots on each side of each processor. Each quarter of a Genoa processor, with its chips of cores and the memory slots wired to it, is one NUMA domain, so the node has 8.

Mahuika's nodes are different in a few ways that can make NUMA domains matter.

  • They're physically much bigger. A Genoa node has two large processors with 168 cores between them. The memory slots surround each processor, and each slot is wired to the part of the processor on its side, so the cores on that side reach it fastest. Memory attached to the other processor is further still, as data from it has to cross the link between the two processors. Every extra step on the way adds a little time.
  • You often share them. Unless your job uses a whole node, other people's jobs run on the rest of it. Slurm gives your job cores from whatever is free, and they can end up spread over several NUMA domains.
  • Jobs run for much longer, and their time is counted. If a job runs 20% slower than it could, a one-minute program on your laptop finishes 12 seconds later, but a two-day job on Mahuika finishes nearly ten hours later and uses more of your project's allocation.

When NUMA domains matter

Even on Mahuika, most jobs never need to think about NUMA domains. They make a difference for programs that run many threads or MPI tasks and move a lot of data in and out of memory. These jobs usually lose time in one of two ways.

Two pictures of one Genoa socket with 4 NUMA domains. In the first, marked good and fast, a job's cores are all in NUMA domain 0 and its data is in domain 0's memory, so every fetch is from its own domain. In the second, marked bad and slower, the job's cores are spread over domains 0, 1 and 3, and most of its data is in domain 0's memory, so the cores in domains 1 and 3 fetch from another domain.

The first is when the job's cores are spread over several NUMA domains. Linux stores each piece of data in the memory of the NUMA domain whose core first uses it, so the job's data ends up spread over the memory of several domains. Whenever a core needs data that's stored in another domain, the fetch takes longer. Slurm doesn't guarantee that a job's cores are all in one NUMA domain, even when they would fit in one.

Two pictures of one Genoa socket with 4 NUMA domains. In the first, marked good and fast, a thread in domain 0 stores its data in domain 0's memory, so every fetch is from its own domain. In the second, marked bad and slower, the operating system has moved the thread to domain 1, but its data is still in domain 0's memory, so every fetch now comes from another domain.

The second is when a thread moves. To keep all of your job's cores evenly busy, the operating system can move a thread from one core to another. If the new core is in another NUMA domain, the thread's data stays behind in the old domain's memory. From then on, every fetch of that data comes from another domain's memory.

If your job's runtime changes a lot between runs with the same input, one of these may be the reason.

Check a job's NUMA domains

To see which cores each task of your job is using, and which NUMA domains they're in, put this command in your batch script before the line that runs your program:

srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\""

Each task prints a report like this one, from a task on a Genoa node:

Task #0 is running on node c001.
c001 has the following NUMA configuration:
NUMA node(s):                            8
NUMA node0 CPU(s):                       0-20,168-188
NUMA node1 CPU(s):                       21-41,189-209
NUMA node2 CPU(s):                       42-62,210-230
NUMA node3 CPU(s):                       63-83,231-251
NUMA node4 CPU(s):                       84-104,252-272
NUMA node5 CPU(s):                       105-125,273-293
NUMA node6 CPU(s):                       126-146,294-314
NUMA node7 CPU(s):                       147-167,315-335
Task #0 has 4 CPUs, their core IDs are 8-11
===========================================

In this report, a CPU is a core, and a NUMA node is a NUMA domain, not a whole node.

  • The NUMA node lines list the cores in each NUMA domain. For example, NUMA domain 0 has cores 0 to 20. The numbers 168 to 188 are the same 21 cores again, as each core can run two threads at once (see Simultaneous Multithreading).
  • The last line lists the cores the task can use, here cores 8 to 11.

To find a task's NUMA domains, look for its cores in the NUMA node lines. Cores 8 to 11 are all in NUMA domain 0, so this task is in one NUMA domain. A task with cores 19 to 22 would be spread over two: cores 19 and 20 are in NUMA domain 0, and cores 21 and 22 are in NUMA domain 1.

Keep jobs in a NUMA domain

The example batch scripts below keep each task's work close to its data. What the options mean explains the options they use, and OpenMP Settings and Thread Placement and Thread Affinity go into more detail.

Sockets in Slurm

On Mahuika, Slurm treats each NUMA domain as a socket, so it shows a Genoa node as 8 sockets of 21 cores rather than 2 sockets. When a Slurm option or Slurm's output mentions sockets, it means NUMA domains.

A Genoa node has 8 NUMA domains of 21 cores. The GPUs are in NUMA domains 1 and 7 (RTX PRO 6000), 5 and 7 (H100), or 1, 3, 5 and 7 (L4).

#!/bin/bash -e

#SBATCH --job-name         numa-mpi
#SBATCH --account          nesi99991
#SBATCH --time             01:00:00
#SBATCH --partition        genoa
#SBATCH --ntasks           8
#SBATCH --cpus-per-task    1
#SBATCH --mem-per-cpu      2G
#SBATCH --cores-per-socket 21
#SBATCH --threads-per-core 1
#SBATCH --distribution     *:block:*
#SBATCH --mem-bind         local

export OMP_PROC_BIND=true
export OMP_PLACES=cores

srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\""

srun MY_PROGRAM
#!/bin/bash -e

#SBATCH --job-name         numa-mpi
#SBATCH --account          nesi99991
#SBATCH --time             01:00:00
#SBATCH --partition        genoa
#SBATCH --ntasks           8
#SBATCH --cpus-per-task    1
#SBATCH --mem-per-cpu      8G
#SBATCH --cores-per-socket 21
#SBATCH --threads-per-core 1
#SBATCH --distribution     *:block:*
#SBATCH --mem-bind         local

export OMP_PROC_BIND=true
export OMP_PLACES=cores

srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\""

srun MY_PROGRAM
#!/bin/bash -e

#SBATCH --job-name         numa-gpu
#SBATCH --account          nesi99991
#SBATCH --time             01:00:00
#SBATCH --gpus-per-node    pro_6000:1
#SBATCH --ntasks           8
#SBATCH --cpus-per-task    1
#SBATCH --mem-per-cpu      4G
#SBATCH --cores-per-socket 21
#SBATCH --threads-per-core 1
#SBATCH --distribution     *:block:*
#SBATCH --mem-bind         local
#SBATCH --gres-flags       enforce-binding
#SBATCH --gpu-bind         closest

export OMP_PROC_BIND=true
export OMP_PLACES=cores

srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\""

srun MY_PROGRAM
#!/bin/bash -e

#SBATCH --job-name         numa-gpu
#SBATCH --account          nesi99991
#SBATCH --time             01:00:00
#SBATCH --gpus-per-node    h100:1
#SBATCH --ntasks           8
#SBATCH --cpus-per-task    1
#SBATCH --mem-per-cpu      8G
#SBATCH --cores-per-socket 21
#SBATCH --threads-per-core 1
#SBATCH --distribution     *:block:*
#SBATCH --mem-bind         local
#SBATCH --gres-flags       enforce-binding
#SBATCH --gpu-bind         closest

export OMP_PROC_BIND=true
export OMP_PLACES=cores

srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\""

srun MY_PROGRAM
#!/bin/bash -e

#SBATCH --job-name         numa-gpu
#SBATCH --account          nesi99991
#SBATCH --time             01:00:00
#SBATCH --gpus-per-node    l4:1
#SBATCH --ntasks           8
#SBATCH --cpus-per-task    1
#SBATCH --mem-per-cpu      8G
#SBATCH --cores-per-socket 21
#SBATCH --threads-per-core 1
#SBATCH --distribution     *:block:*
#SBATCH --mem-bind         local
#SBATCH --gres-flags       enforce-binding
#SBATCH --gpu-bind         closest

export OMP_PROC_BIND=true
export OMP_PLACES=cores

srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\""

srun MY_PROGRAM

Milan compute and large memory nodes have 8 NUMA domains of 16 cores. The A100 80 GB nodes have one processor, split into 4 NUMA domains of 16 cores, with a GPU in each one. The A100 40 GB nodes have 8 NUMA domains of 6 cores, with GPUs in NUMA domains 2, 5 and 6. In the A100 batch scripts, --constraint picks the 80 GB or 40 GB nodes. The A100 40 GB batch script starts 6 tasks, as that's all a NUMA domain on those nodes can hold.

#!/bin/bash -e

#SBATCH --job-name         numa-mpi
#SBATCH --account          nesi99991
#SBATCH --time             01:00:00
#SBATCH --partition        milan
#SBATCH --ntasks           8
#SBATCH --cpus-per-task    1
#SBATCH --mem-per-cpu      4G
#SBATCH --cores-per-socket 16
#SBATCH --threads-per-core 1
#SBATCH --distribution     *:block:*
#SBATCH --mem-bind         local

export OMP_PROC_BIND=true
export OMP_PLACES=cores

srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\""

srun MY_PROGRAM
#!/bin/bash -e

#SBATCH --job-name         numa-mpi
#SBATCH --account          nesi99991
#SBATCH --time             01:00:00
#SBATCH --partition        milan
#SBATCH --ntasks           8
#SBATCH --cpus-per-task    1
#SBATCH --mem-per-cpu      8G
#SBATCH --cores-per-socket 16
#SBATCH --threads-per-core 1
#SBATCH --distribution     *:block:*
#SBATCH --mem-bind         local

export OMP_PROC_BIND=true
export OMP_PLACES=cores

srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\""

srun MY_PROGRAM
#!/bin/bash -e

#SBATCH --job-name         numa-gpu
#SBATCH --account          nesi99991
#SBATCH --time             01:00:00
#SBATCH --gpus-per-node    a100:1
#SBATCH --constraint       a100_80gb
#SBATCH --ntasks           8
#SBATCH --cpus-per-task    1
#SBATCH --mem-per-cpu      8G
#SBATCH --cores-per-socket 16
#SBATCH --threads-per-core 1
#SBATCH --distribution     *:block:*
#SBATCH --mem-bind         local
#SBATCH --gres-flags       enforce-binding
#SBATCH --gpu-bind         closest

export OMP_PROC_BIND=true
export OMP_PLACES=cores

srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\""

srun MY_PROGRAM
#!/bin/bash -e

#SBATCH --job-name         numa-gpu
#SBATCH --account          nesi99991
#SBATCH --time             01:00:00
#SBATCH --gpus-per-node    a100:1
#SBATCH --constraint       a100_40gb
#SBATCH --ntasks           6
#SBATCH --cpus-per-task    1
#SBATCH --mem-per-cpu      8G
#SBATCH --cores-per-socket 6
#SBATCH --threads-per-core 1
#SBATCH --distribution     *:block:*
#SBATCH --mem-bind         local
#SBATCH --gres-flags       enforce-binding
#SBATCH --gpu-bind         closest

export OMP_PROC_BIND=true
export OMP_PLACES=cores

srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\""

srun MY_PROGRAM

The hugemem nodes cl18 and cl19 have 2 NUMA domains of 20 cores.

#!/bin/bash -e

#SBATCH --job-name         numa-mpi
#SBATCH --account          nesi99991
#SBATCH --time             01:00:00
#SBATCH --partition        hugemem
#SBATCH --ntasks           8
#SBATCH --cpus-per-task    1
#SBATCH --mem-per-cpu      30G
#SBATCH --cores-per-socket 20
#SBATCH --threads-per-core 1
#SBATCH --distribution     *:block:*
#SBATCH --mem-bind         local

export OMP_PROC_BIND=true
export OMP_PLACES=cores

srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\""

srun MY_PROGRAM

What the options mean

These options and variables keep each task, its threads and its memory in one NUMA domain.

  • --cores-per-socket only lets the job run on nodes whose NUMA domains have at least this many cores. It only limits which nodes the job can use, and doesn't change how many cores it gets.
  • --threads-per-core 1 gives each task a core of its own, without Simultaneous Multithreading.
  • --distribution *:block:* puts tasks on cores next to each other, and fills one NUMA domain before starting on the next. Without it, Slurm puts each task in the next NUMA domain in turn.
  • --mem-bind local keeps each task's memory in its own NUMA domain.
  • OMP_PROC_BIND=true pins each thread, so the operating system can't move it to another core or NUMA domain.
  • OMP_PLACES=cores pins each thread to a whole core. These two only matter if you give each task more than one core with --cpus-per-task, as Slurm sets OMP_NUM_THREADS to that number.

The GPU batch scripts ask for one GPU. These options keep the tasks next to their GPU:

  • --gres-flags enforce-binding only gives the job cores in the NUMA domain its GPU is attached to. The job may wait longer to start.
  • --gpu-bind closest gives each task the GPU in its own NUMA domain.