11.9. Binding memory
On a machine with more than one kind of memory, e.g
DDR next to HBM, the NUMA node on which a process is cpu-bound to
is not always the node with the best memory for it. Example is
Sapphire Rapids (SPR) with DDR+CPU nodes and HBM-only nodes. On SPR,
the local DDR+CPU node is also the default and closest node for
memory allocation. HBM nodes hold no cpu cores, so are silently ignored by
automatic page allocation and migration. Using HBM is usually done through
custom allocator libraries (e.g. memkind), by explicit C API and system
calls (hwloc, mbind(), set_mempolicy()) or with command-line
binding tools like numactl.
PRRTE can memory-bind each process to the NUMA node(s) best suited to it by
a chosen measure — capacity, locality, bandwidth or latency —
among the nodes local to the cores the process is bound to. This is separate from, and complementary to, cpu-binding
a process to cores (--bind-to). The cpu-binding defines where the process
runs, while the memory-binding decides where its memory is allocated.
Note
PRRTE currently implements these directives with hwloc’s memory attributes, which need hwloc 2.3 or later. PRRTE itself builds with hwloc 2.1 or later, but with an older hwloc this feature is unavailable: asking for a memory binding is reported as a memory-binding failure (see Failure handling).
11.9.1. Usage
Per job or per app, with the --membind option of mpirun,
prterun and prun:
mpirun --membind bandwidth -n 4 ./a.out
mpirun --map-by numa --membind bandwidth -n 4 ./a.out
mpirun --bind-to core --membind bandwidth -n 4 ./a.out
All three bind memory: the first relies on PRRTE’s default cpu binding,
the second on the binding --map-by derives, the third on an explicit
one. What matters is that each process ends up bound to cpus: see
The process must be bound to cpus.
Currently supported directives are: bandwidth, latency,
locality and capacity (and none, the default).
As the default for every job a DVM runs, use the
hwloc_default_membind MCA parameter, which takes the same
values. It is a PRRTE parameter, so it is given with --prtemca or in
the environment as PRTE_MCA_hwloc_default_membind —
PRTE_MCA_, not OMPI_MCA_:
prte --prtemca hwloc_default_membind bandwidth ...
A process is bound by its app’s --membind if it has one, else by its
job’s, else by the MCA parameter.
11.9.1.1. The --membind option
Bind the memory of each process to the NUMA node(s) best suited to it
by the chosen measure, among the nodes local to the CPUs the process is
bound to. This is separate from --bindto, which decides where a
process runs: --membind decides where its memory is allocated. It
needs the process to be bound to CPUs (see below).
Supported directives:
NONEbinds no memory. This is the default.CAPACITY: the local node with the most memoryLOCALITY: every node local to the process’ CPUsBANDWIDTH: the local node with the highest bandwidth from the process’ CPUs. This may be a node with no CPUs of its own — on a processor with on-package high-bandwidth memory (HBM), the HBM nodes hold no cores, so a process bound to a core sits on a DDR node andBANDWIDTHbinds its memory to the HBM node next to it.LATENCY: the local node with the lowest latency from the process’ CPUs
Nodes tied for the best value are all used.
Directives are case-insensitive and can be shortened to the minimum
number of characters to uniquely identify them. Thus, bandwidth,
BANDWIDTH and b are the same request. A shortening that fits
more than one of them is refused rather than guessed at — L
could be LOCALITY or LATENCY.
Memory is only bound for processes that are bound to CPUs, whether by
--bindto, by the binding --mapby derives, or by PRRTE’s
default. --bindto none leaves processes unbound, and so does an
oversubscribed node, on which PRRTE drops its default binding: PRRTE
then warns once per app and runs the job with that memory unbound. A
--membind takes precedence over the
hwloc_default_mem_alloc_policy MCA parameter, and a failure to bind
memory is handled according to
hwloc_default_mem_bind_failure_action. The directives are
currently implemented with hwloc’s memory attributes, which requires
hwloc 2.3 or later, and BANDWIDTH and LATENCY are only as good
as the values hwloc has for the platform.
The hwloc_default_membind MCA parameter takes the same
values, and supplies the memory binding for a job or app that gives no
--membind.
Per-app-context memory binding (MPMD jobs)
The rule is the one described for --bindto: a --membind written
on the first app segment and nowhere else describes the whole job;
otherwise each app that carries one is bound by its own, and apps that
carry none take the default — they do not inherit another app’s
directive.
Example:
prun --bindto core --membind bandwidth -n 4 app1 : --bindto core --membind capacity -n 2 app2
Here the memory of app1 processes is bound to the node with the
highest bandwidth from their core, and that of app2 processes to
the node with the most memory, all within the same job.
The same directives can be supplied via the PMIx_Spawn API with the
PMIX_MEMBIND key: in the job-level info[] array for the whole
job, or in the info[] array of a pmix_app_t for that app alone.
On that path there is no “first app” rule: the array a key was written
in says what it describes.
11.9.1.2. Where the memory goes
The candidates are the NUMA nodes local to the process’ cpuset: those whose locality contains it (the DDR node holding a core, and an HBM node sharing that locality) and those inside it (each node of a package the process is bound to). The choice is never made across the whole machine, so a process is not sent to a node on another socket because it is larger.
Candidates the PRRTE daemon may not itself use (for example because of a cgroup) are dropped before the choice is made, since the process inherits that restriction: if the best node is forbidden, the best permitted one is chosen. Nodes tied for the best value are all kept. If no candidate is left, or none has a value for the chosen measure, the process’ memory is not bound, which is handled as a failure (see Failure handling).
Note
BANDWIDTH and LATENCY are only as good as the values hwloc
has. On platforms that do not expose the HMAT information, or inside a
container that hides it, hwloc has nothing to compare and the directive
cannot be resolved - which is handled as a failure below.
11.9.2. Interaction with other parameters
11.9.2.1. The process must be bound to cpus
The memory binding is computed from the cpuset a process is bound to, so
it is applied to every process the mapper binds to cpus — whether
the binding policy was given with --bind-to, derived from --map-by
(--map-by numa binds to the NUMA domain, for example), or PRRTE’s
default — and to no other. A process is left unbound by
--bind-to none, and when PRRTE drops its default binding because a
node is oversubscribed.
When a memory binding is requested for processes that end up unbound, PRRTE prints one warning per app when it maps the job, naming how many of its processes are affected, and runs the job with their memory left unbound:
$ mpirun --bind-to none --membind bandwidth -n 4 ./a.out
A memory binding was requested for processes that are not bound to
cpus:
...
A process whose default cpu binding is refused by the operating system is tolerated as unbound (the default is a preference, not a requirement); its memory is then not bound either, without a warning of its own.
mem_alloc_policy. hwloc_default_mem_alloc_policy is PRRTE’s older,
coarser control: it either does nothing (none) or restricts a process
to its local NUMA domain (local_only). A memory binding requested by
--membind or hwloc_default_membind takes precedence, for
every process it applies to.
Topology files. A topology read from a file with hwloc_use_topo_file
describes a machine other than the one the process runs on, so no memory
binding is applied.
11.9.3. Failure handling
Failing to bind is governed by hwloc_default_mem_bind_failure_action,
the same parameter that governs local_only:
silent- proceed without comment.warn(default) - warn the first time, then let the job continue, possibly with degraded performance.error- abort the job.
This covers the directive being unavailable (an hwloc older than 2.3, or no
data for it), no usable local node, as well as the operating system
refusing the request. Many container runtimes deny the system call that
sets a memory policy; inside one, use silent or warn unless the job
must not run without it.
11.9.4. Seeing what was chosen
Raise the hwloc verbosity to see the node set PRRTE selected for each process:
prterun --prtemca hwloc_base_verbose 40 \
--bind-to core --membind bandwidth -n 2 ./a.out
and inspect the running process on its node: the mappings in
/proc/<pid>/numa_maps carry a bind: entry naming the node(s), and
numastat -p <pid> shows where its pages actually landed.