9.2.3. Root complexes: device groups and companion NICs

Two extensions to --map-by device=, both prompted by Open MPI’s btl/ofi NIC selection rework (open-mpi/ompi#14800). That PR measured, on EFA, what happens when one process drives NICs under more than one PCI root complex:

NICs per process (64 KB RDMA reads, 64 outstanding)

Bandwidth

Retransmitted

2, one root complex

22379 MB/s

0.00%

2, two root complexes

1940 MB/s

32.95%

4, one root complex

23881 MB/s

0.00%

8, two root complexes

859 MB/s

36.09%

Its response was to group the node’s NICs by root complex and hand each process NICs from one complex only. This document covers what PRRTE should take from that:

  • Part 1 — an ndev group should not span root complexes when it can be avoided. A correction to existing behavior.

  • Part 2 — a process mapped by GPU should be able to be given the NICs under its GPU’s root complex. A new qualifier.

  • Part 3 — PMIx pnet support for the fabrics whose NIC assignments are currently delivered but acted on by nothing: RoCE on NVIDIA, Broadcom and Intel adapters, EFA, and Slingshot. A PMIx change, independent of the other two.

Read it alongside Mapping and binding by device and Device mapping: implementation plan, and src/mca/rmaps/base/AGENTS.md, whose rules are assumed here.

Note

The numbers above are from one fabric. Nobody has measured the effect on InfiniBand, RoCE or Slingshot, so nothing here refuses a placement on the strength of it; both parts are preferences. For GPUs with ndev=2 there is an independent reason in the same direction: GPU peer-to-peer traffic between two root complexes crosses the CPU interconnect, or is not offered at all.

9.2.3.1. What a root complex is, here

hwloc represents each PCI root bus as a Bridge object whose upstream side is the host (bridge_type="0-1" in the XML, upstream type HWLOC_OBJ_BRIDGE_HOST in the API). Every device under that bridge shares a root complex. This is the definition the OMPI PR uses (it climbs from the device’s PCI function to the last bridge before a non-I/O parent), and on the platforms in hand it is one host bridge per root complex.

The key is reachable today without a PMIx change: every pmix_hwloc_device_t carries obj, the device’s position in the I/O tree, borrowed from the same topology as locality. Walking up from obj while the parent is a Bridge or PCIDev ends at the host bridge. A device with no PCI ancestor has no host bridge and is treated as a complex of its own.

The CPU-side locality PRRTE already computes cannot see this. Two host bridges routinely share a cpuset: in test/topologies/test-topo.xml ib0 and eth1 sit under different host bridges with identical cpusets, so a group holding both passes binding_fits() and the degenerate-locality check and nothing reports it. So the key has to come from the I/O tree, not from locality.

9.2.3.2. Part 1: keep an ndev group under one root complex

9.2.3.2.1. Current behavior

prte_rmaps_base_devices_begin() forms group n from devices n*per .. n*per+per-1 of PMIx’s list, which is in PCI bus id order. Bus order keeps a root complex’s devices contiguous, but nothing aligns group boundaries to the runs. test/topologies/turin-4gpu.xml:

Bus id

Device

Host bridge (document order)

0000:03:00.0

mlx5_0

HB3 (with GPU renderD128)

0000:13:00.0

mlx5_3

HB5 (with GPU renderD129)

0000:51:00.0

eno6

HB1

0000:61:00.0

mlx5_1

HB8

0000:61:00.1

mlx5_2

HB8

0000:93:00.0

mlx5_4

HB13 (with GPU card3)

0000:e3:00.0

mlx5_5

HB16 (with GPU card4)

--map-by device=nic:ndev=2 gives {mlx5_0, mlx5_3}, {eno6, mlx5_1}, {mlx5_2, mlx5_4}. All three groups span two root complexes, and the one pair on the machine that shares a complex (the two ports of the card at 61:00) is split between two processes.

9.2.3.2.2. The rule

When ndev > 1 and interleave was not given:

  1. Partition the device list by host bridge, keeping bus order within each partition and ordering partitions by their first device.

  2. Cut each partition into whole groups of per, in order.

  3. Pool what is left over from every partition, in bus order, and cut that pool into groups of per. These are the groups that have to span complexes, and they come after every single-complex group, so a job that does not need them never gets them.

Turin, device=nic:ndev=2: {mlx5_1, mlx5_2}, then {mlx5_0, mlx5_3}, {eno6, mlx5_4}; mlx5_5 is left over, as it is today.

The group count does not change. With partitions of sizes s1 .. sk and p devices per group, the single-complex groups number sum(floor(si / p)) and the pool holds L = sum(si mod p) devices, so the total is sum(floor(si / p)) + floor(L / p) = floor(N / p) - which is what prte_rmaps_base_devices_total() already computes. The pre-count stays as it is, and no job that fits today is refused for want of devices.

Where nothing can be kept together, nothing changes. When no complex holds per devices, every device goes to the pool, and the pool in bus order is today’s list. This is the case for the GPUs on both Turin topologies (one GPU per complex), so the existing ndev golden maps and the ndev=2 worked example in cli-map-by.rst are unaffected.

9.2.3.2.3. Why interleave turns it off

ndev=2:interleave is documented to give each process one device from each package: the user is asking for groups that span. Applying the complex rule first and then interleaving would answer neither request. With ndev of one the rule has nothing to do, so the only combination where the two meet is the one where the user has said what they want.

9.2.3.2.4. Interaction with the binding check — a behavior change to accept

A group’s locality is the common ancestor of its members’ localities. A single-complex group usually has a finer locality than the spanning group it replaces. binding_fits() refuses a binding coarser than a group’s locality, so a --bind-to that every spanning group admitted can be refused for a single-complex one. On Turin, device=nic:ndev=2 --bind-to package would now be refused for {mlx5_1, mlx5_2}, whose locality is one 16-core chunk rather than a package.

This is the same answer ndev=1 already gives for those two devices, and the old acceptance depended on the group spanning, which is the defect being fixed. So the recommendation is to accept it and document it, not to loosen the check. It is the one place Part 1 can make a command that worked stop working, so the release note should name it.

9.2.3.2.5. Where the code goes

  • src/hwloc/: prte_hwloc_base_host_bridge(hwloc_obj_t obj), returning the host bridge above an I/O object or NULL. It belongs beside the other topology helpers rather than in rmaps, since Part 2 and anything else that asks “which complex” needs the same answer.

  • rmaps_base_devices.c: a group_by_complex() step in devices_begin() between the interleave reorder and the grouping loop. It reorders dc->devs the way interleave_devices() does — shallow copy and memcpy back, each entry moved exactly once, so string ownership moves with it — and the existing grouping loop then runs unchanged. Allocation failure leaves the list in bus order, which is a valid (today’s) answer, matching interleave_devices().

  • No change to prte_rmaps_options_t, the module struct, attributes or the wire, so no rmaps version bump and no PMIx change.

9.2.3.2.6. Tests

  • test/unit/rmaps/test_devices.c, on turin-4gpu.xml: device=nic:ndev=2 yields the three groups above in that order, mlx5_5 unassigned; device=gpu:ndev=2 yields bus order unchanged; device=nic:ndev=2:interleave yields today’s interleaved order; on test-topo.xml, two NICs under two host bridges with one cpuset yield one pooled group.

  • test/offline: a golden for --map-by device=nic:ndev=2 on turin-4gpu-nvml, plus a case for the refused --bind-to package. Every existing golden must be byte-identical.

9.2.3.2.7. Docs

  • cli-map-by.rst, NDEV: replace “a group is a contiguous run of that order” with the rule above, and the binding consequence.

  • src/mca/rmaps/base/AGENTS.md: the rule, why interleave turns it off, and that the key comes from obj rather than locality.

9.2.3.3. Part 2: give a GPU-mapped process the NICs on its GPU’s root complex

9.2.3.3.1. The problem

--map-by device=gpu tells each process which GPU it has. Nothing tells it which NIC to use with that GPU, so each communication library guesses. The OMPI PR guesses by assuming gpu = local_rank * ngpus / local_size and assigns contiguous blocks of local ranks to each root complex so that the guess comes out right. That guess is correct only for one placement. PRRTE does not need to guess: it placed the GPU.

On turin-4gpu.xml every GPU shares a host bridge with exactly one InfiniBand HCA (renderD128–mlx5_0, renderD129–mlx5_3, card3–mlx5_4, card4–mlx5_5), which is the common shape on GPU nodes and is already in the tree as test material.

9.2.3.3.2. What already works, verified against PMIx master

The delivery path does not need to change. PRTE_PROC_DEVICE_ID is already an array of pmix_device_t delivered as PMIX_DEVICE_ID, and both PMIx consumers already select their own devices out of a mixed array:

  • pgpu (pmix_pgpu_base_get_visible_devices()) keeps only devices whose vendor matches the component’s, which a NIC never has.

  • pnet (pmix_pnet_base_get_assigned_devices()) keeps only devices whose PCI (vendor, class) matches the component’s list, which a GPU’s class 03xx never does.

So adding NICs to a GPU process’s array sets CUDA_VISIBLE_DEVICES from the GPUs and NCCL_IB_HCA / UCX_NET_DEVICES from the NICs, with no PMIx change. Part 2 is a PRRTE-only change.

Two gaps in what acts on the assignment:

  • pnet/nvd matches class 0x207 (InfiniBand) only, so a RoCE adapter (class 0x200) gets PMIX_DEVICE_ID but no variable, and EFA and Slingshot have no pnet component at all. Part 3 closes this.

  • btl/ofi chooses NICs from the local rank alone and reads no assignment from PMIx. On EFA Part 3 makes that moot: FI_EFA_IFACE is applied inside libfabric’s provider, so btl/ofi and mtl/ofi are only ever shown the assigned NICs. Elsewhere on the ofi path it remains a suggestion for that project.

9.2.3.3.3. Proposed design

--map-by device=gpu:nic[=N] — valid only when the device class is GPU (by class or by naming one GPU), refused by name otherwise. It does not change placement or binding: the process is placed and bound against its GPU group exactly as without it. It only adds NICs to what prte_rmaps_base_devices_record() publishes.

Per node, in devices_begin() when the qualifier is present, enumerate the NIC class (PMIX_DEVTYPE_NETWORK | PMIX_DEVTYPE_OPENFABRICS) a second time and keep it on the context. For each GPU group, the candidate NICs are those under the host bridge of any of the group’s GPUs. Within the candidates, a PCI function carrying an OpenFabrics device ranks ahead of one that does not, then bus order.

When several groups share a complex (two GPUs under one host bridge), the complex’s candidates are dealt so that the groups get distinct NICs while there are enough and share once there are not. That is the OMPI PR’s division of a complex’s NICs among the processes on it, done with the real assignment rather than a guess. Processes that share one GPU (shared) share its NICs: the choice is made per group, which is what devices_record() publishes.

Both mappers go through devices_begin() / devices_record(), but ppr refuses device qualifiers in its parser (ndev included), so in practice the qualifier applies to --map-by device=.

The qualifier is a new field at the end of prte_rmaps_options_t, with a new PRTE_JOB_MAP_NIC / PRTE_APP_MAP_NIC pair at the next free offsets, parsed by both --map-by parsers and added to prte_cli_mapquals. Only the base reads the field, and it is appended, so the rmaps interface version does not change - the same reasoning under which ndev, shared and interleave were added.

9.2.3.3.4. Decisions

Settled before any code was written:

  1. Spelling: ``:nic[=N]`` on the existing directive. nic is already the class word for these devices. Refused by name for a non-GPU class. n and nd stop being unique abbreviations, which the vocabulary matcher already reports as ambiguous.

  2. Bare ``nic`` is one per process. It is predictable and fits the common one-HCA-per-GPU shape; nic=N asks for more. The OMPI PR’s “even share of the complex” rule is not the default.

  3. No NIC under the GPU’s complex: assign none, and say so through show_help so it aggregates. The OMPI PR does not address this case: it groups only the complexes that have NICs and hands every process one of them, never consulting where its GPU is, so a process whose GPU’s complex carries no NIC is given a NIC from another one. Naming a far NIC in UCX_NET_DEVICES overrides a library that would otherwise have chosen sensibly, so for PRRTE a bad assignment is worse than none. The process keeps its GPU assignment either way.

    Clusters are mostly homogeneous, so a node that trips this usually means every node does, and a message per node would be thousands of lines. devices_begin() therefore calls prte_show_help(PRTE_JOB_NSPACE(jdata), "help-prte-rmaps-base.txt", "rmaps:no-nic-near-gpu", ...) once per node that has the problem. PMIx suppresses duplicates by the (nspace, help file, topic) tuple, and a call on the HNP goes through that path (verified in src/util/pmix_show_help.c), so the user sees the first node’s message in full and then one “N more processes have sent help message help-prte-rmaps-base.txt / rmaps:no-nic-near-gpu (job …)” line. This is the pattern rmaps:degenerate-device-locality already follows. The topic’s text names the first node and says the same may hold for other nodes, since the summary line counts calls rather than naming nodes.

  4. Ethernet is a candidate only where nothing RDMA-capable shares the complex. As first written it was a candidate ranked last, which made nic=N beyond the complex’s RDMA NICs pad the assignment with Ethernet ports: every GPU of an AWS p5 given nic=8 got its four EFA devices and four ENA ports, and pnet names no ENA port to any library. Where a complex carries an OpenFabrics device, only those are candidates. A complex with none still offers its other NICs, since a Slingshot NIC is not an OpenFabrics device either and that is where the link-speed rule below does its work. (On a RoCE node the Ethernet view of a port is the same PCI function as its OpenFabrics device, so it is a candidate as an OpenFabrics NIC.)

  5. Fewer candidates than ``N`` refuses the map. It used to give the group what the complex had. A count the user chose and did not get is a different job from the one asked for, so devices_begin() refuses the node with rmaps:too-few-nics-near-gpu, naming the smallest count any group on it could be given - the largest N that would map. A group with no NIC at all stays under decision 3.

9.2.3.3.5. Ranking within a complex

The candidates for a GPU (decision 4) are ordered by:

  1. the depth of the deepest common ancestor of the NIC and the GPU in the I/O tree, deepest first — a NIC behind the same PCIe switch as the GPU ahead of one that only shares the host bridge. This comes from the review of the OMPI PR, which named it as the next step for nodes such as p5en where GPUs and NICs share switches below the root complex;

  2. the faster PCIe link. Added after the user’s HPE Cray EX235n topology (test/topologies/slingshot.xml) showed a 1 GbE management port and the Slingshot NIC under the same host bridge as an A100, neither an OpenFabrics device and both one root port down: without it, only bus order kept the management port off the GPU;

  3. how many groups already hold the NIC, fewest first;

  4. bus order.

With ndev greater than one, a group’s candidates are the union over its GPUs’ complexes, and depth is measured against the nearest of the group’s GPUs.

9.2.3.4. Part 3: pnet support for RoCE, EFA and Slingshot

Part 2 makes NIC assignments common, and an assignment nothing acts on is only reportable. The question is whether the topology carries enough to name each device to the software that will use it, without guessing. Verified against libfabric and hwloc master (November 2026):

Fabric

Variable

What it accepts

What hwloc gives us

RoCE (NVIDIA / Mellanox)

NCCL_IB_HCA, UCX_NET_DEVICES

The verbs device name, exactly as for InfiniBand (mlx5_1, mlx5_1:*)

An OpenFabrics OS device on a class 0x200 function of vendor 0x15b3 / 0x10de. Same selector as InfiniBand.

RoCE (Broadcom)

NCCL_IB_HCA, UCX_NET_DEVICES

The verbs device name (bnxt_re0, or rocep6s0f0 where rdma-core’s PCI-path naming is in use); both libraries drive any IB/RoCE verbs device, not only NVIDIA’s

An OpenFabrics OS device on a class 0x200 function of vendor 0x14e4. bnxt_re is RoCE only.

RoCE or iWARP (Intel E810, X722)

NCCL_IB_HCA, UCX_NET_DEVICES

The verbs device name (irdma0) — but only in RoCE mode; see below

An OpenFabrics OS device on a class 0x200 function of vendor 0x8086. Nothing that says which protocol it runs.

Any of the verbs devices above, through libfabric

FI_VERBS_DEVICE_NAME

A comma-separated list of prefixes: each entry is compared with strncasecmp over its own length (vrb_get_matching_info() in prov/verbs/src/verbs_info.c)

The verbs device names of the node, from which a colliding prefix can be detected

EFA (Amazon)

FI_EFA_IFACE

A comma-separated list of verbs device names, matched exactly (efa_env_allows_nic() in prov/efa/src/efa_env.h, compared against ibv_device->name)

EFA registers under /sys/class/infiniband, so hwloc reports it as an OpenFabrics OS device whose name is the verbs device name (rdmap16s27). Vendor 0x1d0f.

Slingshot (HPE)

FI_CXI_DEVICE_NAME

A comma-separated list of cxiN, validated by the provider (cxip_env_validate_device_name())

hwloc 2.14.0 and later add a network OS device named cxiN from /sys/class/cxi, beside the hsnN interface on the same PCI function. Older hwloc has only hsnN.

So yes for all three, with two things to settle in PMIx first.

A match needs a device type as well as PCI ids. pmix_pnet_pcimatch_t is (vendor, devclass). That cannot tell an RDMA function from a plain Ethernet one of the same vendor and class: an NVIDIA ConnectX port with RDMA disabled, or Amazon’s ENA beside its EFA (both vendor 0x1d0f). Writing an interface name like eth0 into NCCL_IB_HCA or FI_EFA_IFACE is the “another fabric’s device in your variable” case pnet exists to prevent. Add a pmix_device_type_t types field (zero meaning any) that the device’s type must intersect; nvd’s RoCE entries and efa require PMIX_DEVTYPE_OPENFABRICS. With the type doing that work, efa can leave devclass at zero and need not depend on EFA’s PCI class. This changes a struct the components are built against, so pnet’s interface version bumps.

A Slingshot function has two network OS devices, and the enumerator picks one by iteration order. osdev_preferred() already takes the OpenFabrics device over the interface because ib0 is accepted by no fabric variable. hsn0 versus cxi0 is the same problem, and the provider rejects anything not of the form cxiN, so a wrong choice is not a silent narrowing but a provider that fails to initialize. Prefer the cxi device. It is identified by what hwloc records only for it — the CassiniVersion / NID infos from /sys/class/cxi — rather than by its name. Under hwloc older than 2.14 there is no cxi device, the selector is hsnN, and pnet/cxi must set nothing rather than derive cxi0 from hsn0: the two usually correspond, but nothing guarantees it, and the plan’s rule against guessing an ordinal applies.

Intel’s ``irdma`` may be iWARP, and nothing in the topology says so. An E810 runs RoCEv2 or iWARP depending on how the driver was loaded, and an X722 runs iWARP only. Neither UCX nor NCCL’s IB transport drives iWARP. NCCL then falls back to sockets, but a UCX_NET_DEVICES naming only an iWARP device leaves UCX no device for any transport, and it fails rather than falls back. The kernel says which it is in /sys/class/infiniband/<dev>/node_type (1: CA versus 4: RNIC). hwloc records the GUIDs, port states, LIDs and GIDs, but not that file, so the topology the HNP holds cannot tell them apart.

setup_fork runs on the node that forks the process, so the component can read that one file for each device it is about to name, on the machine it describes. An RNIC is left out of UCX_NET_DEVICES and NCCL_IB_HCA. A file that cannot be read means the device is left out too: on Linux the file always exists, and elsewhere none of these devices does. The long-term fix is for hwloc to record the node type as an info attribute, which would let the HNP know it as well. Propose that upstream; do not wait for it.

``FI_VERBS_DEVICE_NAME`` cannot be written exactly. The provider treats each entry as a prefix, so mlx5_1 also selects mlx5_10 through mlx5_19 on a node with that many devices. No spelling of the entry anchors it. The component therefore writes it only when no other OpenFabrics device on the local node has an assigned name as a proper prefix, and otherwise leaves it unset and says why at verbosity. On the common node, with fewer than ten devices of one driver, it is always written.

One component owns each variable. Two components writing UCX_NET_DEVICES for one process would overwrite each other, and whichever ran last would win. So the IB/RoCE vendors are entries in one component’s table, not one component each. That component is nvd today. Its name says NVIDIA, which stops being true; renaming it is cosmetic and can be done separately.

9.2.3.4.1. The components

  • nvd (the IB/RoCE verbs component): add {0x15b3, 0x200, OPENFABRICS}, {0x10de, 0x200, OPENFABRICS}, {0x14e4, 0x200, OPENFABRICS} and {0x8086, 0x200, OPENFABRICS} to mymatch. The class keeps Omni-Path’s 0x208 with opa. setup_fork drops any device whose node_type is not CA, then writes NCCL_IB_HCA and UCX_NET_DEVICES as now, and FI_VERBS_DEVICE_NAME under the prefix rule.

  • efa (new): match {0x1d0f, 0, OPENFABRICS}; write the joined selectors to FI_EFA_IFACE. component_open gates on the vendor, as nvd and opa do.

  • cxi (new): match the HPE/Cray vendor id with NETWORK; write FI_CXI_DEVICE_NAME only when every selector is of the form cxiN, and say once at verbosity why it declined otherwise.

Like nvd, the components overwrite a variable that is already set. A process mapped against a device made the more specific request.

Not needed: the variables take device names, which the topology has. No ordinals are involved, so nothing here repeats the HFI_UNIT problem.

9.2.3.4.2. Still to confirm before landing

  • Test material. Neither this tree nor hwloc’s test suite carries a topology with an EFA or a Slingshot device. The unit tests can use hand-edited copies of turin-4gpu.xml (vendor ids and OS devices changed), but those prove only that PMIx reads what it was written to read. A real lstopo --of xml from a p5 / p5en instance and from a Slingshot node should be obtained and checked in before the components are called done.

  • Slingshot’s PCI vendor id. The Cassini parts are expected to report Cray’s 0x17db. Confirm it from a real topology.

  • Topologies for Broadcom and Intel adapters. Same need as EFA and Slingshot. An Intel one should say which mode its irdma was in.

  • Scrubbing. A topology XML carries the host name, MAC addresses, GUIDs and, for Slingshot, the NID. Replace them before anything is checked in.

9.2.3.4.3. Part 3 as landed

The user’s HPE Cray EX235n topology (test/topologies/slingshot.xml, read by hwloc 2.10) settled several of the items above and changed three decisions.

  • Slingshot’s vendor id is confirmed: Cassini reports [17db:0501], class 0x0200.

  • pnet/cxi reads sysfs when the topology has no cxi device. hwloc 2.10 records only hsnN; the node itself names the matching device under /sys/class/net/<hsnN>/device/cxi/ - which is where hwloc 2.14 finds it too. Exactly one cxiN entry there is used; none, several or an unreadable directory declines. The name is never derived from the hsn number, so the rule against guessing still holds. The root is the pnet_cxi_sysfs_root parameter.

  • The node-type check is Intel’s only. Only irdma can be iWARP, so only Intel’s adapters are asked. Asking NVIDIA’s and Broadcom’s made their naming depend on a file that changes nothing about them, and dropped them wherever it was absent - a topology loaded on a machine without the hardware, which is how this path is tested.

  • A 3D controller is a GPU without OS devices. The node’s A100s (class 0x0302) carry none: that hwloc had no vendor backend and the node no render nodes, so --map-by device=gpu said there was no GPU. PMIx now reports such a function as a GPU named by its PCI bus id, with no vendor identity, so PRRTE refuses it as unnameable instead - which names the fix.

A Broadcom node (test/topologies/mi350-bcm57608.xml, read by hwloc 2.15) confirmed the RoCE rule on real hardware and found one defect elsewhere.

  • Broadcom RoCE is named as planned. Each BCM57608 port ([14e4:1760], class 0x0200) carries an OpenFabrics device, and a process mapped against it gets NCCL_IB_HCA, UCX_NET_DEVICES and FI_VERBS_DEVICE_NAME. The names are rdma-core’s PCI-path names (rocep6s0f0), not bnxt_reN; nothing here depends on the name, and those names cannot prefix one another, so FI_VERBS_DEVICE_NAME is always written. A BCM57508 on the same node with no OpenFabrics device is plain Ethernet and is named to nothing.

  • The grouping and pairing hold. Each host bridge holds one MI350 and one dual-port adapter: device=gpu:nic gives each GPU port 0, nic=2 both ports, and device=nic:ndev=2 keeps each card’s ports together.

  • PMIx’s ``pgpu/amd`` did not open on this node. It asked only for an AMD function in the display class 0x03, and an MI350 reports 0x1200, a processing accelerator, so no process was given ROCR_VISIBLE_DEVICES. PRRTE’s map was unaffected: the device enumeration counts the GPUs by their RSMI devices. The fix is in PMIx (openpmix/openpmix#4442). This is a separate PMIx change, and PRRTE’s offline harness mirrors it in a PRRTE change that lands after it. Class 0x0380 is left alone; those parts get render nodes from their drivers.

  • opa named HFIs to PSM3, which is Intel’s library for its ethernet adapters rather than an Omni-Path one. PMIx no longer sets PSM3_NIC at all; opa names an HFI to libfabric’s OPX provider in FI_OPX_HFI_SELECT instead, and opens for CN5000’s Cornelis vendor id (0x434e) as well as OPA100’s 0x8086.

Separately, and not part of this work: hwloc master (3.0, unreleased) turns the OS device type attribute into a types bitmask. PMIx’s pmix_hwloc.c reads attr->osdev.type throughout and has no HWLOC_API_VERSION guard for it, so it will need adapting before it can build against hwloc 3.0.

9.2.3.5. Build order

Part 1 first and alone: it is a correction, it touches only rmaps_base_devices.c and a new src/hwloc helper, and its proof is that every existing golden stays byte-identical while the new ones pin the change. Part 2 builds on the helper Part 1 adds. Parts 1 and 2 are one PRRTE PR each, and neither needs PMIx.

Part 3 is an openpmix PR and does not depend on the other two: it acts on any NIC assignment, including today’s --map-by device=nic. It can land in parallel, and should land before Part 2 is announced, so that the first users of :nic on EFA or Slingshot get a variable and not just a key.

Verification for both follows Device mapping: implementation plan’s checklist: warning-free --enable-debug build from the top of an out-of-tree build, make check, make -C test/offline check-offline, a live prte/prun/pterm smoke test with one failure case leaving the DVM up, and a -W docs build. Part 2 adds the end-to-end check that a launched process on turin-4gpu-nvml sees CUDA_VISIBLE_DEVICES and NCCL_IB_HCA naming the GPU and the HCA under one host bridge.

9.2.3.6. Task checklist

Part 1

  • [x] prte_hwloc_base_host_bridge() in src/hwloc

  • [x] group_by_complex() in rmaps_base_devices.c, skipped under

    interleave and for ndev of one

  • [x] test_devices.c cases on turin-4gpu.xml and test-topo.xml

  • [x] Offline golden for device=nic:ndev=2; refusal case for

    --bind-to package; existing goldens unchanged

  • [x] cli-map-by.rst NDEV text; rmaps/base/AGENTS.md

Part 2

  • [x] Qualifier vocabulary, both parsers, job/app attributes, options field

    (appended, so no rmaps version bump)

  • [x] Second enumeration and per-complex dealing in devices_begin();

    NICs appended in devices_record()

  • [x] Candidate ranking: OpenFabrics first, deepest common ancestor, bus

    order

  • [x] Refusal for a non-GPU class; the rmaps:no-nic-near-gpu topic,

    shown per node under the job’s nspace so show_help aggregates it

  • [x] Unit, offline and end-to-end envar tests

  • [x] cli-map-by.rst, help-mapby.txt, placement examples

Part 3 (openpmix)

  • [x] types field in pmix_pnet_pcimatch_t, honored by

    pmix_pnet_base_get_assigned_devices(); pnet version bump

  • [x] osdev_preferred(): the cxi OS device over hsnN

  • [x] nvd: NVIDIA, Broadcom and Intel RoCE entries; node_type

    check; FI_VERBS_DEVICE_NAME with the prefix-collision rule

  • [x] pnet/efa: FI_EFA_IFACE

  • [x] pnet/cxi: FI_CXI_DEVICE_NAME, declining non-cxiN selectors

  • [x] Unit tests on hand-edited topologies; the real Slingshot topology

    scrubbed and checked in to both trees

  • [x] pnet/cxi: the cxi device found in sysfs under hwloc < 2.14

  • [x] nvd: the node-type check limited to Intel’s adapters

  • [x] PMIx: a class 0x0302 function is a GPU without OS devices; PRRTE’s

    harness mirrors it once that lands

  • [x] A real Broadcom topology (mi350-bcm57608.xml)

  • [ ] Real EFA, Intel and AMD Pensando topologies (and a Slingshot one from

    hwloc 2.14 or later)

  • [ ] Propose a node-type info attribute to hwloc

  • [x] pnet AGENTS.md

  • [x] the PRRTE cli-map-by.rst list of variables set, with the harness

    change once the PMIx side has landed