NVMe Controller Ownership Registry

August 28, 2026 · View on GitHub

The registry records which orchestrator owns each connected NVMe-oF controller. It is a small, cooperative coordination layer that lets independent tools share a host without stepping on each other's controllers — most importantly, so that a sweeping command like nvme disconnect-all does not tear down a connection some other component depends on.

The code is the source of truth. This document summarizes behavior and intent; for exact signatures see the header kdoc in src/nvme/registry.h and the nvme-registry-* man pages.

Why it exists

NVMe-oF connections on a host are rarely managed by a single actor. Independent orchestrators — agents that decide on their own which controllers to connect and disconnect — coexist on the same machine and share one flat device namespace (/dev/nvmeX), with no record of who created or manages each controller.

This holds even on a plain system with no daemons installed. There are already at least two orchestrators: the initramfs (NBFT and FC-kickstart connections made during early boot) and a human running nvme connect-all / nvme disconnect-all. Installing nvme-discoverd or nvme-stas only adds more.

Not every command is an orchestrator:

nvme connect / nvme disconnect — single, targeted actions; here the human is the orchestrator, deciding what to connect or disconnect.
nvme connect-all / nvme disconnect-all — orchestrating commands that, once invoked, decide on their own. connect-all reads a discovery controller's discovery log page (DLP), connects every DLP entry, and follows referrals into further discovery controllers, recursing through several layers; disconnect-all tears down across the whole host.
nvme discover — in between: it reads a discovery controller's DLP and prints it, but does not connect DLP entries or follow referrals. It is read-mostly; its only state-changing option is --persistent, which keeps the discovery connection itself.

The trigger was issue #2913: disconnect-all has host-wide scope but no way to tell a boot-critical NBFT connection — or one a daemon depends on — from a throwaway manual connect, so it would disconnect them all. To coordinate, an orchestrator must know who owns what.

This is why the registry is a new kind of state for nvme. nvme already reads configuration (host identity, saved connections) and kernel state from sysfs, but it kept no memory of what it had done — each invocation was independent. The registry is that memory: runtime state nvme/libnvme writes itself, automatically, as a controller is connected (under /run — per-boot, not across reboot), recording which orchestrator owns each controller and read back by a later invocation or any other tool on the host.

The registry is a cooperative tool, not an enforcement mechanism. Every orchestrator runs as root and could disconnect anything regardless. The registry simply lets well-behaved tools avoid doing so by accident.

What it is

The registry lives under /run/nvme/registry/ — runtime state that is tied to controller lifecycle and does not survive a reboot. It mirrors the sysfs convention: one directory per live controller, one plain-text file per attribute.

/run/nvme/registry/
    nvme1/
        owner
        seqnum
    nvme3/
        owner
        seqnum

$ cat /run/nvme/registry/nvme3/owner
nbft

Absence means unowned. A controller with no directory (or no owner file) is managed by nobody. There is no explicit "unowned" marker.
The well-known attribute is owner — the orchestrator identity string (e.g. stas, nbft, discoverd). Orchestrators may write additional private attributes; unknown attributes are ignored by everyone else.
libnvme also writes a reserved seqnum attribute next to owner — the value of /sys/kernel/uevent_seqnum captured at connect. It is internal bookkeeping for the cleanup rule (see Automatic cleanup); orchestrators neither set nor read it.
Directories are 0755, attribute files 0644: world-readable, root-writable. Both the root and per-device directories are created on demand on first write.
Writes are atomic (mkstemp → fsync → rename), so concurrent writers never corrupt an entry and readers never see a half-written value.
Controllers the kernel manages directly — PCIe and other memory-based transports — are out of scope: they are not reached over a fabric, and disconnect-all's transport-type check already excludes them.

Setting an owner

An orchestrator declares its identity once, then every controller it connects through that context is registered automatically on a successful connect:

struct libnvme_global_ctx *ctx = libnvme_create_global_ctx(...);
libnvme_set_owner(ctx, "stas");   /* registers owner=stas on every connect */

A process that does not call libnvme_set_owner() produces unowned connections.

What a connect does to the entry depends on the owner and on whether the kernel just created the controller:

owner	new controller	already connected
a name	write the entry, overwriting any existing one	write the entry (an explicit claim)
never set	clear the entry	leave the entry alone
set to `""`	clear the entry	clear the entry (an explicit disown)

A new controller is cleared because it cannot legitimately be owned by anybody: the entry can only be what a recycled instance number left behind. An already-connected controller may belong to another orchestrator, so an unstated intent must not touch it. Written entries are stamped with the current /sys/kernel/uevent_seqnum (used by the cleanup rule below).

From the CLI, pass --owner NAME to register ownership at connect time:

nvme connect     --owner discoverd ...
nvme discover    --owner discoverd ...
nvme connect-all --owner discoverd ...
nvme connect-all --nbft                 # NBFT controllers, owner=nbft
nvme connect-all --owner boot --nbft    # NBFT controllers, owner=boot (overrides nbft)
nvme connect     --owner= ...           # explicitly disown

nvme connect-all --nbft records owner=nbft automatically to protect firmware boot volumes. That nbft is only a default, though: an explicit --owner NAME given alongside --nbft overrides it, so nvme connect-all --owner NAME --nbft records the controllers as NAME. A plain --nbft (no --owner) keeps owner=nbft, which is what lets existing boot scripts get ownership for free without being changed. Without --owner and without --nbft, new connections are unowned and an already-connected controller keeps whatever entry it has.

An empty --owner= is the explicit disown, and it beats the --nbft default: nvme connect-all --owner= --nbft leaves the controllers unowned.

Automatic cleanup

A controller's entry must be removed once the controller goes away, or stale entries accumulate and a recycled device name looks owned. Creation and removal deliberately live in different places. Entries are written by libnvme on connect (see Setting an owner above) because only the connecting process knows the owner — it lives in that process's libnvme context, never in sysfs or the uevent — so nothing else, a udev rule included, could supply it. Removal needs no owner, only the device name, and it must happen even when the kernel drops a controller on its own (connectivity loss, ctrl-loss-tmo expiry) with no orchestrator involved, and even on a host running no orchestrator daemon at all. So removal is driven by the device-removal event and performed by a single, always-present agent — udevd: exactly one party cleans up, and it works with no daemon required.

libnvme ships a udev rule for exactly this. The udev daemon — present on essentially every system — removes a controller's registry entry when the kernel removes the controller:

ACTION=="remove", SUBSYSTEM=="nvme", KERNEL=="nvme[0-9]*", \
    RUN+="/bin/sh -c '[ -e /dev/%k ] || [ $env{SEQNUM} -le $$(cat /run/nvme/registry/%k/seqnum 2>/dev/null || echo 0) ] || rm -rf /run/nvme/registry/%k'"

($$ passes a literal $ to the shell; $env{SEQNUM} is the remove event's sequence number, substituted by udev.)

Why removal is guarded. udev rules run asynchronously: udevd processes a remove uevent some time after the kernel emits it. Meanwhile the kernel is free to recycle the just-freed instance number — it allocates the lowest available id, so the very next controller to connect can be handed the same nvmeN. This opens a rare race: a controller is removed and, at almost the same instant, a new one is connected and inherits its id, all before udevd gets around to running the remove rule for the old controller. By the time that rule finally fires, nvmeN already refers to a different, live controller that has written its own registry entry — and a blind rm would clobber it.

Concretely, the dangerous interleaving for the name nvme4:

The kernel removes the old nvme4, frees instance 4, and emits a remove uevent — now queued in udevd.
Before udevd runs that queued rule, a new controller connects. The kernel allocates the lowest free instance — 4 again — creates /dev/nvme4 (devtmpfs, synchronously), and libnvme, in the connecting process, writes /run/nvme/registry/nvme4/owner.
udevd finally runs the queued remove rule for the old nvme4. Without a guard it would rm -rf /run/nvme/registry/nvme4 and delete the new, live controller's entry.

The rule defends against this in two layers.

First, the [ -e /dev/%k ] check. devtmpfs removes a controller's device node synchronously, before the kernel emits KOBJ_REMOVE, so for the controller actually being removed /dev/nvmeN is already gone, whereas a recycled-and-live nvmeN exists again. If /dev/nvmeN is absent the entry is presumed stale; if it exists the id has been recycled, [ -e /dev/%k ] is true, the || short-circuits, and rm is skipped — preserving the new owner's entry. This catches the common recycling case.

Second, the seqnum comparison, which closes the residual window. The device-existence check alone still leaves a sub-shell TOCTOU: the new controller could create /dev/nvme4 and write its entry in the instant between [ -e ] returning false and rm running. To close it, libnvme stamps each entry at connect with the global, monotonic /sys/kernel/uevent_seqnum, and the rule removes only when the remove event's own SEQNUM (from the udev event environment) exceeds that stamp. The stale remove for the old nvme4 was emitted before the new nvme4's add, so its SEQNUM is necessarily lower than the new entry's stamp: the comparison holds, the || short-circuits, and the entry survives — independent of timing or scheduling. A genuine removal of the current controller carries a SEQNUM higher than that controller's own connect-time stamp, so its entry is removed as intended. (An entry with no stamp — e.g. one written by an older libnvme — compares as 0, and the rule falls back to the device-existence check alone.)

Entries appear and disappear atomically. A new entry would still be vulnerable for the instant between creating its directory and writing the seqnum file — the rule could read the missing stamp as 0 and delete the entry the library is still populating. libnvme avoids this by building each entry in a temporary directory, writing its attributes, and rename()-ing it into place, so an entry is only ever observed complete or absent. Library-driven removals are atomic the same way: the entry is renamed to a hidden sibling and then purged. This is why a missing stamp means a legacy entry from an older libnvme, never a half-written one. (The udev rule's own rm -rf is not atomic, but it only ever deletes an entry that is already going away.)

Note that udevd serialising udev events does not cover this race: the new controller's entry was written by libnvme during connect() — not by a udev rule — so udevd never ordered that write against the remove rule. The two signals the rule does rely on — live device-node state and the monotonic seqnum stamp — are exactly what distinguishes a stale entry from a recycled-and-live one without depending on event ordering.

Because the rule lives in libnvme, cleanup works whenever libnvme is installed, independent of nvme-cli.

`disconnect-all` behavior

This is the payoff. disconnect-all respects ownership by default:

Invocation	Disconnects
`nvme disconnect-all`	only unowned controllers (safe default)
`nvme disconnect-all --owner NAME`	only controllers owned by `NAME` (confirmation required)
`nvme disconnect-all --force`	all fabric controllers regardless of ownership (confirmation required)

--owner and --force are mutually exclusive. Both prompt for a typed yes when stdin is a terminal; non-interactive callers (scripts) proceed, since passing the flag is itself the statement of intent.

In every case, controllers the kernel manages directly (PCIe and other memory-based transports) are left alone — the transport-type check excludes non-fabric controllers before ownership (or --force) is even considered, so --force cannot reach them. This means nvme disconnect-all --force behaves exactly like the original nvme disconnect-all did: it tears down every fabric controller and never touches locally-attached devices. The ownership-aware default is the only new behavior layered on top.

By contrast, nvme disconnect <device> targets one named controller and always disconnects it — the caller's intent is unambiguous, so no guardrail applies.

`connect-all` / `discover --device` behavior

The exclusion list governs connections that don't exist yet; the registry governs ones that already do. Enumerating a discovery controller's log page is an operation on a live controller, so nvme connect-all --device <dev> and nvme discover --device <dev> check that controller's registry owner before proceeding: an unregistered or matching-owner DC proceeds, but a DC owned by someone else is skipped with a log line, regardless of whether --owner was given — there is no ownerless exemption, since the DC's ownership extends to everything discovered through it.

Inspecting the registry

nvme registry list                       # all live entries
nvme registry retrieve <device> -a <attr>
nvme registry update   <device> -a note -V "boot-path SAN"
nvme registry delete   <device> [-a <attr>]   # whole entry, or one attribute
nvme list -v                             # adds an Orchestrator column: owner,
                                         # '-' (unowned), or 'kernel' (PCIe)

Changing an owner (update -a owner) or removing an entry (delete) can stop an orchestrator from protecting a controller, so when run interactively these prompt for a [y/N] confirmation (default no). Updates to other attributes, non-interactive callers, and the C and Python APIs proceed without prompting. This is a guard against accidental mistakes, not access control — anyone with root can edit the files under /run/nvme/registry/ directly.