PCIe lane topology, bifurcation and link verification

Lanes are a fixed budget the board spends on your behalf: the CPU owns a few root ports, everything else sits behind a chipset switch on one shared uplink, and half the sockets are muxed against each other. We map where the lanes go on X670E and Z790, then split a x16 root port four ways across a passive riser. The bench ends with every link read back against its own LnkCap rather than against the manual.

Where the lanes actually come from§

Two populations of lanes, not interchangeable. CPU-attached lanes terminate in root ports on the processor's own root complex: one hop from the memory controller, bandwidth owned outright. Chipset-attached lanes hang off downstream ports of the PCH, a PCIe switch wearing a costume that also carries the SATA, USB and network controllers — all of them reaching the CPU through one uplink nothing can widen.

AM5 exposes 28 lanes: sixteen Gen5 for graphics, two Gen5 x4 groups for M.2, one Gen4 x4 group reserved for the chipset uplink. Vendors spend the first twenty-four differently — a second CPU-attached M.2 socket on one board, an x8/x8 split of the graphics block on another. Raptor Lake offers sixteen Gen5 lanes plus one Gen4 x4 group, and reaches the PCH over DMI 4.0 x8: eight lanes of Gen4 signalling in a private package.

Ceilings follow from the encoding, GT/s × lanes × 128/130 ÷ 8. Gen4 x4 — the X670E uplink and most chipset M.2 sockets — is 7.88 GB/s; Gen4 x8 and Gen5 x4 are both 15.75 GB/s; Gen5 x16 is 63.0 GB/s. Line rates, not throughput, and shared by everything behind the uplink. What matters for placement is not the drive's rated speed but which side of it the socket sits on.

The X670E daisy chain and its far-die tax§

X670 and X670E are not one chip. They are two Promontory 21 dies in series: the CPU drives die A over Gen4 x4, die A drives die B over a second Gen4 x4. B650 uses one die. The block diagram shows this even where the feature list implies a single chipset.

Three consequences. Aggregate bandwidth for both dies is bounded by the CPU-to-A link, so a chipset NVMe drive, a 10 GbE card and a USB enclosure all draw from the same 7.88 GB/s. Anything on die B pays two switch traversals per direction rather than one, each store-and-forward with arbitration, so the cost lands on latency — QD1 random reads notice, sequential transfers do not. And contention is asymmetric: die B's traffic crosses A first, so when A-to-B saturates the far die starves first.

The remedy is placement, not tuning. Boot drives, capture cards and latency-sensitive NICs belong on CPU lanes, or failing that on die A; bulk storage is what the far die is for. Far-die DMA still competes for memory-controller bandwidth, so a fabric already tight from aggressive DDR5 subtiming work has less headroom than the block diagram suggests.

Measure the tax on your own board: same drive, CPU socket then far-die socket, at queue depth one. Not sequential throughput — at QD1 a single Gen4 drive comes nowhere near the uplink ceiling, so both sockets look identical.

fio --name=qd1 --filename=/dev/nvme1n1 --readonly \
    --rw=randread --bs=4k --direct=1 --ioengine=libaio \
    --iodepth=1 --numjobs=1 --norandommap \
    --runtime=120 --time_based \
    --percentile_list=50:99:99.9

Record mean, p99 and p99.9 at the same drive temperature on an otherwise-idle machine; the delta is the hop cost.

Shared lanes, muxes, and reading the manual§

No consumer board has as many lanes as connectors. Vendors close the gap with PCIe quick-switches — ASM1480-class parts, the Pericom PI3DBS series — routing a four-lane group to either an M.2 socket or a pair of SATA ports, or steering the graphics block's upper eight lanes to a second slot. Firmware drives the select as a GPIO at POST, from the socket's presence-detect pin, a setup item, or a board-ID value you cannot see.

Hence the shared-lane table, always in the footnotes. Read the block diagram for which lanes are CPU-attached, then every footnote on the connector list; the wording is consistent across vendors. "M.2_2 shares bandwidth with SATA5_6" means a mux exists and SATA loses. "PCIEX16_2 runs at x4 mode" means that slot was never wired to the CPU. "PCIEX16_1 will run at x8 when PCIEX16_2 is occupied" means the root port bifurcates, a different mechanism given the same footnote style.

Trust none of it: firmware revisions change mux policy and footnotes go stale. Verify by snapshot and diff — record every device's negotiated link state, fit the card, cold boot, record again.

snap() {
  for d in /sys/bus/pci/devices/*/; do
    b=${d%/}; b=${b##*/}
    [ -r "$d/current_link_width" ] || continue
    printf '%s %s / %s  max %s / %s\n' "$b" \
      "$(cat "$d/current_link_speed")" "$(cat "$d/current_link_width")" \
      "$(cat "$d/max_link_speed")"     "$(cat "$d/max_link_width")"
  done | sort
}
snap > /root/links.before
# fit the drive, cold boot, then:
snap > /root/links.after
diff -u /root/links.before /root/links.after

The diff shows what the board took away, including devices that vanished. SATA losses will not show there — a muxed-off port never becomes a PCI device — so read the AHCI ports-implemented mask, which firmware rewrites when it flips the mux.

dmesg | grep -i 'ahci.*impl'

ahci 0000:0e:00.0: AHCI 0001.0301 32 slots 4 ports 6 Gbps 0x3 impl SATA mode

The hex value before impl is a bitmask of implemented ports — two where the manual credits four is the mux having done its work, and more reliable than counting cables that link up.

Bifurcation, and how to spot a switched riser§

A x16 connector carries sixteen lanes to one root port by default. Bifurcation splits that sixteen-lane block into independent links — x8/x8, x8/x4/x4, x4/x4/x4/x4 — each becoming its own root port with its own device or function number, its own LTSSM and its own training. Four devices on a bifurcated slot share nothing downstream of the CPU.

The critical point: bifurcation is a root-port capability, not a slot property. Firmware configures it before link training — AGESA on AM5, PEI-phase reference code on Intel — and no riser, adapter or cable can induce it. The connector is passive. If the setting is absent from your setup menu it is absent from the machine, and the only avenue left is firmware-side — see UEFI firmware modding and recovery.

Names vary — ASUS "Bandwidth Bifurcation Configuration", ASRock a per-slot "Link Width" offering x4x4x4x4, Gigabyte and MSI "PCIe Bifurcation" — and on consumer Intel boards x4/x4/x4/x4 is often missing. Several boards also ignore a four-way setting when a muxed sibling slot is occupied.

Risers come in two kinds. A passive riser is traces, a connector and possibly redrivers; it requires host bifurcation. An active riser carries a real PCIe switch — ASMedia ASM2812/ASM2824, Broadcom PEX8724/PEX8747, Microchip Switchtec — presenting one upstream port and several downstream. It works on any root port, and in exchange adds a switch hop of latency, needs auxiliary power and a heatsink, and makes every device share one upstream link.

Tell them apart from the host without opening anything. A switch inserts a bridge pair — an upstream port plus a downstream port per device — and its vendor ID shows: 1b21: ASMedia, 10b5: Broadcom PLX. A passive riser inserts nothing; endpoints sit directly under their root ports.

The bench§

Two boards, open bench, 22 °C ambient, no case airflow. Bench A: ASUS ProArt X670E-Creator WiFi, AGESA 1.2.0.3c, Ryzen 9 7950X. Bench B: ASUS ROG Strix Z790-A Gaming WiFi II, BIOS 2403, i9-13900K. Both on Debian 13 trixie, kernel 6.12, pciutils 3.13.0, rasdaemon 0.8.0. Record your own firmware string with cat /sys/class/dmi/id/bios_version, not the setup screen.

Under test: one x16 GPU in the primary slot, a passive x4/x4/x4/x4 riser carrying four M.2 drives, and a Gen4 x4 NVMe moved between a CPU socket and a far-die chipset socket. Run everything below as root — lspci -vv truncates extended capability blocks for unprivileged users, and both AER and Resizable BAR live there.

Switch silicon and long-riser redrivers get hot under sustained traffic, and their bit-error rate is temperature-dependent. If a riser only misbehaves after twenty minutes, put the IR thermometer on it, using the soak discipline from the VRM load-line and thermal analysis note.

Reading the real topology on Linux§

The manual states intent; configuration space states fact.

Topology and the path to a device§

Start with the tree. lspci -tvnn prints the hierarchy with numeric and human-readable IDs; what you read is the nesting depth, not the names.

lspci -tvnn

Each level of indentation is another bridge: a chipset device on X670E sits two levels below its root port, a CPU-attached NVMe one. The unambiguous form is the sysfs path.

readlink -f /sys/bus/pci/devices/0000:0d:00.0 | tr '/' '\n' | grep ':'

The output is the chain from the root port down: on a far-die device, CPU root port, die A's upstream port, a die B downstream port, then the endpoint. Bus numbers differ per board — read the shape, not the digits. The first entry is the root port, whose LnkCap answers the bifurcation question and whose AER counters cover the subtree.

LnkCap against LnkSta§

For any one link, a single command gives both halves of the answer.

lspci -vv -s 0000:01:00.0 | grep -E 'LnkCap:|LnkCtl:|LnkSta:'

LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L1, Exit Latency L1 <64us
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkSta: Speed 16GT/s, Width x4 (downgraded)

LnkCap is what this end can do; LnkSta is what it negotiated, and modern lspci annotates the status line (ok) or (downgraded) by comparing them. Read both ends: an endpoint reporting x4 says nothing about the cause until you read the root port above it.

This is also how you prove bifurcation happened. When firmware bifurcates, the root port's capability changes — four root ports, each advertising LnkCap ... Width x4. When it does not, one port advertises Width x16 and reports LnkSta ... Width x4 (downgraded), because only the first four lanes found a partner. Same status line, different cause: a downgrade against a matching capability is a training or signal-integrity problem; a x16 capability where you expected four x4 ports is a setting that did not take.

Reading the registers directly§

When lspci's interpretation is what you doubt, or you are in an initramfs without the PCI ID database, read the register directly. Link Status sits at offset 0x12 in the PCI Express capability, which setpci addresses by name.

setpci -s 01:00.0 CAP_EXP+12.w    # Link Status
setpci -s 01:00.0 CAP_EXP+0c.l    # Link Capabilities
setpci -s 01:00.0 CAP_EXP+10.w    # Link Control
setpci -s 01:00.0 CAP_EXP+30.w    # Link Control 2 (target link speed)

Link Status decodes as bits 3:0 current speed — 1 is 2.5 GT/s, 2 is 5.0, 3 is 8.0, 4 is 16.0, 5 is 32.0 — bits 9:4 negotiated width as a plain integer, bit 13 link active. A returned 3104 is 16 GT/s, sixteen lanes, link up. Link Control 2's low nibble uses the same encoding and holds any deliberate cap.

Train-down, AER, and what correctable errors mean§

Links negotiate downward for several unrelated reasons. Equalisation can fail at the higher rate and fall back a generation, the usual outcome of a marginal riser at Gen4 or Gen5 where the loss budget is unforgiving. Firmware can cap the slot; the endpoint can cap itself through Link Control 2. And most often reported as a fault when it is not one, link power management parks an idle GPU at 2.5 GT/s — always re-read LnkSta under sustained load.

Real marginality shows up in the correctable counters instead. AER lives on the root port, and the kernel exposes per-device counters when it owns error handling.

/sys/bus/pci/devices/0000:00:01.1/aer_dev_correctable

RxErr 0
BadTLP 0
BadDLLP 0
Rollover 0
Timeout 0
NonFatalErr 0
CorrIntErr 0
HeaderOF 0
TOTAL_ERR_COR 0

RxErr counts physical-layer receiver errors, the most direct signal-integrity indicator short of an oscilloscope. BadTLP and BadDLLP are LCRC failures: the packet arrived corrupted and was retransmitted. Rollover and Timeout mean the replay buffer worked repeatedly. Correctable is not harmless: every replay costs latency and link bandwidth, and a rising rate is the normal precursor to an uncorrectable error.

Read them as a rate, never an absolute: record, move a known volume of data, read again, express as errors per gigabyte. If the files do not exist, firmware kept error handling to itself — boot with pcie_ports=native. ras-mc-ctl --summary and --errors give the same events with timestamps, which is what an intermittent fault needs.

Physically, a climbing count on a riser is one of a short list: excessive flex length or a tight bend radius, a cable without proper ground return, redrivers absent where trace length demanded them, a card not fully seated, contamination on the fingers. Clean contacts with isopropyl and a lint-free swab, never an abrasive — oxidation and flux residue raise contact resistance in ways that look exactly like a bad riser. To exonerate it, run the device directly in the slot over the same transfer volume.

Resizable BAR and ASPM§

Resizable BAR§

Without Resizable BAR the host maps a 256 MB aperture into the GPU's memory and the driver windows through it. With it, the endpoint advertises the sizes it supports and firmware programs a BAR large enough to map all of VRAM. This changes addressing, not link width or speed; the effect is workload-specific.

Two firmware conditions gate it: Above 4G Decoding on, because the resized BAR is a 64-bit prefetchable window that cannot live below the 4 GB line, and CSM off, because the legacy video path forces the small aperture regardless. Verify on the host, not in the setup menu.

lspci -vv -s 0000:01:00.0 | grep -A3 'Resizable BAR'
lspci -vv -s 0000:01:00.0 | grep 'Region 0'

The capability block lists current size against the supported set; Region 0 shows what was assigned. If they disagree, firmware declined. Driver-side, dmesg | grep -i 'amdgpu.*BAR' prints detected VRAM size beside BAR size — they match only when resizing worked — and nvidia-smi -q | grep -A3 'BAR1 Memory Usage' shows a BAR1 total near VRAM size, not 256 MB. Recent kernels also expose resourceN_resize in sysfs, but the write succeeds only with no driver bound.

ASPM and idle exit latency§

ASPM trades idle power for wake latency in three tiers. L0s is a fast one-direction electrical idle, inconsistently implemented. L1 idles both directions and may shut the PLL down, costing microseconds to exit. The substates go further: L1.1 keeps common-mode voltage alive for a faster return, L1.2 drops it for the lowest power and longest exit. Both need CLKREQ# routed, and on risers that signal is often missing — disabling L1SS if you are lucky, dropping the link if you are not.

Read the current state from the same capability dump: LnkCap lists supported states with exit latencies, LnkCtl what is enabled, L1SubCap and L1SubCtl1 the substates. Policy is one file.

/sys/module/pcie_aspm/parameters/policy

[default] performance powersave powersupersave

The bracketed entry is active. Set it at boot with pcie_aspm.policy=performance, or pcie_aspm=off as a diagnostic. On a desktop bench the cost is not bandwidth but the first access after idle: ioping -c 100 -i 1 /dev/nvme0n1 forces a second of idle between requests and surfaces the outliers a back-to-back benchmark hides. NVMe compounds this with its own APST states, readable via nvme get-feature -f 0x0c -H /dev/nvme0.

Failure modes and validation§

Both failure classes are intermittent at POST; one good boot proves nothing.