Skip to content

Instantly share code, notes, and snippets.

@jetnet
Last active November 4, 2026 17:00
Show Gist options
  • Select an option

  • Save jetnet/94c8dcfd5381985efd5030d8a4dca36d to your computer and use it in GitHub Desktop.

Select an option

Save jetnet/94c8dcfd5381985efd5030d8a4dca36d to your computer and use it in GitHub Desktop.
Stop spinning: wait for the GPU with WFE on DGX Spark (GB10)

Stop spinning: wait for the GPU with WFE on DGX Spark (GB10)

Most inference engines keep 1–3 CPU threads at 100% while serving. Those threads mostly wait: for a GPU kernel, a doorbell or the next step. They wait by polling a flag in a tight loop (spin-wait), because sleeping in the kernel costs tens of microseconds per wake-up.

On GB10 there is a cheaper way to wait. CPU and GPU are cache-coherent over NVLink-C2C, and on the tested configuration a core parked in the ARM WFE/WFET instruction, with the exclusive monitor armed on the flag's cache line, was woken by the GPU's store to that line with near-spin latency at much lower power. That GPU-triggered wake-up is a measured property of this platform, not an architectural guarantee: validate it on your own hardware before relying on it (section 5).

Measured on GB10 (driver 580.178.04, kernel 7.0.0-1019-nvidia)

A CPU core waits for a GPU thread that stores into pinned host memory. Times are the overhead on top of the GPU's work; power is the wall plug, with identical GPU work in every window.

Wait method Wake overhead p50 / p99 Extra wall power per waiting core
Spin (tight load-acquire loop) 1.43 / 1.73 µs +6.2 W
LDXR + WFET 1.50 / 2.25 µs +1.1 W
WFET without the monitor armed (control) 64 / 130 µs +1.3 W
nanosleep(50 µs) 32 / 52 µs +1.6 W

The control has no monitor-triggered wake-ups: timer events (the kernel's event stream, ~131 µs on GB10), interrupts and other events still wake it. Its flat 0–131 µs latency next to the near-spin latency of the armed variant is what shows that the GPU store wakes the armed core on this platform. One node, one GPU thread, one run per point; no tail-latency guarantee.

Applied to one thread in a real engine (TensorFold's RoCE proxy, TP2 on two Sparks, TEB hardmode, stock clocks, outputs byte-identical): −3.4 to −3.8 W per node, −3.2% energy for the same work, same wall time, head-node peak temperature 79 → 67 °C. Patch: https://gist.github.com/jetnet/206190be74c3b552a08874fd610889f0

1. The wait primitives (C, aarch64)

wait_change_u32 is the same sequence as the Linux kernel's __cmpwait (behind smp_cond_load_acquire), with WFET (wait with an absolute timeout, Armv8.7 FEAT_WFxT, present on GB10). Never execute it unless wfe_available() returned 1.

#include <stdint.h>
#include <sys/auxv.h>
#ifndef HWCAP2_WFXT
#define HWCAP2_WFXT (1UL << 31)
#endif

/* Detects instruction support only. It says nothing about whether GPU stores wake WFE on this
 * platform: validate that separately (section 5) before deploying. */
static int wfe_available(void) { return (getauxval(AT_HWCAP2) & HWCAP2_WFXT) != 0; }

static inline uint64_t cntvct(void) {
    uint64_t t;
    __asm__ volatile("isb\n\tmrs %0, cntvct_el0" : "=r"(t)::"memory");
    return t;
}
static inline uint64_t cntfrq(void) {
    uint64_t f;
    __asm__ volatile("mrs %0, cntfrq_el0" : "=r"(f));
    return f;
}

/* Park until *p may have changed from `old`, an event arrives, or CNTVCT reaches `deadline`.
 * Returns early and spuriously by design; the caller re-reads the value. Requires wfe_available(). */
static inline void wait_change_u32(volatile uint32_t *p, uint32_t old, uint64_t deadline) {
    uint32_t tmp;
    __asm__ volatile("mov x9, %[dl]\n\t"
                     "sevl\n\t"
                     "wfe\n\t"                        /* consume any stale event */
                     "ldxr %w[tmp], %[v]\n\t"         /* arm the exclusive monitor on *p's line */
                     "eor %w[tmp], %w[tmp], %w[old]\n\t"
                     "cbnz %w[tmp], 1f\n\t"           /* already changed: do not sleep */
                     ".inst 0xd5031009\n"             /* wfet x9 */
                     "1:"
                     : [tmp] "=&r"(tmp), [v] "+Q"(*p)
                     : [old] "r"(old), [dl] "r"(deadline)
                     : "x9", "memory");
    (void)tmp;
}

/* Wait until the 32-bit sequence counter *p reaches `want`.
 * Contract: *p is initialised before the writer may publish; the writer only increases it, and
 * `want` is less than 2^31 ahead of the current value (wrap-safe signed compare).
 * Returns 0 when reached, -1 on overall timeout, -2 if timeout_ns is too large to represent.
 * Returns only on success or timeout: for cancellation or housekeeping (e.g. CQ draining), call
 * wait_change_u32 in your own loop and do that work between slices. */
static int wait_seq_ge(volatile uint32_t *p, uint32_t want, uint64_t timeout_ns) {
    const uint64_t hz = cntfrq();
    const unsigned __int128 ticks = (unsigned __int128)timeout_ns * hz / 1000000000u;
    const uint64_t start = cntvct();
    if (ticks > (unsigned __int128)(UINT64_MAX - start)) return -2;
    const uint64_t end = start + (uint64_t)ticks;
    const uint64_t slice = hz / 1000 ? hz / 1000 : 1;  /* park at most ~1 ms at a time */
    uint32_t v;
    while ((int32_t)((v = __atomic_load_n(p, __ATOMIC_ACQUIRE)) - want) < 0) {
        const uint64_t now = cntvct();
        if (now >= end) return -1;
        const uint64_t left = end - now;
        wait_change_u32(p, v, now + (left < slice ? left : slice));
    }
    return 0;
}

Without FEAT_WFxT, keep your original wait implementation. Plain WFE has no architectural timeout, and the Linux event stream's availability and period cannot be assumed, so an outer deadline check cannot bound an individual WFE. Even WFET deadlines are not hard real-time: a descheduled thread returns later.

2. The writer side

The waiter must see the value published with release semantics, after the data it guards.

  • Another CPU thread: __atomic_store_n(flag, seq, __ATOMIC_RELEASE);

  • A GPU kernel, system scope, on memory the GPU can write coherently (on GB10: host memory from cudaHostAlloc(..., cudaHostAllocMapped) or cudaHostRegister(..., cudaHostRegisterMapped), device pointer from cudaHostGetDevicePointer; pageable memory via ATS also worked on the tested system):

    #include <cuda/atomic>
    __global__ void set_flag(uint32_t *flag, uint32_t seq) {
        cuda::atomic_ref<uint32_t, cuda::thread_scope_system>(*flag).store(seq, cuda::memory_order_release);
    }

3. A drop-in helper for "wait until this stream is done" (C/C++ and Python)

cudaStreamSynchronize() spins inside the closed driver, so you cannot add WFE there. Instead, enqueue a one-thread kernel that releases a sequence number after the stream's work, and wait for it on the host. One small library, usable from C, C++ and Python:

// wfe_wait.cu   build: nvcc -O2 -arch=sm_121 -Xcompiler -fPIC -shared -o libwfe_wait.so wfe_wait.cu
// (paste the code from section 1 above this line)
#include <cuda/atomic>
#include <cuda_runtime.h>
#include <stdlib.h>
#include <string.h>

__global__ void wfe_set_flag(uint32_t *flag, uint32_t seq) {
    cuda::atomic_ref<uint32_t, cuda::thread_scope_system>(*flag).store(seq, cuda::memory_order_release);
}

extern "C" int wfe_ok(void) { return wfe_available(); }

// The flag lives alone on its own page-aligned 4 KiB page, mapped for the GPU.
// Returns the host pointer (NULL on error) and stores the device pointer in *dev.
extern "C" void *wfe_flag_alloc(void **dev) {
    void *p = NULL;
    if (posix_memalign(&p, 4096, 4096) != 0) return NULL;
    memset(p, 0, 4096);                          // initialise before any GPU write
    if (cudaHostRegister(p, 4096, cudaHostRegisterMapped) != cudaSuccess) {
        free(p);
        return NULL;
    }
    if (cudaHostGetDevicePointer(dev, p, 0) != cudaSuccess) {
        if (cudaHostUnregister(p) == cudaSuccess)   // never free memory that may still be registered
            free(p);
        return NULL;
    }
    return p;
}

// Enqueue "flag = seq" after everything already submitted to `stream`. 0 = ok, else a cudaError_t.
extern "C" int wfe_signal(void *dev, uint32_t seq, void *stream) {
    wfe_set_flag<<<1, 1, 0, (cudaStream_t)stream>>>((uint32_t *)dev, seq);
    return (int)cudaGetLastError();
}

// 0 = flag >= want, -1 = timeout, -2 = timeout too large.
extern "C" int wfe_wait_seq_ge(void *host, uint32_t want, uint64_t timeout_ns) {
    return wait_seq_ge((volatile uint32_t *)host, want, timeout_ns);
}

C++ use (keep the original sync as the fallback):

static uint32_t seq = 0;   // one flag + counter per waiter; never share them between threads or streams
if (wfe_ok()) {
    if (wfe_signal(dev, ++seq, stream) != 0) { /* launch error: handle like any CUDA error */ }
    if (wfe_wait_seq_ge(host, seq, 10ull * 1000000000ull) != 0) { /* timeout, see below */ }
} else {
    cudaStreamSynchronize(stream);
}

Python use (ctypes.CDLL releases the GIL during the call, so other Python threads keep running):

import ctypes, threading, torch
lib = ctypes.CDLL("./libwfe_wait.so")
lib.wfe_ok.restype = ctypes.c_int
lib.wfe_flag_alloc.restype = ctypes.c_void_p
lib.wfe_flag_alloc.argtypes = [ctypes.POINTER(ctypes.c_void_p)]
lib.wfe_signal.argtypes = [ctypes.c_void_p, ctypes.c_uint32, ctypes.c_void_p]
lib.wfe_wait_seq_ge.argtypes = [ctypes.c_void_p, ctypes.c_uint32, ctypes.c_uint64]

dev = ctypes.c_void_p()
host = lib.wfe_flag_alloc(ctypes.byref(dev))   # keep alive for the lifetime of all GPU writes
seq = 0

def _wait_stream_unlocked(stream=None):
    global seq
    stream = stream or torch.cuda.current_stream()
    if not lib.wfe_ok() or not host:
        return stream.synchronize()
    seq += 1
    if lib.wfe_signal(dev, seq, stream.cuda_stream) != 0:
        raise RuntimeError("wfe_signal launch failed")
    if lib.wfe_wait_seq_ge(host, seq, 10_000_000_000) != 0:
        raise TimeoutError("GPU did not signal")

_wait_lock = threading.Lock()

def wait_stream(stream=None):
    with _wait_lock:
        return _wait_stream_unlocked(stream)

This wrapper serializes waits because its flag and sequence are shared. Concurrent independent waits need separate flag/counter state per waiter; independently progressing streams must not publish into one shared flag (a faster stream's higher sequence would satisfy a slower stream's wait before its work finishes).

What this does and does not replace:

  • Completion covers only the work already ordered into that stream, like cudaStreamSynchronize of that stream. The tiny signal kernel itself may still be retiring when the flag flips, so cudaStreamQuery can briefly report "not ready". It does not report asynchronous kernel errors: check them where you check them today.
  • CUDA graphs: if the signal kernel is captured in a graph, update its seq argument for every launch (e.g. cudaGraphExecKernelNodeSetParams); captured arguments otherwise repeat and a stale value would count as completion.
  • On timeout the GPU work may still be running: do not free, reuse or reset its buffers or the flag until completion is established some other way.

4. Rules that matter

  1. Validate GPU-triggered wake-ups on your hardware first. Use a positive control (a CPU thread as the writer) and a negative control (plain ldr instead of ldxr, which has no monitor-triggered wake-ups). Deploy only if the GPU-writer latency is within a few µs of a spin and clearly unlike the negative control; otherwise keep spinning. The probe used for the numbers above did exactly this.
  2. Give the flag its own page (as wfe_flag_alloc does). Writes to neighbours on the same reservation granule cause spurious wake-ups.
  3. Bound every wait. wait_seq_ge returns only on success or overall timeout; if you need cancellation or periodic work, loop over wait_change_u32 yourself and do it between slices.
  4. Re-read with acquire after every wake-up. WFE/WFET return spuriously: interrupts, timer events, unrelated stores.
  5. top will still show 100% CPU. WFET is an instruction, not a syscall, so Linux counts the thread as running. Measure watts at the wall, not CPU%.
  6. x86 has a similar mechanism, UMONITOR/UMWAIT (Intel WAITPKG) and MONITORX/MWAITX (AMD). Not tested here.

Tested on DGX OS 7 / GB10 with CUDA 13.0: libwfe_wait.so builds with nvcc from exactly this text (nvcc prints a harmless warning #550 about tmp, an asm output that is intentionally unused). A C++ check (2000 GPU-signalled waits after ~0.1 ms kernels, plus one after a long kernel) and the Python wrapper (500 waits, ctypes, through the locked wait_stream) passed; the timeout path returned -1 after the requested 3.00 ms, and a too-large timeout returned -2.

Snippets above: public domain, use them however you like.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment