wseaton

kind drain test for llm-d/llm-d-router#3029

Compares the standalone chart at the PR's merge-base (base) and head (pr) on kind, with the same values. Active-passive EPP with priority routing and flow control, 2 service-mode Envoy replicas, 3 llm-d-inference-sim engines with ~2.2 s streams. loadgen runs in-cluster; a request is OK only on a 200 whose stream ends in data: [DONE].

• run.sh VARIANT SCENARIO [RATE] [SECONDS]: install, load, fault at +15 s, output in runs/
• loadgen.go: fixed-rate streaming client
• values.yaml, sim.yaml: chart values, simulators and loadgen pod

wseaton

What limits llm-d-async dispatch rate

We started by load testing the Postgres transport in llm-d-async #452 and ended up measuring the dispatcher itself. Three limits sit in front of the store, and each one is set by the dispatcher's own code or config:

1. The poll tick. Each queue dispatches one batch per poll_interval_ms tick, even with a full backlog, so the rate can never exceed batch_size / poll_interval. The shipped defaults (batch 10, 1,000 ms) cap a queue at 10 req/s. Our bench ran 64 per 50 ms, a 1,280 req/s cap, and both transports sat at about 1,250 req/s with 1k-token prompts. A 1 ms poll raised dispatch to about 4,400 req/s on redis and 4,500 on Postgres, with Postgres using under 3 of its 8 cores.
2. The prompt travels with the envelope. One goroutine per queue decodes every request, and encoding/json reads each prompt four times, which was 80% of dispatcher CPU at 128k-token prompts. Storing the prompt apart from the envelope removes th

wseaton

Cause: llm-d-router #1940 removed --model-server-metrics-scheme (a 0.10 change). The KServe scheduler preset still passes it because its app.kubernetes.io/version annotation says 0.9.0, which is older than the image actually shipped (a 0.10 prerelease). KServe already has a migration that moves the scheme into the metrics-data-source plugin, but it's gated >= 0.10.0, so it never fires.

Fix: bump the annotation so it matches the image, which is a pre-release.

 annotations:

wseaton

FROM registry.access.redhat.com/ubi9/ubi

RUN dnf -y install \
        libibverbs \
        libibverbs-utils \
        librdmacm \
        infiniband-diags \
        libatomic \
        pciutils \
        iproute \

wseaton

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "huggingface_hub>=0.24",
# ]
# ///
"""
Check whether Gemma model configs satisfy the vLLM unify_kv_cache_spec_page_size
divisibility requirement:

wseaton

#!/usr/bin/env nu
#
# vLLM Rust frontend: TLS termination + graceful-shutdown drain test.
#
# Runs the real `vllm-rs` frontend against the vllm-vcr inference simulator
# (a mock engine-core backend, no GPU or weights) and checks that the frontend
# drains correctly on SIGTERM. Companion to the upstream PRs adding OpenSSL TLS
# termination and hyper-util GracefulShutdown to the Rust frontend.
#
# Scenarios:

wseaton

# Auto-generated by Hermes (RDMA Cluster Analyzer)
# https://github.com/llm-d-incubation/hermes
#
# Configuration:
#   workload: deepep-lowlatency-test
#   namespace: llm-test
#   image: ghcr.io/llm-d/llm-d-cuda-dev:sha-d58731d@sha256:ba067a81b28546650a5496c3093a21b249c3f0c60d0d305ddcd1907e632e6edd
#   sriov_network: roce-p2
#   gpus_per_node: 1
#   rdma_resource_type: rdma/roce_gdr

wseaton

# Auto-generated by Hermes (RDMA Cluster Analyzer)
# https://github.com/llm-d-incubation/hermes
#
# Configuration:
#   workload: deepep-lowlatency-test
#   namespace: llm-test
#   image: ghcr.io/llm-d/llm-d-cuda-dev:sha-d58731d@sha256:ba067a81b28546650a5496c3093a21b249c3f0c60d0d305ddcd1907e632e6edd
#   sriov_network: roce-p2
#   gpus_per_node: 1
#   rdma_resource_type: rdma/roce_gdr

wseaton

{
  "0": {
    "0": {
      "engine_id": "3c8e544e-31a9-4129-a54c-2898ff20b7d4",
      "agent_metadata": "bml4bFNlckRlc3xBZ2VudCQAAAAAAAAAOGNiZjBkNDktYmY0Yy00ZWU3LThkYmEtNjBhNWVhNWE1Nzg3fENvbm5zCAAAAAAAAAABAAAAAAAAAHx0AwAAAAAAAABVQ1h8Y6wCAAAAAAAAIIVk3O9YpWkOIABjdXfMKzIAQJdQAAAAAABPAwCI3eFals2HO8EhJgAAAsCoyfjPGRe3UThXdI1O4Z21NgATIwCCzxUhEgECdBHD0wjHFIsAAADwAAAAAM8ZF7dROFOePkvX4Dc3ARMjAILQB0IIdBHD0wjHFAuNU3fMKzIAAHBQd8yrMwBPIwCIJYAYAAAAAABDCHQRw9MIxxQLp9N3zCsyAABwUHfMqzMATyMAiEuIDsAtAADAZCMBAAEAJ4qwD6EzXSDSTjSrSjVG3yIAADHRXHzhM10g0k7KuFI1RgMnAAO2KwB+OXfMKzNdINJOYMZaNUbfI8AHvCsAuAwAAHrUd8wrM10g0k40q0o1Rt8iwAEAsw93zKszXSDSTsq4UjVGAycAg8ErAGUlAgQCAP9/J4qwD6EzXSDSTjSrSjVG3yIAADHRXHzhM10g0k7KuFI1RgMnAAOnKQB+OXfMKzNdINJOYMZaNUbfIwAHqykAeAwEAHrUd8wrM10g0k40q0o1Rt8iAAEAsw93zKszXSDSTsq4UjVGAycAg7MpAGYlAwQFAP9/J4qwD6EzXSDSTjSrSjVG3yIAADHRXHzhM10g0k7KuFI1RgMnAAMnKQB+OXfMKzNdINJOYMZaNUbfIwAHKykAZAwCAHrUd8wrM10g0k40q0o1Rt8iAAEAsw93zKszXSDSTsq4UjVGAycAgzQpAGclBAQGAP9/J4qwD6EzXSDSTjSrSjVG3yIAADHRXHzhM10g0k7KuFI1RgMnAAMbKgB+OX

wseaton

const message = document.createElement("div");
message.style.padding = "20px";
message.style.backgroundColor = "#d4edda";
message.style.color = "#155724";
message.style.border = "1px solid #c3e6cb";
message.style.borderRadius = "5px";
message.style.margin = "10px 0";
message.style.fontFamily = "Arial, sans-serif";
message.style.fontSize = "14px";