This is a practical single-node smoke recipe for Mistral Large 4 FP8 with the rebuilt Midstream vLLM image. It proved model load, OpenAI API readiness, and one completion on a single eight-B200 node.
This is smoke validation, not production sizing guidance. It deliberately uses a 4K context, one sequence, eager execution, and no speculative decoding. Add DSpark, DCP, tool calling, or long context only after this baseline passes.
This is a practical single-node preview smoke recipe for Reflection Beam 1.0 using the rebuilt Midstream vLLM image. It documents the configuration that proved model load, OpenAI API readiness, and one completion on four B200 GPUs.
This is preview validation, not a production sizing guide. Start with the conservative four-GPU, 4K-context shape below. DFlash2 speculative decoding is deliberately excluded: its drafter checkpoint was not staged or validated in this run.
This is a practical, single-node preview smoke recipe for Muse Spark Open using the rebuilt Midstream vLLM image. It proves a conservative four-B200 serving configuration: model load, FlashInfer first-use compilation, OpenAI API readiness, and one completion.
This is preview validation—not a production sizing guide. The upstream published configuration uses 8x Blackwell GPUs. Begin with the four-GPU, 4K-context configuration below, then scale only after a clean smoke test.
NVIDIA Nemotron 3.5 Super VL 120B-A12B FP8: vLLM smoke test
This is a practical, single-node smoke-test recipe for the Red Hat AI FP8
checkpoint using the rebuilt nm-vllm-ent runtime image. It is deliberately a
small proof: one OpenAI-compatible chat completion and one image request.
This is a preview validation recipe, not a production sizing guide. Start with the published model card for supported configurations and limits.
Smoke test of the Qwen 3.8 Max model (2.4T total parameters, 95B active) using community NVFP4 weights on 8x NVIDIA B300 SXM6 275GB GPUs, served via the midstream v0.27.1 container image.
| Field | Value |
|---|---|
| Model | Inferact/Qwen3.8-2.4T-A95B-NVFP4 (community NVFP4 quant of Qwen/Qwen3.8-2.4T-A95B) |
| Container Image | quay.io/vllm/automation-vllm:cuda-31490750632 |
A guide for serving NVIDIA's Nemotron 3.5 Lightning model using the Red Hat AI Inference Server with vLLM.
Nemotron 3.5 Lightning is a 30B parameter Mixture-of-Experts model with 3B parameters active per forward pass (A3B). The NVFP4 variant uses NVIDIA's ModelOpt mixed quantization (NVFP4 weights + MXFP8 + FP8 E4M3 KV cache), bringing the on-disk footprint down to ~20 GiB.
The model includes reasoning capabilities — chat responses include a thinking/reasoning trace before the final answer.
Epic: INFERENG-9476 From: Doug Smith Date: 2026-07-29
From: Midstream (Doug Smith) For: Srija Ganguly (AIPCC container infra), Andre Lustosa (AIPCC Ecosystems) Date: 2026-07-22 Related tickets: AIPCC-21244, INFERENG-9335, INFERENG-9278, INFERENG-9323
Note: This is our best understanding from the midstream side based on what we currently have. We appreciate Srija's expert input on AIPCC conventions and infrastructure — this is meant to accelerate, not prescribe. If anything conflicts with AIPCC patterns or tooling, AIPCC's
A guide for serving Poolside's Laguna S 2.1 model using the Red Hat AI Inference Server with vLLM.
Laguna S 2.1 is a 118B parameter Mixture-of-Experts agentic coding model with 8B parameters active per forward pass. It uses 256 experts with top-10 routing across 48 layers (36 sliding-window + 12 global attention in a 3:1 ratio) and supports up to 256K context length.
The FP8 quantized version weighs ~121 GB on disk and fits comfortably on 4x H100 80GB GPUs.
Codename Arcee is a day-0 preview release. Open weights go live Sunday, July 27th. Doug is on PTO Friday July 25 through Monday July 28, so this needs to be picked up by someone else on the team.
This doc covers what's already done, what needs to happen, and where to find the detailed playbooks.
The gist is...