HDCharles

Reproduction material moved to the PR

The runnable MXFP4A16 imatrix quantization/evaluation code, full experiment settings, and instructions now live in PR #3477.

The numerical result table was removed from this gist; it will be shared separately.

HDCharles

"""
Benchmark for MSE observer grid search. (results at bottom)

Profiles the grid search that finds optimal min/max ranges for quantization.
The hot path is `_calculate_error` (called once per shrink step: default 20
steps), which runs `calculate_qparams` + `fake_quantize` on the full weight
tensor each time.

Usage:
    python benchmarks/bench_mse_observer.py [--device cuda] [--rows 4096] [--cols 4096]

HDCharles

GPTQ Memory Characteristics

This document compares GPTQ GPU memory usage on main and the batched GPTQ PR, with and without Hessian offloading. Measurements were collected on an NVIDIA H100 using torch.cuda.max_memory_allocated() and torch.cuda.max_memory_reserved(). Process RSS was additionally sampled for the MoE experiment to show the CPU cost of Hessian offloading.

These are memory-focused, reduced-calibration experiments. Both used eight synthetic calibration samples of 256 tokens rather than the 512 samples of 2048

HDCharles

"""Repro: W2A8 humming GEMM produces NaN at large shape_m on SM90.

The NaN only appears at shape_m >= some threshold (around 16384 in vllm's
profiling run). Smaller batch sizes used during CUDA graph capture (1-512)
pass fine, which is why earlier repros missed this.
"""

import subprocess
import sys

HDCharles

import gc
import time

import torch
import triton
import triton.language as tl

SIZE = 844_000_000_0
device = "cuda:0" if torch.cuda.is_available() else "cpu"
N_RUNS = 200

HDCharles

#!/usr/bin/env python3
"""Repro: humming GEMM produces NaN with quantized input activations.

Run: python tests/repro_int8_nan.py

Sweeps all combinations of weight bit width (3-8), input quantization
(int8, int4, bf16), batch size, and shape. int4 inputs require weight
nbits < 4 and SM >= 80.
"""
import json

HDCharles

"""
Benchmark comparing eager vs compiled vs Triton GPTQ quantize_weight.

All three implementations are fully inlined:
  - quantize_weight_eager: the original loop from main
  - quantize_weight_compiled: the torch.compiled block path from this branch
  - quantize_weight_triton: hand-written Triton kernel for the block quantization

Generates random weights and a synthetic positive-definite Hessian,
then times all three.

HDCharles

import torch
from compressed_tensors.offload import init_dist
from compressed_tensors.quantization.quant_scheme import (
    FP8_BLOCK,
    NVFP4,
    QuantizationScheme,
)
from datasets import load_dataset
from transformers import AutoTokenizer, InklingForConditionalGeneration

HDCharles

import gc
import time

import torch
import triton
import triton.language as tl

SIZE = 844_000_000
device = "cuda:0" if torch.cuda.is_available() else "cpu"
N_RUNS = 20

HDCharles

"""Minimal repro: packed integer zero-point failures for uint7/uint8.

The humming kernel handles integer ZPs correctly when tensors are
unpacked. But the packed path (packed=True in prepare_humming_weight
and prepare_humming_zero_point) — which is what vLLM uses when loading
pack-quantized checkpoints — produces wrong results for num_bits > 6.

This causes catastrophic accuracy loss for W7A16asym and W7A8asym
models (20M word perplexity vs ~10 for symmetric).
"""