Michael Lazos mlazos

Steady-state end-to-end latency

Positive lead means lower Torch latency. Normal rows use the midpoint of fresh-cache Torch and cache-hit Torch medians around a tuned FlashInfer arm. Flagged rows use the valid warmed adjacent pair closest to parity; repeated pairs are retained in the CSV.

Model	Batch	Torch (ms)	Tuned FlashInfer (ms)	Torch lead	Validation	Status
Llama-3.1-8B-Instruct-NVFP4	8	327.314	353.125	+7.31%	1 warmed pair	torch win
Llama-3.1-8B-Instruct-NVFP4	32	350.939	389.881	+9.99%	A/B/A; Torch drift 0.90%	torch win
Llama-3.1-8B-Instruct-NVFP4	64	357.985	379.558	+5.68%	A/B/A; Torch drift 0.13%	torch win
Llama-3.1-8B-Instruct-NVFP4	128	481.231	484.008	+0.57%	1 warmed pair	parity

	## Request: broader grouped GEMM coverage in CUTLASS Operators

	We are working toward grouped GEMM parity between PyTorch’s NVGEMM and Gluon grouped gemm. Our
	initial scope is dense BF16 grouped GEMM on Blackwell.

	The current CUTLASS Operators package exposes one provider for GroupedGemmArguments:
	ContiguousOffset2D3DGemmDenseOperator. It supports:

	- A stacked 2D activation: [sum(M_i), K]
	- Batched 3D weights: [G, K, N]

	# PyTorch Usage of the Legacy CUTLASS Python Package

	> AI-assisted draft prepared for human review. If posted on GitHub, quote the
	> report and accompany it with the reviewer's own commentary, per PyTorch's AI
	> policy.

	## Scope

	This covers PyTorch Inductor's legacy CUTLASS backend at CUTLASS v4.6.1
	(`e05f953a5b3d38adc240df2ff928e0421c2abba3`). It excludes the CuTeDSL and

	---
	CUDA Streams + torch.compile: GitHub Issues

	Feature Requests / Support Gaps

	#: #118204 (https://github.com/pytorch/pytorch/issues/118204)
	Title: streams x torch.compile: stream is treated as None sometimes
	Filed: Jan 2024
	By: zou3519
	Status: Closed

	---
	# User code:
	def fn(x):
	s = torch.Stream()
	with s:
	y = x + 1 # runs on side stream
	e.record() # record_event on side stream
	e.wait() # wait_event on default stream
	return y * 2 # consumer on default stream

	"""
	User-code repro: control_deps mixed-validity partitioner crash.

	Crashes WITHOUT the fix in partitioners.py with:
	AssertionError: Node _unsafe_view was invalid, but is output

	Root cause:
	1. s1.wait_stream(default_stream) generates a wait_stream op in the graph.
	2. _collect_wait_stream_forward_deps scans ALL subsequent ops on stream s1 --
	including backward ops -- and collects their inputs defined before the

	#!/usr/bin/env python3
	"""
	Multi-agent code review tool.

	Runs 4 review agents in parallel (2 Claude, 2 Codex):
	- Claude high-level design reviewer
	- Codex high-level design reviewer
	- Claude low-level bug finder (real bugs only)
	- Codex low-level bug finder (real bugs only)

	CUDA Streams in torch.compile
	We've added native CUDA stream support to torch.compile, enabling multi-stream GPU programs to be compiled and optimized automatically.
	Why it matters: CUDA streams are the primary mechanism for achieving concurrency on GPUs. A stream is a queue of operations that execute in order relative to each other, but can run concurrently with operations on other streams. This enables critical performance patterns like overlapping computation with communication in distributed training, pipelining microbatches across pipeline stages, and offloading activations to CPU during the forward pass and prefetching them during the backward pass. Until now, these patterns required eager execution or careful manual optimization that bypassed the compiler entirely.
	Tracing and representing streams in TorchDynamo
	TorchDynamo traces torch.cuda.stream() context managers by maintaining a symbolic stream stack — pushing on entry, popping on exit — and annotating every FX graph node with a stream_idx metadat

	[DEBUG]:Output code:
	# AOT ID: ['0_inference']
	from ctypes import c_void_p, c_long, c_int
	import torch
	import math
	import random
	import os
	import tempfile
	from math import inf, nan
	from cmath import nanj