geohot

Compilers for Machine Learning

A hands-on one semester course where students build their own compiler from scratch, starting from elementwise programs and ending with training SOTA LLMs on GPUs. This course aggressively builds on the previous week, and is an exercise in slop management. If you let any slop in early, it will compound and you will not finish the class.

Course description: This course covers the design and implementation of a modern machine learning compiler, and examines the interaction between IR design, hardware capabilities, and the structure of machine learning programs. Topics covered include term rewriting, code generation, movement operators, kernel fusion, memory hierarchies, GPU architecture, automatic differentiation, and flash attention. It is a project course, providing experience with performance-oriented programming, managing a codebase that grows all semester, and working in 1 or 2 person teams, culminating in a compiler capable of training modern LLMs.

Prerequisites: This c

TalalMash

#!/bin/sh

#Youtube playlist/video archiver, downloads changes on subsequent run, keeps track in archive.txt file.

#Install yt-dlp from pip: pip install yt-dlp --break-system-packages
#Reopen terminal, run `sh ytdl.sh https://playlistvideourlhere <optional command>`
#Use cookies and remove sleep-interval max-sleep-interval for much faster playlist parsing and speeds.
# --cookies ${COOKIES} \
#https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc

akarpov89

using System.Runtime.CompilerServices;
using static ParsingExtensions;

string input = "Name: Andrew; Age: 31";

string? name = null;
int age = 0;

if (input.TryParse($"Name: {Placeholder(ref name)}; Age: {Placeholder(ref age)}"))
{ 

timvieira

"""
Simple example of manually performing "automatic" differentiation
"""
import numpy as np
from numpy import exp, sin, cos


def f(x, with_grad=False):
    # Need to cache intermediates from forward pass (might not use all of them).
    a = exp(x)