geohot

Compilers for Machine Learning

A hands-on one semester course where students build their own compiler from scratch, starting from elementwise programs and ending with training SOTA LLMs on GPUs. This course aggressively builds on the previous week, and is an exercise in slop management. If you let any slop in early, it will compound and you will not finish the class.

Course description: This course covers the design and implementation of a modern machine learning compiler, and examines the interaction between IR design, hardware capabilities, and the structure of machine learning programs. Topics covered include term rewriting, code generation, movement operators, kernel fusion, memory hierarchies, GPU architecture, automatic differentiation, and flash attention. It is a project course, providing experience with performance-oriented programming, managing a codebase that grows all semester, and working in 1 or 2 person teams, culminating in a compiler capable of training modern LLMs.

Prerequisites: This c

HassanElDesouky

Contribution Starter Guide

Introduction

This guide aims to help you start your first PR on Swift! The guide contains links for blogs and talks that are ralated on how to start contribution to Swift.

Compiler Pipeline

It's very important that we understand the general pipeline of the compiler. This is useful to understand where new changes or where to fix things should go. In each section, I'll show you an example of what the process is doing, and at the end of each section I'll give you an error that happens in that stage.

Lexer

mikeash

import Cocoa

enum CoroutineState {
    case Fresh, Running, Blocked, Canceled, Done
}

struct CoroutineCancellation: ErrorType {}

class CoroutineImpl<InputType, YieldType> {
    let body: (yield: YieldType throws -> InputType) throws -> Void