cosminscn

Science Fiction

This last week, I feel humanity has reached what I dreamed about when I watched IBM's Deep Blue beat Garry Kasparov in 1997.

The more I learn about recent results in mathematics and theoretical computer science, the more I realize we're in superhuman territory.

Two years ago, trekking to Everest Base Camp reminded me of the Romanian countryside of my childhood. 40–80 years ago, my parents and grandparents walked to neighboring villages to buy something, close a deal, or deliver a message. Many households had no car or telephone. Some had horses.

Getting anywhere on foot requires a lot of effort, and I feel like we're in the that place intellectually.

In Plato’s “Republic”, society comprises men of gold, who are fit to rule, men of silver, destined to be soldiers, and those of bronze, who become farmers. Since 1959 the world’s top teenage mathematicians (and, more recently, those of great ability in other fields) have tested their mettle in international competitions called Olympiads. Around 300 medals of gold, silver and bronze are awarded for maths annually. A smaller number are given for physics, chemistry and informatics (ie, computing). From the ranks of the winners has emerged an elite unlike any in the history of capitalism. Behold: Olympiad Man.

The crisis over the safety of artificial intelligence is best understood as a crisis among Olympiad Men. Last year AI models managed gold-medal performances in maths. Relentless advances since have led a pair of researchers at Anthropic, who made up a third of Britain’s maths Olympiad team in 2017, to resign. Anthropic’s boss, Dario Amodei, was once a member of the squad, but not quite the starting line-up

Dwarkesh Patel × Jensen Huang Interview — Key Questions & Timestamps https://www.youtube.com/watch?v=Hrbq66XqtCo

1. The Supply Chain and Hardware Moat

Commoditization (00:06) — If software gets commoditized by AI, does Nvidia's hardware business get commoditized as well, since TSMC and others handle the actual manufacturing?
Scarce Components (04:28) — With over $100B in purchase commitments, is Nvidia's true moat simply locking up years of scarce components (logic and memory) so competitors can't physically build their accelerators?
Scaling Upstream (08:32) — How do you continue to 2x production year-over-year when you are already taking up the vast majority of TSMC's advanced nodes? Are we entering a regime where AI compute growth has to slow down because of the upstream supply chain?

A100 - spec 312 TFLOPS/s

40GB? 80GB HBM ram 20MB cache

	import numpy as np
	from PIL import Image
	import math, os

	# Classic demoscene-style plasma GIF:
	# layered spatial sine waves + a phase-shifting RGB cosine palette.
	W, H = 480, 360
	N_FRAMES = 72
	DURATION_MS = 45

	"""
	Optimized NanoGPT Adder: 66 params + 0 buffers

	transformer.wte.A [10, 1] = 10
	transformer.wte.B [1, 4] = 4
	transformer.h.0.attn.q_proj.angle_h0 [1] = 1
	transformer.h.0.attn.q_proj.angle_h1 [1] = 1
	transformer.h.0.attn.q_proj.scale [1] = 1
	transformer.h.0.attn.k_proj.weight [4, 2] = 8
	transformer.h.0.attn.v_proj.u [4, 1] = 4

	"""58-param nanoGPT that adds any two 10-digit numbers. No training.

	Down from 130 params. HONEST counting: every stored numerical value is an
	nn.Parameter. Zero buffers. The only "free" things are structural choices
	(which dim connects where), control flow, and pure math (sin/cos/arange
	for PE generation — same convention as the original).

	Parameter budget:
	wte.A (10×1) + wte.B (1×4) = 14 [factorized embedding]
	Q (2 angles + 1 scale) = 3 [rotation-parameterized]

	"""
	Dynamic NanoGPT Adder: 130 params + 0 buffers

	transformer.wte.A [10, 1] = 10
	transformer.wte.B [1, 4] = 4
	transformer.h.0.attn.c_attn.weight [12, 4] = 48
	transformer.h.0.attn.c_proj.weight [4, 4] = 16
	transformer.h.0.mlp.c_fc.weight [4, 4] = 16
	transformer.h.0.mlp.c_fc.bias [4] = 4
	transformer.h.0.mlp.c_proj.u [4, 1] = 4

	"""190-param nanoGPT that adds any two 10-digit numbers. All weights hand-coded."""
	import math
	from dataclasses import dataclass
	import torch, torch.nn as nn
	from torch.nn import functional as F

	# === NanoGPT (from github.com/karpathy/nanoGPT/blob/master/model.py) ===
	# Modifications: sinusoidal PE buffer, configurable mlp_hidden, c_fc bias always on

	class CausalSelfAttention(nn.Module):

	# This solver covers a rectangular grid of '.' and 'X' with two non-rotatable tiles:
	# • A (U-shape): on row i it occupies (i,j) and (i,j+3), and on row i+1 it occupies (i+1, j..j+3).
	# • B (horizontal domino): occupies (i,j) and (i,j+1) on the same row.
	# We scan left→right, row→row using top-down DP with memoization. The DP state is
	# (i, j, needA, anchors):
	# – i, j: current row and column being decided.
	# – needA: tuple[bool] of length W marking cells in the CURRENT row that are forced to 'A'
	# because they are the bottom strip of some A placed in the PREVIOUS row.
	# – anchors: tuple of columns where we have already anchored an A in the CURRENT row before j;
	# each anchor at column a also forces (i, a+3) to be 'A' (its top-right leg) and

cosminscn

Science Fiction

Dwarkesh Patel × Jensen Huang Interview — Key Questions & Timestamps https://www.youtube.com/watch?v=Hrbq66XqtCo

1. The Supply Chain and Hardware Moat

Large model run