Martin Casado × Dhaval Adjodah & Owen He (MakerMaker.AI) — recursive self-improvement — denser notes · Don't Panic Digest More
What this is: Nearer-verbatim notes from a public X interview video. Not a court transcript. Gaps labeled. Disfluencies lightly cleaned. Hosting ≠ endorsement.
Not Code Solutions LLC viewpoints or advice. Speakers' claims stay speakers' claims ([unverified] / host-said).
Hedge legend: [inaudible] · [unclear: …] · [unverified] for numbers and company claims · host-said for Martin Casado framing.
Speaker claims: Statements by the guests are their own account [unverified] — not checked, not restated as fact. Examples: that recursive self-improvement (RSI) is "happening" at their company with less energy for better results; that they have seen results "grow and grow" "over the past few months"; that GPT-2 can now be trained for about $100 on a single GPU within hours (nanoGPT benchmark, as told by Owen He); and the compute upper bound from roughly 10^80 particles times the speed of light. Timestamps are approximate (player time); within a turn they mark the nearest preceding cue.
Source URL: https://x.com/a16z/status/2109666842101072286
Full video duration: 32:06 (native X) · Scope: full video 0:00–32:06 (00:00–00:58 is a teaser montage of clips repeated later in the interview)
Host: Martin Casado (a16z) · Guests: Dhaval Adjodah and Owen He (co-founders, MakerMaker.AI)
Posted: Mon 2026-10-12 · Notes desk: 2026-10-13
Contest OUT · Third-party names (DeepMind, MIT Media Lab, nanoGPT, GPT-2, GPT-5, Kimi K3 [unclear], B200, Slack, etc.) kept verbatim when spoken.
Disclaimer: https://legal.codesolutionsllc.com/si-disclaimer — Notes of a public interview — not Code Solutions LLC views or advice.
Lane: Digest More feed only. Not a Daily X release from this pack alone.
[00:00] Dhaval Adjodah
The fundamental idea here is to build AI that improves itself, and I think we're getting there.
[00:00] Owen He
For the first time, we could swap the optimizer as the optimizee, and this leads to a reinvestment of cognitive improvement into itself.
[00:00] Dhaval Adjodah
You can think of RSI at many different levels of the stack. Prompts, harness, weights, you can go all the way to basically chips. Our goal here is to optimize all the levels of the stack.
[00:00] Owen He
Our approach to RSI is that we have a multi-agent system working together that's being recursively improved.
[00:30] Martin Casado
I am so impressed by your belief that like, this idea of a model improving itself...
[00:30] Dhaval Adjodah
But it's not a belief.
[00:30] Martin Casado
That is a dramatic statement, which is I think it's the first time I've heard you say it.
[00:30] Dhaval Adjodah
Because we have been so thorough, we have not released things until we're super sure it's correct.
[00:30] Martin Casado
Do you envision a system where the amount of energy that goes into it somehow decreases over time relative to the improvement?
[00:30] Dhaval Adjodah
We can't share too much, but one of the very important things that we've tried to do...
[01:00] Martin Casado
Welcome both of you for joining, we're going to be talking about everybody's favorite topic, which is RSI, recursive self-improvement. Maybe to start it'd be great to just introduce yourselves and your backgrounds. Then we'll talk a little bit about what you're working on now, and then we'll dig into the content.
[01:00] Dhaval Adjodah
Yeah, I think of myself as like a physicist of some kind. I studied that, then I did a PhD in reinforcement learning at the MIT Media Lab. I really like building things and I'm very excited to be building a cool startup MakerMaker with Owen and Grace.
[01:30] Owen He
My name is Owen. I was first studying math, and then one day I realized that all mathematicians do is proving theorems. And I was taking a class called Formal Language and Logic, and they were showing how you can build a theorem prover. And I was like, "Oh, that would be cool. I can just build a machine that proves theorems for me." That's how I got into AI.
[01:30] Martin Casado
Amazing.
[01:30] Owen He
And then for my PhD, I worked on continual learning and meta-learning, then worked for a few years at DeepMind as a research scientist. Now I'm working on RSI.
[02:00] Martin Casado
I see a lot of pitches come in, and a lot of them use the term RSI. And I'll be like, "Oh, so what do you do?" And they'll say, "We use AI to create AI." That sounds very interesting. "So what do you do?" "We use, yeah, AI to create GPU kernels," which that doesn't, to me, kind of map to the definition of what I would consider to be RSI. So maybe can we start by you defining what you think, or what you would include in RSI, definitionally?
[02:30] Owen He
Yeah, I think RSI is definitely an overloaded term today. Many people when they say RSI they mean different things. So I think the important thing when you are differentiating them is to ask the question: what is the optimizer and what is the optimizee? Like, when you're self-improving, who is improving, right? So there are people who are saying, "Okay, we're doing RSI, we have humans using AI to develop better AI." But that to us is just auto-catalysis.
[03:00] Owen He
In the sense that the improver is still human with tools, even though these tools are getting better and better, but the ideas and the improvement come from human brains.
[03:00] Martin Casado
Right. And just, I don't mean to interrupt, but to be clear, most technical epochs have had auto-catalytic effects, right? Like there's no way you can create a computer chip without a computer, right? You could argue that the internet development was sped up by the fact that people that weren't geographically located could coordinate, right? I mean, the steam engine, you could build better steam engines 'cause you could get coal faster, etc. So this notion of like using the thing to build the thing faster is not new. And so you're saying something different.
[03:30] Owen He
Yes. I think for us, in the history, right, of our civilization, we have this auto-catalysis, you know, computer using computers to build better computers, but the ideas for this change, for this improvement, come from human brains. But that human brain was never the improvee, right, never the optimizee.
[04:00] Owen He
Whereas right now, for the first time, we could swap the optimizer as the optimizee. So the brain of the agents, which is the LLM, the model, can be the thing that you're optimizing. And this leads to a compound... this reinvestment of cognitive improvement into itself.
[04:00] Martin Casado
But there has to be a reason that you decided to take on this hard problem as opposed to what anybody else would do, which is just like, chew off pieces of the problem.
[04:00] Dhaval Adjodah
Yeah. I mean, I guess before I answer your more important question here, the reason I did this is I think it's a hard problem and it's fun working on the hardest problem. And I think...
[04:30] Martin Casado
But do you have an intuition that this is the actual right answer as opposed to like something that's more pragmatic?
[04:30] Dhaval Adjodah
Yeah, I think a pragmatic thing would be, just from a control theory perspective, if you have a gap in the system, you can't have a closed loop. And auto-catalytic has no closed loop, right? So whatever you improve, there's still like a gap. Humans have to come in, you have to pour in other resources to make it work.
[05:00] Dhaval Adjodah
But if you have a closed loop, you still have to pour in resources, but the marginal gains ideally get either constant or larger over time. And that's the fundamental difference.
[05:00] Owen He
What's really different here can be explained pretty easily. So if you think of capability, what we're trying to do here is we're trying to improve the rate of improvement of capability. Right? So like the first derivative of capability with respect to time is the thing that everybody's trying to improve.
[05:30] Owen He
Now you can put that to be equal, driven by two things: the numerator and the denominator. The numerator would be performance, which is what everybody is trying to improve. And then the denominator is some kind of cost, let's say capital, right, or difficulty. And this one is pretty set, because the higher the capability, the higher the difficulty of the next step, right?
[06:00] Owen He
Now the numerator is the interesting part. For things like steam engines and what not, that was fixed. Right? So then you end up with a very simple differential equation where you basically plateau. Right? Now, if you have a system where the improvee, the thing being improved, is also the thing that's doing the improvement, you now have a new way to think about it. And the simplest case would be that the numerator is also correlated with capability. Which makes sense, right? Because the better you are, the better you are at also improving.
[06:30] Owen He
And if you do that, the math cancels out and you basically get constant improvement over time. That could be one level of RSI. Right? And then there's one more case which is now that the numerator could be correlated with a higher power of capability.
[06:30] Owen He
And then you get exponential takeoff.
[06:30] Martin Casado
Yeah. Can we dig in a little bit more between RSIs having closed the loop and actual convergence properties?
[06:30] Dhaval Adjodah
Mm.
[06:30] Martin Casado
Like for some reason when people see a graph where there's like a circle with an arrow to another circle, and it goes back, they look at that closed control system and they say "takeoff".
[07:08] Dhaval Adjodah
Yeah.
[07:08] Martin Casado
Right? Which, I mean, if this is a non-linear control system, it's very difficult to know what the hell happens. By the way, most people don't seem to understand this distinction, because we very quickly go to like "takeoff" and "foom" and all that stuff when we talk about these things.
[07:08] Dhaval Adjodah
Well as a network guy, you know that it depends on the exponent of your function that's self-referencing. And what we found is that it's almost not useful to speculate on what we think could happen, because this is really an empirical problem. And the real way to know what's happening is to test things. And testing things is really, really hard. Like reward hacking is one of many problems that we have to deal with. But the question you're asking is very empirical, and for some algorithms...
[08:03] Martin Casado
Oh interesting. So I'm a networker, that's how I would have answered. I would have thought math people would have been like, "We've got this differential equation and I took out my pencil and I wrote down the answer."
[08:03] Owen He
I mean it is possible to write down the differential equation, but you still have to empirically measure the variable, the exponent.
[08:03] Martin Casado
So there's no analytical solution.
[08:03] Dhaval Adjodah
I mean honestly we don't even have a learning theory of deep learning. We don't know like fundamentally why it works. Right? And now you're like five, six layers on top of deep learning, and so we don't have a good theory. I think the closest things we have are scaling laws, and these are empirical laws.
[08:44] Owen He
I think one misunderstanding people have about our approach to RSI is that they think the entire system has to be fully automatic and digital. But actually if you look at our mathematical foundation, we have a multi-agent system, not just one digital agent. And in this multi-agent system, human researchers and engineers are actually special cases of these agents. And it's this hybrid system of multiple agents working together that's been recursively improved. And we don't want to leave out any component to not optimizing them, because if it's possible to optimize the model, why wouldn't we do that? Because you will have more gains if you do include those. Anything that you don't optimize will later become your bottleneck.
[09:28] Martin Casado
So one observation that we have made here is, there are claims of speed is improving, which is actually the case, right? So if you look at like the model release time, it is reducing. And so on one hand you could be like, okay, so clearly we're in some sort of a rapidly improving phase. But if you divide by the actual amount of capital raised by these companies, that's no longer the case, right? And so you're like, well maybe they just raise a whole bunch of money and then like AI allows you to apply that money.
[10:02] Martin Casado
So how would you, this is not for yourselves, but how would you encourage people that are either evaluating claims about this or even working on them, to think about not fooling yourself by throwing more money at the problem? Like you're actually showing that you're making an improvement here that's meaningful with the denominator.
[10:02] Dhaval Adjodah
I think for us it's pretty straightforward. It's because if you get improvement and they just come at more and more resources being poured in, it's not RSI. You have to be able to back your claim of how much improvement you're getting...
[10:38] Martin Casado
But do you include like all the man hours and experimentation that went into that? Because I feel like at some level, it's like talk about the actual flops that go in and the money that go in, but the reality is this is like the 23rd experiment and there was actually a human spending a lot of mental capital to get to that final configuration. Like do you actually include that in the denominator?
[10:38] Dhaval Adjodah
We do. And I think it's good company discipline to do that.
[10:38] Martin Casado
Yeah, yeah.
[10:38] Dhaval Adjodah
And we've tried again to very intentionally from the beginning, have a culture of tracking where things go, cost, time, all that kind of stuff. I'll share a funny thing, internally we have this thing... people talk about leaderboards, you know, which is like score...
[11:23] Martin Casado
Yeah.
[11:23] Dhaval Adjodah
You know what I'm about to say. We have something called a bleederboard [unclear: word heard as "bleederboard", a play on "leaderboard"; ASR renders it "leaderboard"].
[11:23] Martin Casado
Mhmm.
[11:23] Dhaval Adjodah
Where we keep track of the cost of experiments, of everybody who's running experiments.
[11:23] Martin Casado
That's awesome.
[11:23] Dhaval Adjodah
And everybody any day can look at like who's high up on the bleederboard.
[11:43] Owen He
I can also give an example from AI. I don't know if you know this nanoGPT benchmark people are running.
[11:43] Martin Casado
Yeah.
[11:43] Owen He
It's basically asking you to retrain GPT-2, which when it just came out is a very expensive model to train and takes weeks I think on multiple GPUs, and right now people can train it on a single GPU within hours. And that number has been shrinking constantly.
[12:08] Martin Casado
But that's also totally separate, right? Like that's just like whatever, Moore's Law, right?
[12:08] Owen He
No, no, it's the same GPU.
[12:08] Martin Casado
It's the same hardware.
[12:08] Owen He
Using the same hardware and then you improve the techniques of retraining.
[12:08] Martin Casado
Oh interesting.
[12:08] Dhaval Adjodah
Yeah, what has improved is like algorithmic techniques, data techniques, kernels, all of that kind of stuff.
[12:08] Owen He
Do you know how much it costs to train GPT-2?
[12:08] Dhaval Adjodah
I think it's a few hundred dollars now.
[12:31] Owen He
It's a hundred dollars. I just saw this recently.
[12:31] Martin Casado
Oh wow, so the same GPU...
[12:31] Owen He
Same GPU, same data.
[12:31] Martin Casado
And this is just algorithmic and where did those come from, humans or did those come from?
[12:31] Owen He
Agents, a lot of them are auto research results.
[12:31] Martin Casado
But that's definitely the denominator. That's not the numerator. Right? What you want to see is like, you can do that with GPT-5, which we're not seeing. Right? Like the cost of these things is going up dramatically.
[12:31] Dhaval Adjodah
Well I think one of the most closely guarded secrets in the industry is...
[13:00] Dhaval Adjodah
exactly this thing which is how much compute you can get out of this.
[13:00] Martin Casado
Yeah.
[13:00] Dhaval Adjodah
Because in some sense even if all you were improving was your denominator, you could get faster training because now you can get more rollouts.
[13:00] Martin Casado
Right, yeah, of course.
[13:00] Dhaval Adjodah
Right? And that means now you can sample, what you were describing, like the manifold of the space of language more extensively, and then you can get better trajectories which means you can get smarter models when you use that to train.
[13:00] Martin Casado
For sure improving the denominator is good.
[13:00] Dhaval Adjodah
Yeah. He: Yeah.
[13:00] Martin Casado
For sure improving the numerator is good.
[13:34] Martin Casado
I have yet to see, which is great that you have, you know, glimmers, where you can improve both the numerator and the denominator. I mean that's a pretty meaningful statement which is awesome.
[13:34] Dhaval Adjodah
What we found is that a lot of the papers that claim to do RSI, when we test it in our internal infra, we find that they don't work. And a lot of it is because they're cool ideas that humans try to introspect and say, "Oh, you know, this is what I think should happen."
[13:34] Martin Casado
Yeah.
[13:34] Dhaval Adjodah
They're complicated, they sound cool, they get you a good paper, but then when we test it at scale, on benchmarks, on real economic problems, they don't scale.
[14:09] Dhaval Adjodah
And what we found is that the simpler thing that closes the loop is the thing that scales. I would say one thing that we do that's probably unique is that we always start with a very grounded mathematical framework that basically lays out the map of the whole space.
[14:09] Martin Casado
Yeah.
[14:09] Dhaval Adjodah
And the reason is because this space is so large that if you don't do that and you don't have this very principled way of doing it, you get lost very quickly in this complexity.
[14:38] Dhaval Adjodah
And the other reason is that often when you have these really complex systems, there's two extremes of how you can build it. You can take an approach where you basically build like 200 different systems and you hope that, what is it, the Rube Goldberg machine will kind of work out.
[14:38] Martin Casado
Yes, yeah.
[14:38] Dhaval Adjodah
Or you take the other way and you basically spend some time thinking about these 200 things and maybe you realize they're actually just 12 abstractions that generalize to these 200 things.
[15:11] Dhaval Adjodah
And so what we have tried to create a culture of, let's really think from first principles, let's look at the math and then we realize what are the assumptions we make to be practical because you have to do both, and then we see what works. And when you do that, you then can see that, "Oh, these multiple papers or ideas are actually subclasses of the same general framework."
[15:11] Martin Casado
Yeah.
[15:11] Dhaval Adjodah
And if you do that well, you can scale.
[15:41] Dhaval Adjodah
And I think that's one thing that I see that's missing and that's hard to do and takes a very unusual group of people to really make that work.
[15:41] Owen He
I think Richard Feynman said once that there are two kinds of sciences, there are stamp collections and then there are physics. So without a unified math framework, all these research are sort of just stamp collections.
[16:07] Owen He
A lot of these papers we're reading, they're just ideas inspired by some human behavior and then they try to mimic that and say, "Oh look, we have RSI." But then if you don't put it into a unified system, you don't know if the overall system is actually improving or you're just locally applying an idea that's getting better. So what we have is that we're putting all these components that you could possibly recursively improve into a unified framework and look at it from a holistic view so that improving one part does not negatively impact the other part.
[16:07] Martin Casado
Right, I see, I see.
[16:44] Owen He
So for example, there are people who are doing recursive improvement just on harness and then there are people who are only doing on weights. But weights and harnesses are not isolated components, they have impact on each other.
[16:44] Dhaval Adjodah
You can think of RSI at many different levels of the stack, right? You can think of prompts, harness, weights,
[16:44] Martin Casado
Yep.
[16:44] Dhaval Adjodah
and you can go all the way to basically chips, right?
[16:44] Martin Casado
Sure, yeah.
[16:44] Dhaval Adjodah
And our goal here is to optimize all the levels of the stack, right?
[17:12] Dhaval Adjodah
And we started with harness, we've tried to be always very capital efficient because we care so much about the denominator. And then we started doing weights, weight updates, and what we found is that you get more gains but you still have to do harness stuff, right? And you can keep going down the stack and my guess is that you probably will need to do a lot of the kind of co-optimization of many different parts of the system to make it work.
[17:42] Martin Casado
Can you talk a little bit about the problem domain that you're going after just to prove that this works?
[17:42] Dhaval Adjodah
We are trying very hard to pick areas of the AI space that are as close as possible to AI itself. Again, we try to be recursive.
[17:42] Martin Casado
Yeah, yeah.
[17:42] Dhaval Adjodah
And if you think about it, there's not that many problems that are super close to AI and give you improvement on both the numerator and denominator. And one of those is the inference problem.
[18:13] Dhaval Adjodah
Right? Because the more you can get improvements and efficiency out of inference, the more everything kind of unlocks, right? And it's not just for technical reasons, but also the most promise of both the numerator and denominator being improved, right?
[18:13] Martin Casado
It kind of breaks the mind, right? So just for everybody that's listening to this, there's two improvements going on here. One of them is the ability of the model to make inference faster, and then the second one is, now that the inference is faster, it can do that faster.
[18:50] Dhaval Adjodah
Exactly.
[18:50] Martin Casado
And again, that does not necessarily mean takeoff, that can still asymptote at some point, but it's just wild to think of this kind of compounding impact.
[18:50] Dhaval Adjodah
Yeah. We're a startup, we're not a research lab. And one of the reasons we chose it is because everybody in the world right now wants to serve their own model.
[18:50] Martin Casado
Yeah, sure.
[19:13] Dhaval Adjodah
And everybody wants to use the latest, for example, open weight model, Kimi K3 [unclear: model name as heard; not verified] or whatever, just coming out, and they want to be able to serve it as efficiently as possible. Right? So it's an economically extremely important problem right now.
[19:30] Martin Casado
Just for the sake of discussion, if I wanted to make the inference for Kimi K3 faster, would I use Kimi K3 in a recursive way, or would I use a separate model and system to do that for Kimi K3?
[19:30] Dhaval Adjodah
To improve the serving stack of Kimi K3 you mean?
[19:30] Martin Casado
Yeah.
[19:30] Dhaval Adjodah
Well, our system is not just using one LLM. You have many different LLMs that are doing many different things.
[19:30] Martin Casado
Oh, so are you doing the inference stack for all of them?
[20:00] Dhaval Adjodah
I mean we pick whichever is the most effective and because it's being used the most time. But the interesting thing is...
[20:00] Martin Casado
I see. Yeah.
[20:00] Dhaval Adjodah
...the choice of which LLM to use for which thing is also not always up to us. We let the model choose.
[20:00] Owen He
To be honest, today it's not really us who're deciding what to optimize. It's our agent.
[20:00] Martin Casado
Interesting.
[20:00] Owen He
Because it's recursively self-improving. And what we're trying to do is not to limit itself to only one component.
[20:30] Martin Casado
So it chooses whether to...
[20:30] Owen He
It chooses what to...
[20:30] Dhaval Adjodah
Yeah, we try to be as hands-off as possible, let it do its thing.
[20:30] Martin Casado
Do you have like a single master prompt? Is it like, "improve yourself"?
[20:30] Owen He
But even that master prompt has been improving.
[20:30] Martin Casado
[laughs] Really? It rewrote the master prompt?
[20:30] Dhaval Adjodah
Yeah. And it's not one RSI system. Because you can think of it as there's the thing that does the improvement, but then you also have to have the thing that knows what to improve. And both are dynamic problems. And they are very co-linked, right?
[21:00] Martin Casado
How general do you think this ends up being? Like right now your focus is on speeding up inference in a specific domain. Do you think that whatever approaches you're coming up with here generalize or do you think there'll be a different one per target solution space?
[21:00] Dhaval Adjodah
Mathematically, we have defined it so that it's one system that generalizes to everything. I think the one part of the system...
[21:30] Martin Casado
And by everything you mean any problem domain that you can point AI to.
[21:30] Dhaval Adjodah
Correct. But then there are physical limits, for example. And one big limit of this is the human element. I think the way that a lot of AI companies are thinking is, let us use AI to replace humans. But what we found is that there's so much that if you have good researchers and good engineers...
[22:00] Dhaval Adjodah
...that they can bring to the table that's gonna be very hard to replace by agents. And so that's why from the very beginning we've tried to think of basically our entire company as a multi-agent system where humans are a kind of agent that's cohabitating with other AI agents.
[22:00] Martin Casado
Can we at least reduce bounds? So for example, do you envision a system where the amount of energy that goes into it somehow decreases over time relative to the improvement?
[22:30] Dhaval Adjodah
We can't share too much, but one of the very important things that we've tried to do in our company is not just work on the numerator, which is what I was describing earlier, which is your performance, but to work on the denominator, which is how much you can get out of the same amount of physics, money, whatever you want to call it.
[22:30] Martin Casado
Right. But do you think it's possible to have a system where the numerator continues to increase the same way with the denominator being fixed or decreasing?
[23:00] Dhaval Adjodah
It's not just possible, I think it's happening.
[23:00] Martin Casado
Really?
[23:00] Dhaval Adjodah
Yeah.
[23:00] Martin Casado
So it takes less energy to get better results.
[23:00] Dhaval Adjodah
Yeah, yeah. Over time.
[23:00] Martin Casado
Do you think there's a universe in which that continues without asymptoting? Because if that was the case, then you actually have takeoff. I mean, that just seems to disobey information theory and lots of physics...
[23:00] Dhaval Adjodah
Yeah. I mean, I think at some point you're going to reach Planck constant, you know, Planck length, Planck energy, Planck time, right? So it will asymptote at some point.
[23:30] Dhaval Adjodah
But I think...
[23:30] Martin Casado
Oh, I see. And so you think that asymptote is basically like the computation of the universe.
[23:30] Dhaval Adjodah
At least, yeah. You know, how many particles are there, 10 to the 80. We know speed of light. You multiply those two and you basically get an upper bound of computation. And so given that fixed amount of computation you can do every second, we know we have a limit.
[24:00] Martin Casado
I am so impressed by your belief that this idea of a model improving itself...
[24:00] Dhaval Adjodah
But it's not a belief, it's happening.
[24:00] Martin Casado
No, [unclear] that's amazing.
[24:00] Owen He
I mean, also it's a question of what metric you use to view the progress. Right. So if you use, I don't know, benchmark accuracy, it's upper bounded by 100%. You're never going to see exponential improvement on that, because eventually you bottleneck at 100%. So for a long time we had this question, how do we even know if something has taken off, right, if we don't have the right metric.
[24:30] Dhaval Adjodah
It's not a belief that this is happening. I think we're already seeing glimmers of that happening.
[24:30] Martin Casado
But that is a dramatic statement, which is, I think it's the first time I've heard you say it. Like, has this been kind of a continuous result, or is it relatively recent that you're starting to see glimmers of this?
[24:30] Dhaval Adjodah
No, we've been seeing these results kind of grow and grow and grow over the past few months.
[25:00] Martin Casado
How do you have confidence in the results given, you know, like it's not working yet and you don't know...
[25:00] Dhaval Adjodah
No, but it's not a belief, it's empirical measurement. But also as a company there's a very funny thing that is happening, which is because we have been so thorough, we have not released things until we're super sure it's correct. Right? And every few months there's like an announcement of something RSI related. We look at it, we're like, oh, we should have released our results or something, and then we try to replicate it and we realize, oh, it's noise, oh, it's cherry-picking.
[25:30] Dhaval Adjodah
And so it's very, very hard to know in this sea of, I don't know what you want to call it, like everybody's trying to do AI RSI stuff, to know what's legit or not. The technical answer is you have to be really, really good at measuring things. So for example, we have one of our engineers that we named him the Viceroy of Variance. That's a real thing. Like his job is to...
[26:00] Dhaval Adjodah
...to go and like destroy the results we have and find what's wrong. And we even have little Slack emojis, there's one where there's like red laser stuff coming out of his eyes when he finds that there's bad variance and stuff. One of the things that we found is that when you're dealing with a new benchmark or a new problem of economic value that you want to apply this to, so far we have not found a replacement for what we call quality time. Literally running hundreds of experiments, looking at it...
[26:30] Dhaval Adjodah
...developing an intuition, trying to figure out what is wrong, what is not wrong, what is just variance, what is not. What is this weird thing that actually shows that there's some kind of reward hacking like two levels down, the agent that built the agent that built the agent...
[26:30] Martin Casado
Yeah.
[26:30] Dhaval Adjodah
...did hack something and now you have these weird results.
[26:30] Martin Casado
Right.
[26:30] Dhaval Adjodah
It's not easy.
[26:30] Martin Casado
So when I have dealt with non-linear control systems, in networking there's a bunch of degenerate behaviors that you deal with. It's not just convergence and divergence, you have things like oscillations.
[27:00] Martin Casado
And this is a very multi-dimensional problem. So do you have to deal with all of that stuff, like potential oscillations and...
[27:00] Dhaval Adjodah
Absolutely.
[27:00] Martin Casado
Meaning, it'll get good, and then it'll get bad, and it'll get good.
[27:00] Dhaval Adjodah
Absolutely. And then on top of that, you have noise.
[27:00] Martin Casado
Yeah.
[27:00] Dhaval Adjodah
LLMs are fundamentally super stochastic. So what we find is that a specific agent on a specific benchmark, there's levels of noise that sometimes are unpredictable and some of them are extremely...
[27:30] Dhaval Adjodah
...like, weird. Like you even have some weird noise that comes from this GPU and this other GPU, they are both B200, but they have slightly different behaviors, like on the physics, and you can see some of that being replicated at the high level. So that's why it has to be an empirical thing.
[27:30] Owen He
So from that you could already see that intelligence is not a single scalar. It's more like a multi-dimension vector. And there will be domains and fields where this RSI will give you a takeoff, and there will be dimensions where you might plateau, or you will have relatively slow improvement.
[28:00] Owen He
And I think the defining factor is what is the slowest component in it. So you're an optimization person. You know that there's this Amdahl's Law where the speedup you can get by optimizing a component is upper bounded by that fraction, right?
[28:00] Martin Casado
[unclear] yeah.
[28:00] Owen He
You just take it to zero, yeah.
[28:00] Martin Casado
Yeah, nothing else dominates it.
[28:00] Owen He
Exactly. So the digital part we could optimize and accelerate, you could get like a thousand times faster or a million times faster.
[28:30] Owen He
But if it's only 20% of your overall time, you're only going to get 20% speedup, right? So for domains where you still need to move atoms that interact with the physical world, that's really slow, you're still going to be bottlenecked by that. So let's say every evaluation you need to do, let's say drug discovery, you still need to do wet lab experiments, and then clinical trials, and that take months. Then every new idea has to go through months or multiple months of evaluation.
[29:00] Owen He
So those fields probably won't be accelerated as fast as coding or mathematics where you have very fast and cheap evaluations.
[29:00] Martin Casado
But I've yet to see, has anybody put a bound on that? Like, to say, in order for me to improve this by X, to solve this problem, I need this much net new energy to do it, right?
[29:30] Martin Casado
And if you had that, you could actually predict, like how big of a warehouse do I need to solve a drug discovery problem, or how many different robots I would need to solve. So do you think that we're getting any closer to actually having bounds, because I think this is such a critical question for all of us to answer.
[29:30] Dhaval Adjodah
One of the internal jokes we have in the company is that the secret to RSI is somewhere in a garden hose company in Wisconsin. And the joke here is that if Einstein hadn't discovered relativity...
[30:00] Dhaval Adjodah
...he would not have these laws that tell you how speed and contraction of time works. And if some unfortunate engineer had to design the GPS system, they would have had to find like a correction of time. And they probably would have said like, no, no, it's clearly not the fabric of spacetime that's contracting, right? But I don't know why, but this is the formula I've devised empirically.
[30:30] Martin Casado
Some absolute, you know, like some fudge factor.
[30:30] Dhaval Adjodah
Yeah, exactly. Right. And then people later would have been like, holy [expletive], this is like spacetime contracting, right? And we have this theory that RSI right now is being thought of, let's solve all these super hard, kind of fashionable problems. But what we think is that the complexity of the world is such that...
[31:00] Dhaval Adjodah
...the real challenges probably exist in spaces that are not being applied to, right? And so that's the joke, that there is some workflow somewhere in a garden hose company in Wisconsin that if we solve it, we get, you know, foom.
[31:00] Martin Casado
I love that. I just want to quickly summarize. So you're going after RSI. In your case this is not auto-catalytic effects, this is actual RSI where the system improves the full system. And for you it's an actual system which includes the harness and the weights.
[31:30] Martin Casado
If you think about the problem as you've got a numerator and a denominator, the numerator is like how good the system gets and the denominator is how much energy goes into it, you believe you're seeing glimmers of the sun where both improve at the same time using this type of approach. Is that correct?
[31:30] Dhaval Adjodah
Yep.
[31:30] Owen He
Yep.
[31:30] Martin Casado
When can we expect to hear more from you?
[31:30] Dhaval Adjodah
Very soon.
[31:30] Martin Casado
Amazing. Thanks so much both of you for joining.
[31:30] Owen He
Thank you.