20 korrents from this recording
Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.
Their wordsthe model does not seem to be prompt injectable anymore.
Their wordsit's it's literally we're looking at neurons in the model's brain that light up when prompt injection happens. So the model won't even tell you but we can actually see those neurons and we can figure out and diagnose that it's happening. And then you combine that with the auto mode classifier and with these three layers we just cannot demonstrate prompt injection anymore.
Their wordsAnd the reason is every model is very different. So, something that you did for one model maybe 3 months ago, it just might not translate at all to the next model.
Their wordsAnd what's interesting is that the model is actually a little bit more intelligent without these prompts.
Their wordsfor people that aren't building agentic products, but you're using Claude code, every 6 months delete your Claude MD. Delete your skills. Delete your hooks.
Their wordsAnd you don't want to guess what's the instruction that the model needs because you might not predict it correctly. The thing that you want to do is you want to run it.
Their wordsI I think this is sort of the crazy thing about building on models. It's just so different than all the engineering that I've ever done. Like in the past when you built on systems, you built these like big beautiful systems and you really think about the system design up front. You have like a big suite of unit tests. You think about everything and you know, like a re-architecture is a big project.
Their wordsAnd the model is not like that. It's um the way to think about it is almost like a like a living creature, like it's something more organic.
Their wordsI think evals, they outlive the harness a little bit, but not by that much. Like an eval might live for maybe one, two, three model generations, but nowadays the you know, we're on the exponential. The model is improving so quickly, very often we just saturate the eval, and then we have to throw it away, and we have to come up with a new eval.
Their wordsBecause the model can do this at every given model generation, but there is often not a product that lets the model do this and lets it express this kind of ability to do this.
Their wordsOne is you should give the model slightly harder tasks than what you think it can do.
Their wordsone example is the model can now rewrite essentially any code base from one language to a different language.
Their wordsAnd my hypothesis is there's probably dozens, hundreds of opportunities like this with the models of today that no one has yet realized.
Their wordsI think the skill nowadays is less about prompt engineering and more about figuring out how do you give Claude a hard task that seems a little bit too hard. And then how do you make it possible for Claude to verify its work along the way? And the verification I think is probably the single most important thing that people do not get right
Their wordsYou don't need slash goal, you don't need slash loop. These help, but really all you need is give the model the task, give it a way to verify the output of its work so it doesn't get stuck, and it will just go.
Their wordsSo when I look at engineers that have been, you know, coding for a long for a long time, you know, like for for years or for decades, this is a really really common failure mode is trying to over specify and it's trying to be overly specific and then, you know, get the model to do the to do the task exactly the way that you would have done it. And that that's just not the way the model works.
Their wordsLike this is actually like a new form of test time compute. Like when we talk about the scaling laws and kind of we talk about the model getting more intelligent over time, historically, it's been a function of the size of the neural net, the amount of training data, and the number of flops that you put in to the training. And then recently, we also added test time compute. So this is essentially a fancy way a researcher way of saying how many tokens does it generate. And now dynamic workflows are essentially a new way to orchestrate test time compute.
Their wordswe're on the path to fully automating the maintenance of our apps by doing this. And this is again hundreds of agents running every day, sometimes thousands of agents every day. It's doing the work of you know, dozens or hundreds of engineers.
Their wordsSo coding is solved for the kind of coding that I do. It's not solved for everyone. You know, there's still code bases that are like super deep systems code bases where quad still struggles.
Their wordsAnd it's really about being empirical. So forget all of the things that you learned about past models. Forget everything that you've learned about computer science theory in class. Look at the model, try to do a task, see where it struggles, and then based on that adjust. So it's just like very much become it's not a theoretical science, it's become an empirical science.