korrents

korrents · video

Stanford CS146S: How always-on agents work

Lee Robinson · 45m · youtube.com ↗

22 korrents from this recording

Lee Robinson did not write this page.

Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.

  1. 5:06 · watch on youtube.com↗

    a separate AI model that's checking all of the commands that get run. So, you know, kind of funnily enough, that's actually more secure than the fatigue of having to watch hundreds of shell commands and approve each one.
  2. 5:20 · watch on youtube.com↗

    We've kind of reached the point where if you did a blind test with what the AI model's checking on the commands and what a human's checking on the commands, most of the time the the model's actually doing a better job.
  3. 1 min later
  4. 5:50 · watch on youtube.com↗

    And I think the main reason is because we all know how to text, you know, everyone knows how to text, everyone is pretty proficient with using their phones, and we've kind of distilled down this way of working with AI that feels like you're just texting a friend
  5. 2 min later
  6. 7:25 · watch on youtube.com↗

    It's funny, you don't really hear people talk about hallucinations anymore, because it's kind of a solved problem, mostly.
  7. 7:54 · watch on youtube.com↗

    otherwise known as computer use, which basically means that any SaaS product or any website that didn't expose an API or an MCP server can be used headlessly, basically.
  8. 7 min later
  9. 14:50 · watch on youtube.com↗

    Um but the most critical thing here is that if you put the agent loop on the virtual machine, then you can't shut down the virtual machine independently from the uh from the agent server.
  10. 2 min later
  11. 16:50 · watch on youtube.com↗

    So, what you really want here is you want a durable workflow, which means that it's safe for you to actually retry that, and when you retry after a failure, it picks right up on step three.
  12. 5 min later
  13. 21:31 · watch on youtube.com↗

    In general, the principle here really is to try the cheapest option or the most reliable option first and kind of work your way up the ladder here.
  14. 4 min later
  15. 25:57 · watch on youtube.com↗

    But, if you see the model doing, you know, 90% of its time doing shell commands for this one thing, it actually can be more efficient to make a dedicated tool for some of that stuff.
  16. 26:17 · watch on youtube.com↗

    Like, okay, this thing failed. Like, click on this link or here's what you need to do next. Turns out that's also very helpful for agents.
  17. 1 min later
  18. 26:51 · watch on youtube.com↗

    It would be great if, you know, when these untrusted inputs like a email in your inbox could be a phishing email or some webhook, you don't want to send that to the model and act like it was a user instruction.
  19. 1 min later
  20. 27:26 · watch on youtube.com↗

    Okay. Context engineering is al- also known as harness engineering. Like, I feel like it's kind of the same thing. They've evolved over time.
  21. 4 min later
  22. 31:45 · watch on youtube.com↗

    eventually it's going to hit a point where it needs to summarize and all compression is lossy. So you have to figure out the best algorithm to do the compression and not lose important details
  23. 6 min later
  24. 37:51 · watch on youtube.com↗

    And what we mean by this is you kind of have to internalize and assume that the models today are the worst they'll ever be, which means that in 6 months uh you might need to completely rebuild the UI.
  25. 1 min later
  26. 38:50 · watch on youtube.com↗

    I think it's very possible that pretty soon your agents will be working for weeks or months. And it does bring up some kind of interesting questions in terms of how we train and evaluate those models.
  27. 1 min later
  28. 39:21 · watch on youtube.com↗

    I think ideally if you have this digital colleague, you give it literally all the tools that you have if you were onboarding a real human. And so I think organizations are still warming up to that idea, of course.
  29. 1 min later
  30. 40:30 · watch on youtube.com↗

    what do you do when you need to evaluate an agent that runs for a month, but the models ship every month?
  31. 1 min later
  32. 41:18 · watch on youtube.com↗

    uh, I don't think that we have it perfect yet and I think trying to thread the needle between being proactive and suggesting
  33. 41:38 · watch on youtube.com↗

    I also think that it's a pretty safe bet to assume that at this time next year there will be significantly, significantly more tokens flowing in the world
  34. 1 min later
  35. 42:31 · watch on youtube.com↗

    And the reality is, this is just kind of the new normal now where every new product is going to have to have some kind of agentic side to it, either integrating or it will become a you know, a headless piece of data for another agent to use.
  36. 1 min later
  37. 43:38 · watch on youtube.com↗

    Um so I'm still I'm still pro looking at code. You know, I still look at the code that I'm writing. Some people not looking at code. I still think it's helpful, especially to understand the architecture that you're shipping.
  38. 1 min later
  39. 44:48 · watch on youtube.com↗

    But I think it's helpful to understand just like when you're building a web app, it's helpful to know how databases work because increasingly the there are so many decisions you make at the product and the harness level that are influenced by the models themselves.