korrents

prompt injection

Getting an agent to follow instructions it found instead of the ones it was given.

FilterEveryone, all time10 from 5 people
Group All Technologists 2Scientists 1Founders & executives 1
Who Everyone SWSimon Willison 4 BCBoris Cherny 2 PSPeter Steinberger 2 THThePrimeagen 1 ZMZvi Mowshowitz 1
When All time Past 1 yearPast 3 yearsPast 5 years
  1. ZM

    Zvi Mowshowitz quoted

    Writes Don't Worry About the Vase

    Our readingFor agents built on Claude Fable 5.1, it is in practice safe for now to assume they will not get prompt injected.

    Anthropic

    Their wordsIt is in practice safe to assume your agents are not going to get prompt injected, even if you are kind of asking for it.

    ↗Claude Fable 5.1 and Mythos 5.1: The System Cardthezvi.substack.com

  2. 4 weeks earlier
  3. SW

    Simon Willison quoted

    Co-creator of Django and creator of Datasette

    Our readingAuto-mode does not yet convincingly fix prompt-injection risk for coding agents.

    coding agents

    Their wordsI REALLY want to believe that this fixes prompt injection risks for coding agents, but I'm just not there yet

    ↗@simonw on Xx.com

  4. 12 days earlier
  5. BC

    Boris Cherny quoted

    Creator of Claude Code at Anthropic

    Our readingPrompt injection is no longer a live attack on the frontier model: it simply does not follow instructions it reads on the internet any more.

    Their wordsthe model does not seem to be prompt injectable anymore.

    ↗Boris Cherny: We Cut 80% of Claude Code's Promptyoutube.com 1st of 20 in this recording

  7. BC

    Boris Cherny quoted

    Our readingPrompt injection can be caught by watching the neurons that fire when it happens, so a defence no longer depends on the model reporting the attack.

    Their wordsit's it's literally we're looking at neurons in the model's brain that light up when prompt injection happens. So the model won't even tell you but we can actually see those neurons and we can figure out and diagnose that it's happening. And then you combine that with the auto mode classifier and with these three layers we just cannot demonstrate prompt injection anymore.

    ↗Boris Cherny: We Cut 80% of Claude Code's Promptyoutube.com 2nd of 20 in this recording

  8. 4 months earlier
  9. SW

    Simon Willison quoted

    Our readingPrompt injection has no equivalent of parameterized queries: there is no reliable way to tell a language model which text is data and which is instructions.

    LLMs

    Their wordsProblem is, you can solve SQL injection by parameterizing your queries. You can't do that with LLMs.

    ↗Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summityoutube.com 10th of 18 in this recording

  11. SW

    Simon Willison quoted

    Our readingThe only guaranteed defence against a prompt-injection attack is to remove one of its three legs — private data, exposure to untrusted content, or a way to send data out.

    Their wordsI call it the lethal trifecta because the only guaranteed solution is to cut off one of the legs.

    ↗Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summityoutube.com 11th of 18 in this recording

  13. SW

    Simon Willison quoted

    Our readingSandboxing is the single most important protection for a coding agent, because it is the only one that bounds the damage when the agent is successfully tricked.

    Their wordsSo, I think the most important thing is sandboxing. You want your coding agent running in an environment where if something goes completely wrong, if somebody gets malicious instructions to it, that the damage is greatly limited.

    ↗Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summityoutube.com 12th of 18 in this recording

  14. 5 weeks earlier
  15. PS

    Peter Steinberger quoted

    Founded PSPDFKit in 2011 and ran it for a decade

    Our readingPrompt injection is unsolved, but the latest models are post-trained to resist it, so it takes far more than 'ignore all previous instructions' now.

    scaling laws

    Their wordsSo, so the latest generation of models has a lot of post-training to detect those approaches, and it's not as simple as ignore all previous instructions and do this and this. That was years ago. You have to work much harder to do that now. Still possible.

    ↗OpenClaw: The Viral AI Agent that Broke the Internet - Peter Steinberger | Lex Fridman Podcast #491youtube.com 5th of 30 in this recording

  17. PS

    Peter Steinberger quoted

    Our readingDo not run a personal agent on a cheap or local model: weak models are gullible and easy to prompt-inject.

    documentation

    Their wordsThat's why I warn in my security documentation, don't use cheap models. Don't use Haiku or a local model. Even though I, I very much love the idea that this thing could completely run local. If you use a, a very weak local model, they are very gullible. It's very easy to, to prompt inject them.

    ↗OpenClaw: The Viral AI Agent that Broke the Internet - Peter Steinberger | Lex Fridman Podcast #491youtube.com 6th of 30 in this recording

  18. 11 months earlier
  19. TH

    ThePrimeagen quoted

    Michael Paulson, known online as ThePrimeagen

    Our readingPrompt injection is an extremely hard attack to defend against, and hardly anyone integrating AI is talking about it.

    Their wordsOne thing that I don't think a lot of people are talking about as we integrate more and more AI is that prompt injection is an extremely hard thing to defend against because it's not really clear how you defend against it.

    ↗ThePrimeagen: Programming, AI, ADHD, Productivity, Addiction, and God | Lex Fridman Podcast #461youtube.com 9th of 31 in this recording