cybersecurity
Attack and defence in software: vulnerabilities, hardening, and what changes when a model is good at finding them.
FilterEveryone, all time99 from 47 people
- GR
Guillermo Rauch quoted
CEO of Vercel; creator of Next.js and Socket.IO
Their wordsSecurity will become a larger and larger function in software companies.
- 10 days earlier
- ZM
Zvi Mowshowitz quoted
Writes Don't Worry About the Vase
Our readingClaude Opus 5.5 is a Tier 2 cyber model, despite Anthropic placing it in the lower category.
Their wordsAnthropic claims that Opus 5.5 also is not Tier 2 here, but seems ashamed to say it outright, saying in 3.1 it is in the ‘lower category,’ but without a defense of that beyond ‘we see no indication it can develop novel offensive capabilities.’ I don’t think that is defensible given the evidence presented below. Thus, I disagree even more strongly for Opus 5.5. This is a Tier 2 cyber model.
- GR
Guillermo Rauch quoted
Their wordsBtw, breaking apart the agent into these independent parts not only optimizes costs in a big way, it also *massively* improves security and auditability. I'd argue you can't even run a secure agent otherwise!
- 4 days earlier
- ZM
Zvi Mowshowitz quoted
Their wordsRarer is better, but I still think the right rate of misbehaviors on the level of social engineering to put malicious packages into a public registry is still zero.
↗Anthropic Looks At Some Of Its Alignment Problemsthezvi.substack.com
- 2 days earlier
- RS
Richard Stallman quoted
Founder of the Free Software Movement and the GNU Project
Their wordsHistorically, whenever companies get power over what people's computing facilities can do, they use it against us, building malicious functionalities, designed to benefit them, into the software they invite us to use.
↗Urgent: Raise the Wage Actstallman.org 7th of 10 in this piece
- ZM
Zvi Mowshowitz quoted
Their wordseven when given a benign task to retrieve public information, your AI agents could still spontaneously decide to do so via hacking third party websites with malicious software packages.
- SW
Simon Willison quoted
Co-creator of Django and creator of Datasette
Their wordsAny piece of software that depends on open source (which is almost every piece of software) has a network of human beings who are potential attack vectors - everyone with publishing rights to any of the packages in the dependency network for that software.
↗Be alert: targeted attacks on prominent Rustaceanssimonwillison.net
- SW
Simon Willison quoted
Their wordsI guess our best defense right now is dependency cooldowns - giving new package releases a few days before upgrading to them, in the hope that supply chain attacks like this will be spotted by someone else.
↗Be alert: targeted attacks on prominent Rustaceanssimonwillison.net
- 1 day earlier
- TC
Tyler Cowen quoted
Economist at George Mason University
Their wordsWe need much more effort in this direction, and I hope this subfield rises in status rapidly.
- 1 day earlier
- TP
Thomas Ptacek quoted
Security engineer; founder of Matasano Security and Latacora
Their wordsIncreased reliance on CFAA prosecutions for public policy goals couldn't possibly be a cursed monkey's paw situation, said no practitioner in this space ever.
- 3 days earlier
- AP
Aaron Patterson quoted
Ruby and Rails core committer known online as "tenderlove"
Their wordsIn other words, it looks like OpenAI's bots knew about this problem and attempted to exploit it.
- SW
Simon Willison quoted
Their wordsIf that’s true there are two options: After the Hugging Face and Wiki attacks OpenAI were still unable to review their previous logs and determine that they had previously attacked RubyGems. They knew about the attack on RubyGems and made the decision not to reach out to the RubyGems team about it. Both of these are bad!
↗OpenAI agents attacked RubyGems back in Maysimonwillison.net
- GR
Guillermo Rauch quoted
Their wordsThe concerns over AI safety and cybersecurity are legitimate, but we’re risking talking America, the global AI leader, into self-inflicted obsolescence and the obscurity of bureaucracy.
- GR
Guillermo Rauch quoted
Their wordsOur adversaries have the training techniques, the data, and the will to attack. And they won’t be slowed down with “embedded evaluators.” They’ll likely have embedded accelerators!
- 2 days earlier
- NL
Nathan Lambert quoted
Machine-learning researcher on open language models
Their wordsI put the probability of complete extinction as being so low it isn't worth discussing, but the probabilities of AI caused disasters - e.g. cyber attacks on critical infrastructure or bio-risks - as being worth debating.
↗One resignation turned the embers of AI fear into a wildfireinterconnects.ai 1st of 3 in this piece
- ZM
Zvi Mowshowitz quoted
Their wordsIf there is a 3x productivity boost now, the correct assumption is that in the future the productivity boost, in places it carries over, will a lot more than 3x. Indeed, given how this scales, a better model might be in practice 10x or 100x or even 1,000x or more.
- 1 day earlier
- SP
Sunil Pai quoted
Systems and developer-tools engineer at Cloudflare as of 2026
Their wordswe really need 1 million experiments in this space of "security against autonomous attacks and entities"
- 3 days earlier
- TC
Tyler Cowen quoted
Their wordsI am happy to admit that the answers here are far from obvious, and that I am myself expecting prices to rise somewhat.
- JP
Jakub Pachocki quoted
Chief Scientist at OpenAI
Their wordsA clear risk discussed throughout this year is to cybersecurity: the models are becoming superhuman in their ability to break in and out of computer systems.
- 1 day earlier
- NS
Noah Smith quoted
Economics writer; author of the Noahpinion newsletter
Our readingAI-enabled cyberwarfare lacks an equivalent of nuclear deterrence's mutually assured destruction.
Their wordsAI hacking doesn’t have mutually assured destruction, like nuclear warfare does.
↗America is still beating China in the AI racenoahpinion.blog
- 1 day earlier
- BS
Bruce Schneier quoted
Security technologist and author
Their wordsThis kind of thing will be exploited. Think Solar Winds–style supply chain attacks.
↗AI Coding Agents Are Installing Unknown/Untrusted Code on Corporate Networksschneier.com
- SW
Simon Willison quoted
Their wordsIt looks to me like OpenAI’s sandbox for this agent suffered from the (quite naïve) assumption that GET requests cannot be used to update data. That’s certainly how the web is supposed to work, but clearly there are applications that don’t hold to that contract.
↗OpenAI's rogue agents were caught communicating via public wikissimonwillison.net
- 3 days earlier
- AC
Ajeya Cotra quoted
AI researcher and grantmaker
Their wordsI do think I want to push back on the cyber on the brain hypothesis that you raised a couple of times. We didn't find like particular evidence for the cyber nature of the task making all the hacking and crimes that they did more likely versus the impossible nature of the task.
↗Ajeya Cotra – "This might be the clearest warning shot we ever get"youtube.com 9th of 26 in this recording
- AC
Ajeya Cotra quoted
Their wordseven in this incident we saw there was a lot of pressure um as a result of this incident to stop doing cyber security evaluations and I really don't think that stopping doing evaluations and like sort of blinding ourselves to the result of evaluations is the right reaction to this problem.
↗Ajeya Cotra – "This might be the clearest warning shot we ever get"youtube.com 23rd of 26 in this recording
- TC
Tyler Cowen quoted
Their wordsWhen it comes to the cybersecurity risks from advanced AI, very likely they will not come close to being as bad as either Covid or global warming.
- ZM
Zvi Mowshowitz quoted
Their wordsIt is highly fortunate that the OpenAI agents hacked HuggingFace. This is the only reason we know about all the severe internal failures at OpenAI, and gives us an opportunity to wake up before it is too late.
↗HuggingFace Attack Postmortem: Civilizations, Reactions and Next Actionsthezvi.substack.com
- 1 day earlier
- RK
Rohit Krishnan quoted
Writes Strange Loop Canon
Our readingAI models are gullible enough that other AI models can induce them into harmful behavior.
Their wordsThe models are pretty gullible and can be induced to do bad things, including and especially by other models
↗How to control an agent swarmstrangeloopcanon.com 1st of 3 in this piece
- ZM
Zvi Mowshowitz quoted
Their wordsA less bad version of it is known to have happened, and from the outside it seems likely that worse things have happened internally that we never heard about.
↗HuggingFace Attack Postmortem: Fleshing Out the Factsthezvi.substack.com
- 1 day earlier
- PC
Patrick Collison quoted
Co-founder and CEO of Stripe and co-founder of the Arc Institute
Their wordsOverall, I’m very surprised at how little media coverage there’s been around the OpenAI / Hugging Face attack. It’s clearly one of the most important things to happen this year.
- 3 days earlier
- XI
Xe Iaso quoted
Technical educator, conference speaker and developer…
Our readingC is the only programming language where this kind of memory-safety vulnerability regularly happens.
Their wordsThis is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen.
↗"No way to prevent this" say users of only language where this regularly happensxeiaso.net