Code Orange: Fail Small — our resilience plan following recent incidents

Dane Knecht · 19 Dec 2025

Dane Knecht did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

Our reading
Resilience work is best delivered as iterative improvements, not one big-bang change at the end.
Mental modelFeedback loops

Their wordsThese projects will deliver iterative improvements as they proceed, rather than one “big bang” change at their conclusion.
Our reading
Any change to how Cloudflare serves traffic should get the same tested caution as a change to its software.
Mental modelMargin of safety

Their wordsThe past two incidents have demonstrated that we need to treat any change that is applied to how we serve traffic in our network with the same level of tested caution that we apply to changes to the software itself.
Our reading
Near-instant global configuration deploys are useful but rarely needed.

Their wordsWhile the ability to deploy changes to our network on a near-instant basis is useful in many cases, it is rarely necessary.
Our reading
Every interface between critical services should assume failure will occur and handle it as reasonably as possible.
Mental modelMargin of safety

Their wordswe are in the process of reviewing the interface contracts between every critical product and service that comprise our network to ensure that we a) assume failure will occur between each interface and b) handle that failure in the absolute most reasonable way possible.
Our reading
A service that reads a corrupted configuration should fall back on validated defaults rather than panic.
Mental modelMargin of safety

Their wordsInstead of panicking, there should have been a sane set of validated defaults which would have allowed traffic to pass through our network
Our reading
Strict access controls are right for a security company, but they slowed Cloudflare down when speed mattered most.

Their wordsAs a security company, all our tools are behind authentication layers with fine-grained access controls to ensure customer data is safe and to prevent unauthorized access. This is the right thing to do, but at the same time, our current processes and systems slowed us down when speed was a top priority.

Keep reading

The other pieces are the ones whose claims are worded most like this one's. That measures language, not agreement: they may argue the opposite.