korrents

korrents · video

Master Flaky Systems With Retries in Python 🐍 (avoid self-DDoS)

Hynek Schlawack · 16m · youtube.com ↗

20 korrents from this recording

Hynek Schlawack did not write this page.

Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.

  1. 0:21 · watch on youtube.com↗

    So you see, the difference between a robust system that lets you sleep at night and a system that everybody mocks on social media is how you deal with failure. And fortunately, most of failures are temporary.
  2. 1 min later
  3. 0:54 · watch on youtube.com↗

    However, the lesser-known proverb is, if you indiscriminately try, try, try again, you can take down whole systems, your systems.
  4. 1 min later
  5. 1:55 · watch on youtube.com↗

    At certain scale and under real-world conditions that we all live under, regardless of your level of denial, every system is flaky. And the more components it takes to do its job, aka how complex it is, the more things and the more connections between things can fail, which makes it flakier.
  6. 1 min later
  7. 2:37 · watch on youtube.com↗

    So if you don't design your systems for reliability, you won't get a robust system. Period. And because failure is inevitable, a system that is not built for failure will make you do a lot of manual work to keep things working instead of building cool new stuff.
  8. 1 min later
  9. 3:17 · watch on youtube.com↗

    So yeah, big picture. The danger is that you might run a distributed denial of service attack against yourself.
  10. 1 min later
  11. 4:04 · watch on youtube.com↗

    And of course, you don't know which of those reasons is at play. So you need to cater to all of them by starting with a short backoff and then exponentially increase every failed retry.
  12. 4:17 · watch on youtube.com↗

    But it also means that you have to stop trying eventually because not everything recovers within a time frame that makes sense for your operation.
  13. 4:39 · watch on youtube.com↗

    Forgetting to use timeouts on everything remote is the kissing cousin of unbounded retries. It can literally take down whole companies at once, blocking calls, start cascading until everything ends up in a standstill.
  14. 4:59 · watch on youtube.com↗

    But if all clients have the same backoff, they still retry at the same time, so they still take the target down just a little bit later. Which brings us to the final piece of a well-behaved retry. We add a random amount of time to each backoff to spread out the retries.
  15. 1 min later
  16. 6:14 · watch on youtube.com↗

    One simple one is where you retry only at the edges. So only the first client is allowed to retry. But in sufficiently complex systems, it's unclear where the edge is. No server has complete knowledge of the topology around itself. And it shouldn't.
  17. 1 min later
  18. 6:53 · watch on youtube.com↗

    However, by default, this code will do everything wrong I've just explained to you. It's unbounded. So it will retry until the heat end of the universe or your system, whichever comes first, and has neither a backoff nor a jitter.
  19. 1 min later
  20. 7:26 · watch on youtube.com↗

    But I've always said Tenacity is wonderful. And it truly is. Because it has all the knobs you need to get really great retries.
  21. 2 min later
  22. 9:09 · watch on youtube.com↗

    If it's just implied by the defaults of a function, one has to know that it does the wrong thing by default, adding mental burden when trying to reason about the behavior of code.
  23. 1 min later
  24. 9:40 · watch on youtube.com↗

    And while you should retry on a 500 error, retrying on errors in a 400 range, especially 401 and 403, is probably gonna get you banned rather sooner than later.
  25. 10:09 · watch on youtube.com↗

    And these parameters should be fine for most applications. But, of course, you can tune all numbers to your liking as long as it's exponential and has an upper bound.
  26. 2 min later
  27. 12:14 · watch on youtube.com↗

    I care a lot about automated testing because I don't trust myself to write correct code and if I did by accident, I don't trust myself to not break it in the future.
  28. 1 min later
  29. 13:11 · watch on youtube.com↗

    Firstly, of course, you want to know that something went wrong, even if it worked out eventually, because otherwise, regular retries look like weird performance regressions.
  30. 13:23 · watch on youtube.com↗

    stamina comes with built-in support for the best Python logging package, structlog, with the best logo, thanks to Lynn Root, and falls back to the worst logging package, which is the standard library.
  31. 1 min later
  32. 14:04 · watch on youtube.com↗

    If one or more systems suddenly generate a lot of retries, that's a sign that something is off and you should investigate.
  33. 2 min later
  34. 16:07 · watch on youtube.com↗

    unless I listen to the siren's call of SEO and serve zeitgeist topics everyone is searching for, which is something I emphatically do not want to do.