korrents

korrents · The Pragmatic Engineer Podcast

Designing Data-intensive Applications with Martin Kleppmann

Martin Kleppmann · 1h 25m · youtube.com

26 korrents from this recording

1h
Martin Kleppmann did not write this page.

Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.

  1. 0:23:17 · watch on youtube.com

    for me, reliability means fault tolerance primarily. So, meaning that a system should, on the whole, continue working even if like a network link is interrupted or a node crashes or something like that.
  2. 1 min later
  3. 0:24:01 · watch on youtube.com

    Scalability is just like what mechanisms we have for dealing with changes in load. If load increases, how can we add computing capacity to a system, for example, so that the system still continues working?
  4. 2 min later
  5. 0:25:37 · watch on youtube.com

    generally, like you just want the the cost and the computing capacity to be roughly proportional to the load that you have. And at the low end, that means actually being able to scale down to something that is extremely cheap to run.
  6. 4 min later
  7. 0:29:38 · watch on youtube.com

    And now suddenly people are building databases on top of object stores, for example. And now the replication happens at the object store level, no longer at the database level.
  8. 2 min later
  9. 0:31:55 · watch on youtube.com

    You're building higher-level business logic, actually I think it's just fine for people not to care about memory management.
  10. 0:32:17 · watch on youtube.com

    But, somebody still has to build those lower-level abstractions and from lower-level components. Somebody's got to implement the cloud services.
  11. 3 min later
  12. 0:35:39 · watch on youtube.com

    And so those are those places where I feel like knowing a bit about the the internals is actually like a superpower.
  13. 2 min later
  14. 0:37:25 · watch on youtube.com

    So, what if geopolitics was to go horribly wrong and tensions escalate and Europe find itself suddenly locked out of US cloud services? I hope that doesn't happen. I still think it's fairly unlikely, but it's no longer unthinkable.
  15. 1 min later
  16. 0:38:47 · watch on youtube.com

    Maybe as AI writes more and more code of our code, it's less about like the details of how you express logic in a particular programming language and much more about those kinds of high-level trade-offs.
  17. 4 min later
  18. 0:42:22 · watch on youtube.com

    But, at least this sort of sharding across multiple machines is maybe become less of a pressing issue just because more and more workloads can just run on a single machine. Some people still have very large-scale workloads that do have to be sharded across multiple machines. So, it's not going away entirely.
  19. 0:42:37 · watch on youtube.com

    And uh replication is still relevant even at smaller scales because that's for fault tolerance, that's not for scalability.
  20. 1 min later
  21. 0:43:07 · watch on youtube.com

    we just assume that there's no upper bound on how long it might take for a message to go over the network. So, you send a message, it might arrive within 100 microseconds, or it might take 10 years.
  22. 1 min later
  23. 0:44:34 · watch on youtube.com

    it's very easy to assume that your clocks are correct and most of the times the clocks are pretty correct but we just can't rely on it because actually they're just not precise enough uh on the whole.
  24. 2 min later
  25. 0:46:40 · watch on youtube.com

    And so, the goal of this chapter is really just to give people the information in order to make an educated decision. But I don't want to make that decision for people. That's for businesses themselves to decide.
  26. 1 min later
  27. 0:47:11 · watch on youtube.com

    In particular, for example, coverage of MapReduce was quite detailed in the first edition. But basically, MapReduce is dead. Nobody uses it anymore.
  28. 3 min later
  29. 0:50:32 · watch on youtube.com

    people say that uh you get into tech in order to change the world. If you want to change the world, then thinking about the impact that your technologies have on the world is part of your job.
  30. 4 min later
  31. 0:54:10 · watch on youtube.com

    But a proof can reason about potentially infinite state spaces. So, it can tell you things about like every possible thing that could possibly happen in the entire universe.
  32. 5 min later
  33. 0:58:54 · watch on youtube.com

    One is that the LLMs are getting increasingly good at writing these proofs. And if we don't have to write the proof by hand as humans, it just becomes feasible to do them in situations where previously it would have not been economical.
  34. 0:59:06 · watch on youtube.com

    But also LLMs increase the need for these formal proofs because, you know, we're live coding a bunch of stuff. If we have to manually review all of that code, then that will become the bottleneck.
  35. 3 min later
  36. 1:02:07 · watch on youtube.com

    local-first software, which is this idea that we want to take away a bit of the power from cloud operators and give it back to end users. So, end users should be more in control of their own data and less dependent on cloud services for providing the applications and the data that that the users need.
  37. 1:02:34 · watch on youtube.com

    software as a service businesses, for example the whole reason why they can charge a subscription is because they are able to essentially hold a gun to the customer's head and say, "Pay us at your subscription, otherwise we will delete all your data."
  38. 1 min later
  39. 1:03:11 · watch on youtube.com

    there I feel like in academia, I have the freedom to work on things that go against this commercial incentive of companies and say like, "Actually, no, I'm going to do what I think is right for the users."
  40. 5 min later
  41. 1:07:44 · watch on youtube.com

    if a revoked user still wants to, say, vandalize a document, they can just backdate their edits, give it an earlier timestamp. So, relying on clocks is absolutely useless here because people can forge the timestamps from those clocks and thereby then potentially undermine the access control mechanism.
  42. 8 min later
  43. 1:15:49 · watch on youtube.com

    We don't ask the students to write essays because we love reading their amazing essays. We ask them to write essays because they we want them to go through a thought process, which helps them learn something.
  44. 6 min later
  45. 1:21:51 · watch on youtube.com

    It's more that for myself, the process of writing is the way how I figure things out. And figuring things out is really my goal here. So, I'm I'm trying to figure it out in my own heads, and for that I just have to write it myself.
  46. 2 min later
  47. 1:23:42 · watch on youtube.com

    Often people in industry I feel like sort of have short circuit reasoning. Like don't maybe don't quite reason something through from first principles but just like oh I heard this from a conference talk I'm just going to go with that