The Path Towards Simplifying Consistency in Distributed Systems

Transcript

1. The Path Towards Simplifying Consistency in Distributed Systems

2. Caitie McCaffrey Distributed Systems Engineer caitiem.com @caitie

3. Distributed Systems Engineer Feral Concurrency Control Programmer Caitie McCaffrey

4. 2015

5. 2015 “Application-level Mechanisms for maintaining database integrity”

6. Halo 4 Statistics Service Halo 4 Statistics Service Record &

   Aggregate Statistics From Every Multiplayer Halo Game a User Plays

7. Halo 4 Statistics Service

8. Observability at Twitter Make Dashboards & Alerts Always Available

9. Cmd Line Tool Viz / Dashboad Alerting Svc Cuckoo-Read Cuckoo-Write

   Indexing Svc Relay Svc Twitter Front End Twitter Svc Twitter Statsite Twitter Svc Twitter Svc Scribe Collection Agent HDFS Manhattan Database Public Cloud Observability at Twitter

10. Notify Users When We Receive Abuse Reports

11. Notify Users If An Action is Taken

12. Report Events Stream Modification Events Stream Enrollment Events Stream Listener

    Service Daemons Service Executor Service Notification Events Stream Notification Service Manhattan Macaw

13. Feral Concurrency Control

14. Service Service Service Monoliths

15. The Rise of NoSQL

16. None

17. Microservices

18. State of Affairs

19. Linearizable Sequential Causal Pipelined Random Access Memory Read Your Write

    Monotonic Read Monotonic Write Write From Read Consistency Models
20. None
21. None
22. None
23. None
24. None
25. None

26. Can We Do Better?

27. Finance & Accounting

28. VisiCalc 1979

29. 1983 Lotus 123

30. None

31. 1987 Excel

32. None

33. & Standardization Computer Aided Computation

34. Standardization for Consistency in Distributed Systems

35. 1988

36. 1988 “We believe the form of communication that is needed

    is remote procedure call with at- most-once semantics”

37. 1988 “An adequate language must provide a modular, reasonably automatic

    method for achieving consistency ”

38. Branch B Account 789: $250 Branch A Account 123 :

    $100 Account 456: $50 SubAction: Deposit (123, 50) Transfer Action SubAction: Withdraw(789, 50) Transfer Deposit Withdraw enter topaction coenter action branchA.Deposit(123, 50) action branchB.Withdraw(789, 50) end end

39. What about Spanner?

40. 2015

41. 2015 “The biggest barrier…is that consistency mechanisms must integrate across

    many stateful services”

42. Microservices

43. CRDTs * Stolen from Chris Meiklejohn in practice

44. CRDTs * Stolen from Chris Meiklejohn in practice

45. There Is No One Size Fits All Consistency Model

46. Mixed Consistency Data Stores Manhattan

47. None

48. Computer Aided Computation for Consistency in Distributed Systems

49. 2016

50. Bank Application Bank Account must be > 0 Deposit Money

    Withdrawal Money

51. ‘Cause I’m Strong Enough: Reasoning About Consistency Choices in Distributed

    Systems

52. Define Invariants & Actions Deposit Money Withdrawal Money Determine Consistency

    Required Bank Account > 0 Eventual Consistency Strong Consistency Code Generation?

53. & Standardization Computer Aided Computation

54. Thank you @caitie