Timeline for Database consistency for a distributed system
Current License: CC BY-SA 4.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 10, 2021 at 21:00 | history | tweeted | twitter.com/StackSoftEng/status/1359608118569754626 | ||
| Jan 26, 2021 at 20:09 | comment | added | widavies | @user253751 I added some more details on how I'm thinking of using Kafka, any thoughts on this? | |
| Jan 26, 2021 at 20:08 | history | edited | widavies | CC BY-SA 4.0 |
added 729 characters in body
|
| Jan 26, 2021 at 19:22 | comment | added | Stack Exchange Broke The Law | @wdavies973 Each thread would process a partition of the calls, not just one call. All the synchronization would happen in your code, not in the database. As I said I'm not sure about the routing. | |
| Jan 26, 2021 at 19:19 | comment | added | widavies | @user253751 This would be possible, but wouldn't that mean I need a thread for every call? Would that be performant? Yes, each call itself has very few events to process. Kafka seems interesting. I'm not sure how an event consumer would look though. A challenge is that some requests need to pull a call and inspect some data of it. I'm not sure how this would work in an event driven system aside from pre-emptively sending the call, but its hard to know which server will handle the request (my frontend servers are distributed as well). | |
| Jan 26, 2021 at 18:58 | comment | added | Stack Exchange Broke The Law | You could also consider routing the events about a call to the same thread on the server in the first place, if your architecture allows it, so no locking is needed in the database. As I understand it, each call only has a moderate rate of events which is low enough for one thread to process many calls? But I don't see an obvious way to do the routing for this idea. (Edit: Kafka may be able to do it?) | |
| Jan 26, 2021 at 18:56 | comment | added | widavies | @user253751 This is what I'm currently doing, phone numbers are partitioned. Secondly, one of the reasons I chose a database over a cache is because I do want the data to be logged for improving the system and understanding its performance (a little data lost doesn't matter much though). Great idea though, will be using some sort of a cleaning process or TTL. | |
| Jan 26, 2021 at 18:54 | comment | added | widavies | @s.lg It's not a hard requirement, just chosen for its horizontal scalability. Horizontal scalability is important in this application. | |
| Jan 26, 2021 at 18:53 | comment | added | widavies | @BartvanIngenSchenau I am trying to model most requests that way as much as possible, but there will be an occasional update that overwrites a value. | |
| Jan 26, 2021 at 16:58 | comment | added | Stack Exchange Broke The Law | +1 for Bart's suggestion. You can partition the changes by phone number for example. You can also have an hourly/daily process that processes all "closed" calls to reduce the amount of data stored. | |
| Jan 26, 2021 at 14:01 | answer | added | Ryathal | timeline score: -1 | |
| Jan 26, 2021 at 10:23 | comment | added | s.lg | Is noSql a hard requirement, or have you concidered using a database with support for transactions (postgresql.org/docs/current/tutorial-transactions.html)? Also, retry loops are indeed sloppy, look for an event queue pattern | |
| Jan 26, 2021 at 7:43 | comment | added | Bart van Ingen Schenau | Is it an option to store the changes themselves in an append-only manner and only reconstruct the full state of a call when needed? | |
| Jan 26, 2021 at 6:21 | history | edited | widavies | CC BY-SA 4.0 |
added 146 characters in body
|
| Jan 26, 2021 at 6:15 | history | asked | widavies | CC BY-SA 4.0 |