Timeline for Scaling Heavy Writes When Fetching Data From External API
Current License: CC BY-SA 4.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 10, 2024 at 20:40 | comment | added | candied_orange | I can fill up any amount of memory with the same character. Can you reliably reproduce this crash on demand? First find a way to do that and it'll make diagnosing the problem much easier. | |
| Apr 10, 2024 at 19:42 | answer | added | J_H | timeline score: 4 | |
| Apr 10, 2024 at 18:59 | comment | added | JimmyJames | @amon Maybe I misunderstand your point here but I think the question states that the data has to be retrieved in bulk. The biggest challenge I see here is making sure every transaction is captured exactly once. If transactions can be modified or appended to, that might present additional complexity. | |
| Apr 10, 2024 at 18:48 | comment | added | amon | It is good that you're thinking about consistency, but that doesn't require that you write everything in one go. If using an ACID DB, start a transaction, then write each bit of data as you receive it. Finally, commit the tx. Otherwise, perhaps you can write to a staging area that is not yet used by live queries that need consistency. Append-only designs might be helpful. The banking system itself runs on fairly relaxed consistency, with transactions taking anywhere from seconds to days to settle. Perhaps you too can embrace this in your system. | |
| Apr 10, 2024 at 18:36 | comment | added | Ewan | it seems weird that your container would crash, what's happening? | |
| Apr 10, 2024 at 18:13 | comment | added | CommunityBot | Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. | |
| Apr 10, 2024 at 18:03 | comment | added | Ewan | i mean in Gigabytes. there 86400 seconds in a day so 20k transactions isnt alot | |
| Apr 10, 2024 at 17:36 | comment | added | Sachin Saini | A merchant can have around 20k transactions a day, and we get all the transaction info for those 20k transactions and then we repeeat it for M providers and N merchants. | |
| Apr 10, 2024 at 17:23 | comment | added | Ewan | exactly how much data are we talking? | |
| Apr 10, 2024 at 17:19 | review | Close votes | |||
| Apr 25, 2024 at 3:03 | |||||
| S Apr 10, 2024 at 17:04 | review | First questions | |||
| Apr 10, 2024 at 18:13 | |||||
| S Apr 10, 2024 at 17:04 | history | asked | Sachin Saini | CC BY-SA 4.0 |