Timeline for Scaling Heavy Writes When Fetching Data From External API
Current License: CC BY-SA 4.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 11, 2024 at 22:31 | comment | added | J_H | The OP offered few details. I assumed “crash” meant “too much malloc”. I assumed your app doesn’t leak. I assumed at least one of the NxM API queries for past 24 hours returned “big” results. I assumed requesting 24 one-hour slices would turn that into “small” results which fit in the provisioned container. // As soon as possible, persist the query results and free up the RAM. | |
| Apr 11, 2024 at 19:59 | comment | added | Sachin Saini | the third party exposes the data after a specific time but all at once, I don't quite understand your point dividing the query into small parts by issuing the query at different times, even if I issue the query at different times the third party apis make the full data available, are you suggesting reading the data chunk by chunk by making same api call repeatedly? | |
| Apr 11, 2024 at 19:52 | comment | added | Sachin Saini | Thanks a lot for taking the time to answer the question. You point about not having spec for the containers hits the spot, this is the first thing I am going to do, performance test the containers and check their limits and have a spec for the containers. Clarifying the NXM issue, each merchant has M payment providers mapped to them, so we need to fetch the data from all M payment providers for a merchant, this makes it a NxM problem. | |
| Apr 10, 2024 at 21:21 | history | edited | J_H | CC BY-SA 4.0 |
added 68 characters in body
|
| Apr 10, 2024 at 19:42 | history | answered | J_H | CC BY-SA 4.0 |