Timeline for Retry Architecture
Current License: CC BY-SA 3.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 16, 2018 at 14:44 | comment | added | Laiv | To my experience, implementing a queue in tables force us to implement too much boilerplate code that solves things already solved in off-the-shelve MQ products such as RabbitMQ, ActiveMQ or even Redis | |
| Jan 16, 2018 at 14:38 | comment | added | Laiv | Tasks are never lost. They are in memory, being processed and waiting to be stored somewhere (table, file, queue) at its end. The thing is, withdrawing the task from the queue so that no other worker can pull the same task. The lifecycle would be (more or less): 1. withdraw the task (remove from the table and loaded in memory) 2. Pre-conditions 3, Execution, 4.Post-condition, 5. stored has processed/failed. If we were speaking about queues, the task would be enqueued again in the very same queue it came from, or into another different one dedicated. exclusively to retries. | |
| Jan 16, 2018 at 14:32 | comment | added | Husain | @Laiv But here is the problem. With this simple implementation of queues, if a take is taken by an instance and then the instance failed. The task is lost. Verse in a table implementation, the task is merely updated. Is there a way to keep the task if the instance fails (due to shutdown or something like that)? | |
| Jan 13, 2018 at 22:13 | history | tweeted | twitter.com/StackSoftEng/status/952302571900764160 | ||
| Jan 13, 2018 at 19:18 | comment | added | Laiv |
If I try message X from the queue and it fails, where do I put it given that there are more messages that are coming in. Into different queue; from where a scheduled jobs will withdraw the messages. Looks like you are thinking in a single table, queue and process. You can have many of them as you need it. Having a single table/queue/job is like having a whole dining room eating from the same plate.
|
|
| Jan 13, 2018 at 19:02 | answer | added | Max | timeline score: 1 | |
| Jan 13, 2018 at 18:37 | comment | added | Robert Harvey | If you use a separate list for each instance/request, you won't have threading problems, since each thread will only touch its own list. | |
| Jan 13, 2018 at 18:24 | comment | added | Husain | @Jules Well, messaging queue will be good for queuing the messages. But once the retry part is introduced, I don't know how to solve it with queues. If I try message X from the queue and it fails, where do I put it given that there are more messages that are coming in, while I want to try X again after D delay. | |
| Jan 13, 2018 at 18:23 | comment | added | Max | What would be more efficient than doing the retry in memory? You could write the data to disk and then read it back later. Are you trying to do it that way? | |
| Jan 13, 2018 at 18:21 | comment | added | Husain | @RobertHarvey Could you please explain what you mean? If I use a fresh copy, would not that mean that I will process the same request twice? | |
| Jan 12, 2018 at 23:09 | comment | added | Jules | You seemto be trying to solve the problem that message-queue middleware solves, e.g. ActiveMQ. Have you looked into systems like that? | |
| Jan 12, 2018 at 23:02 | comment | added | Robert Harvey | Use a fresh copy of the table for each instance. | |
| Jan 12, 2018 at 22:31 | history | asked | Husain | CC BY-SA 3.0 |