Questions tagged [message-queue]
A message queue is a service that receives events or messages from producers and provides them to consumers.
260 questions
0
votes
0
answers
10
views
Design a queue solution for multiple consumers that must operate synchronously
Transaction Workflow:
Validate Funds – Verify the sender’s account has sufficient balance.
Withdraw Funds – Deduct the specified amount from the sender’s account.
Transfer Funds – Credit the ...
- 101
3
votes
3
answers
543
views
Design question for exactly-once processing in a message-driven system using a unique ID
To achieve exactly-once processing where messages are consumed from a queue with at-least-once delivery, many sources (e.g. here and here and here) suggest attaching a unique ID to messages in the ...
- 139
0
votes
0
answers
191
views
How does each application replica read a unique message from the message broker
In RabbitMQ streams or Kafka, messages are not deleted after being consumed. If you have a consumer application replicated across multiple Kubernetes pods, how can you ensure that each pod picks up a ...
- 9
0
votes
5
answers
165
views
How can I measure which messages are causing queue backups?
Consider the following multi-producer, multi-consumer system:
For simplicity, assume that we only have one queue
Messages are dequeued on a first-in-first-out basis
Oldest-message-pickup-latency has ...
- 119
1
vote
1
answer
199
views
Background thread processing vs queue based processing for relatively short tasks (max 30-40 seconds)
I need suggestion / recommendation on the design approaches mentioned below.
UseCase: I have a usecase where a client uses my system to generate some recommendations. Now, these recommendations are ...
- 19
3
votes
3
answers
252
views
Handling rate limits / delays in consumers without affecting performance of other operations
I have a producer that generates a batch job that consists of multiple operations (approx. 100 - 10000). These operations can be processed in any order, ideally as fast as possible.
The processing of ...
- 141
0
votes
1
answer
177
views
How to rebalance data across nodes?
I am implementing a message queue where messages are distributed across nodes in a cluster. The goal is to design a system to be able to auto-scale without needing to keep a global map of each message ...
- 115
7
votes
5
answers
427
views
What can I do to get a message processor to slow down the rate of writes that it is making to a database?
We have this architecture:
queue -> message processor (horizontal scaling) -> RDBMS
Sometimes external systems dump 10k messages onto the queue and the message processor of course dutifully ...
- 229
0
votes
3
answers
2k
views
Backend to client communication: WebSocket or Message Queue?
I have a hobby project, for simplicity imagine a poker game, consisting of a backend (Spring) and a client application (Flutter) where the client can join rooms in which he can take a seat. The client ...
- 9
2
votes
4
answers
305
views
Ensure concurrent processing based on a message key
I've been wondering about the following: Say you have an async message/task and you want that message only processed one at the time. for example: Only process one order for each customer at the time. ...
- 174
0
votes
2
answers
331
views
How to implement event payload isolation in an event driven architecture?
I have an event, let's say order.placed The payload consists of some general information about the order and 3 confidential fields. For example part of the message should only be visible to consumer A,...
- 129
0
votes
1
answer
93
views
Design a sequential processing of records
About 10 to 15 records are processed per day with the time interval of 5 minutes between each record.
System A inserts a record in DB and sends id of that record to active mq.
System B Listener ...
- 119
1
vote
0
answers
133
views
golang: pattern for handling message queues? Are named functions anti-idiomatic somehow?
Had a discussion today in how to implement services that work with messages coming in from event queues. We call these services processors. One of us argues for using several functions, while the ...
2
votes
0
answers
136
views
parent correlation id in message with rabbit queue and open telemetry
I have a rabbit queue that I use to communicate between applications.
I have a chain of three message.
App 1 generate and push a message in rabbit(message 1)
App 2 catch the message and generate ...
- 307
3
votes
2
answers
493
views
Can multiple producers cause events with the same key to be enqueued out of order within a specific partition of a topic?
I'm aiming to create a simple event-driven system where each microservice operates with its own database. The idea is to share database changes across microservices through events. To ensure proper ...
