1

For a complex service that my team and several others maintain, two of my team's applications are called when processing one incoming request. The flow is as follows: an upstream system invokes an HTTP endpoint of our application A, then several other applications managed by other teams are called, one of them publishing an event Topic1 to a message broker that our application B then consumes. B needs to do some extra processing on this event and publish a different event Topic2 to the message broker. The part needed to serve the incoming request (synchronous) ends somewhere between A and B, and the rest is (simplifying) needed for our analytics.

The problem: in order to process the events it consumes, B needs to know some information that is known by A, and that is transient (specific to each request). In the past, we put this information in the response A produces and asked systems in between to forward it to B. But there are several systems and teams involved, which have their priorities and concerns about passing this information, taking several weeks and plenty of discussions.

In summary, we need another way of passing information from A to B, controlled only by our team. What is the best way of doing this? Something we cannot do is having B directly call A (e.g. having A locally store the info and then B fetching it). We are considering these two approaches, but can consider other solutions:

  • A publishes an event Topic3 with the needed info, B merges Topic1 and Topic3 by requestId and extracs the info from Topic3
  • A writes to a cache that B then queries by requestId
Improve this question
4
  • Are you sure you don't just need to know that all of the other sub tasks have completed? Maybe each sub task should publish an event when it is done, and the main task can listen for these events and collect all the necessary information. Sounds like this will need to turn into a very asynchronous process. Commented May 14, 2019 at 12:44
  • @GregBurghardt We don't need only to know that other tasks have completed, we need some pieces of information. Also, I'm not sure that talking about 'main task' would be correct, I'd rather talk about most-upstream service and other downstream services. Commented May 14, 2019 at 12:55
  • 1
    "which have their priorities and concerns about passing this information" - could you edit the question to expand on that a bit? Seems like this is one of the key sources of the problem, so help us understand what is actually going on. What is the nature of these concerns? In what ways this affects them? Do they read and/or rearange this information, or simply pass it along without examining it? Maybe you just need better interfaces between the systems/services. Commented May 14, 2019 at 13:57
  • Is it right that some teams don't want to provide that additional info when sending Topic1 events? Perhaps introducing an augmented Topic1a with events allowing to transfer the same data as Topic1 events PLUS your information specific to A is the solution? B needs to subscribe to it and perform its extended processing only on that topic's events. Commented May 14, 2019 at 18:56

2 Answers 2

Reset to default
2

A design allowing A to throw messages on the message bus B reads strengthens the message passing abstraction. A design requiring B reading from A's cache might be more likely to produce a close coupling of A's and B's internals.

In a message passing implementation, if Topic 3 arrives first B leaves it in its mailbox. When Topic 1 arrives, B checks its mailbox. If Topic 1 arrives first, B leaves it in its mailbox. When Topic 3 arrives, B checks its mailbox. In the generalized case, message receipt can trigger re-evaluating every message in the mailbox. This allows purging messages that are no longer relevant.

An implementation where B reads from A's cache, A will be responsible for flushing the cache. Coordinating this with the producer of Topic 1 would require further close coupling, not just with B.

Improve this answer
0

It seems to me that what you really want is URI that provides this information on demand. The underlying implementation could be a cache, a topic (I guess), or a key-value store. Anything that allows you to quickly retrieve the information from by it the requestId will suffice. Ideally, that detail is something you want to keep out of the interaction with B.

Which storage approach you use depends on a lot of different factors. By abstracting that detail away from B, you can easily change the storage without any impact on B.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.