How to combine data from multiple sources into the same object?

Question

I've been running into a common pattern when requesting data from multiple sources:

Have a list of objects from one source (e.g. a list of Cars) with an id property and a few other properties populated (e.g. make, model, year).
Call some service (second source) with the list of Car ids to get data for a few other Car properties (e.g. lastOwner, lastPurchasedDate). The service returns list of CarSupplementalDatas that have the Car's id, lastOwner, lastPurchaseDate.

Add data from the service (second source) to the object from the first (e.g. the CarSupplementalData to the Car data by:)

for (Car car : cars) {

    for (CarSupplementalData supplemental : supplementals) {

        if (car.getId() == supplemental.getCarId()) {

            car.setLastOwner(supplmental.getLastOwner());
            car.setLastPurchaseDate(supplmental.getLastPurchaseDate());
        }
     }
}

The problem is that this boilerplate code / pattern is repeated many times in my application. Ideally, I would like to get rid of the nested looping and checking if ids match.

The only way I know to eliminate the inner looping and id matching is to change the service (second source) to take only one Car id:

for (Car car : cars) {            
    CarSupplementalData supplemental = carSupplementalService.getData(car.getId());
    //I know I have the supplemental data for this car. No id matching needed.
    car.setLastOwner(supplmental.getLastOwner());
    car.setLastPurchaseDate(supplmental.getLastPurchaseDate());
}

When performance is not an issue, is this the way to handle this pattern? When performance is an issue, is there some better way to avoid the boilerplate code induced by this pattern?

You used the tag microservices. If you do this a lot (call other microservices to enrich data), your boundaries are probably wrong. — Rik D
– Rik D, Commented Sep 1, 2021 at 19:32

Doc Brown · Accepted Answer · 2021-09-02 07:35:40Z

The boilerplate code will probably look much friendlier when you:

Convert the cars list into a map, indexed by their IDs (see here on SO how to do this in Java, a one-liner in Java 8).
Call the service with the list of IDs of the cars, as you scetched it initially.
Then iterate over the returned supplementals, retrieve the related car from the map and transfer the results (simple loop, no nestings).

That allows to call the service just once, eliminates the nested loops and makes the code look almost as simple as your second code snippet.

Note step 1 now can be put into a separate, reusable function (or the resulting map may be stored somewhere where one can reuse it directly), so it reduces the code duplication you mentioned.

The well-known programming principle behind this is to "process whole lists", a.k.a "pipes-and-filters" a.k.a. "stream processing". Separate your code into processing units where each unit gets a whole "set" (list / array / container) as input and as output, that gives you small, reusable code units which can still work pretty fast.

Stack Exchange Network

How to combine data from multiple sources into the same object?

1 Answer 1

Hot Network Questions

How to combine data from multiple sources into the same object?

1 Answer 1

Related

Hot Network Questions