Pattern to transform and cache data from a DB?

Question

We have a server that receives queries. We use the command pattern for this, e.g.

q := NewQuery(database)

q.Execute(request1)
q.Execute(request2)

The queries read a "model" from the database

type database interface {
    ReadModel(id string) *Model
}

Problem #1: The model is stored raw. But when executing the query, we need to convert it to an ExtendedModel that has methods for easy manipulation: like this

model := database.ReadModel(id)
extendedModel := extended_model.New(model)
// or 
extendedModel, err := extended_model.NewAndValidate(model)

How do we avoid repeating that?

I feel uneasy about changing the database interface method. It's not the DB's responsibility to convert the model...
If I keep the code as is, we run into:

Problem #2: To reduce database lookups, we want a cache of ExtendedModel that all the queries can use. Each server has its own cache in memory.

How do we solve this in a way that doesn't allow us to "forget" that there is a cache of ExtendedModels?

Right now, we manually read from the cache (like here), but in some commands we forgot to remove the unneeded DB read.

It's not the DB's responsibility to convert the model . Why? If the conversion is done consistently after each fetch, it looks like the best place to do the mapping to the extended model is database. You could do it automatically or pass a mapper along with the ID to the get function, so the function either returns Model or the the type generated by the mapper. — Laiv
– Laiv, Commented Nov 18, 2023 at 9:48
In many data access layers, row/data mapping is fairly common and a legit concern to solve from within the layer — Laiv
– Laiv, Commented Nov 18, 2023 at 9:49

dm03514 · Accepted Answer · 2023-11-17 20:14:30Z

Problem #1: The model is stored raw. But when executing the query, we need to convert it to an ExtendedModel that has methods for easy manipulation:

How do we avoid repeating that?

If ExtendedModel is required for every caller, I would consider making ExtendedModel as part of the exposed API, and force callers to depend on this interface instead of the database interface, which sounds like it only exposes partial functionality needed by callers.

type Database interface {
  ExtendedModel(id string) (*ExtendedModel, error)
}

Problem #2: To reduce database lookups, we want a cache of ExtendedModel that all the queries can use.

Exposing and requiring the ExtendedModel interface enables encapsulation of the cache calls. Consider the following cached db:

type CachedDB struct {
  db database
  cache LRUCache
}

func (c *CachedDB) ExtendedModel(id string) (*ExtendedModel, err) {
  // get the db model
  m, err := c.db.ReadModel()
  if err != nil {
    return nil, err
  }
  // do cache work 
}

This ExtendedDatabase knows how to perform db lookups and has caching logics built in. The caller just needs to call extended model to retrieve their model, all of the database specifics and cache logic is encapsulated for them.

The caller will have no idea if their model gets pulled from the cache or the database, they just know that when they have an Id, they can get an ExtendedModel

J_H · Accepted Answer · 2023-11-17 18:42:37Z

First, ensure that extendedModel can do anything you'd want model to do, or at least can hand back a model. Now app code needn't keep track of a model any more.

There's two changes to the Public API that you still need to add. You want app code to usually obtain an extended model via a caching layer. And you want to do that by supplying an id, since the caching layer will be supplying that to ReadModel. May as well bundle both changes into a single method.

Maybe call it GetExtendedModel(id). That becomes "the front door" by which app developers are expected to access the DB. Deprecate the interfaces they were previously using, so that only the caching layer and unit tests access them. Do a code audit, and use PR code reviews to ensure that new application calls always arrive via the front door.

Code audits might benefit from some additional support. It could be added during a transition period and then removed from the codebase once you're happy with app code.

Add a flag to models and/or to extended models. Your existing code knows nothing about the flag. The caching layer will set the flag, indicating that app developer did the Right Thing.

Eventually Execute will be called, and it can log any requests which failed to set the flag. Treat such logs as a burn-down list of call sites you still need to fix, so they use the new caching layer.

Stack Exchange Network

Pattern to transform and cache data from a DB?

2 Answers 2

Hot Network Questions

Pattern to transform and cache data from a DB?

2 Answers 2

Related

Hot Network Questions