[10.x] Enable DynamoDB as a backend for Job Batches #49169
Conversation
driesvints
changed the title
| 'application' => ['S' => $this->applicationName], | ||
| 'id' => ['S' => $batchId], | ||
| ], | ||
| 'ConsistentRead' => true, |
There was a problem hiding this comment.
Would it be safer to use ConsistentRead in the first place for find() and get()? What's the impact of returning potentially stale data here?
There was a problem hiding this comment.
The updates are where it actually matters and those are done atomically where AWS guarantees it updates the right data.
The updates are also independent from the read data because they use update expressions SET count = count + 1.
Because of this a ConsistentRead isn't required in general.
ConsistentRead is also 2x the cost of a read on AWS so I don't think it should be imposed as a default.
|
@khepin Will this allow the batches to be displayed in the Horizon UI still? |
|
I have no idea. Horizon is a part of the Laravel ecosystem I am really not familiar with. |
Context
Similar to how it's already possible to use DynamoDB to store failed jobs, this would enable DynamoDB as a store for job batches.
Key Proposed Differences with DB version
Make use of TTLs
The configuration makes it possible to (optionally) use TTLs on the created batches.
This way batches can be set to be fully removed after, for example, 30 days since last update to the batch.
The TTL for a batch gets reset on every update to the batch itself.
There is a monetary benefit to this because deletes based on TTLs do not cost anything in DynamoDB (as of today at least).
No pruning
Pruning in the DB version is done based on
finished_atwhich isn't part of the key for the data we're storing here.It would therefore require a secondary index in DynamoDB which incurs cost on AWS.
Since the TTL mechanism is available for a store like Dynamo, this seemed like a better option in this specific case.