Use AWS step functions for processing large amount of data?

Question

We want to use AWS step function to processing a large amount of data from a CSV file but we are not sure if this is the best choice.

Our use case is below : - We upload a CSV with a large amount of lines (Like 50K) and for each line we process a small traitements (Each traitement is processed by a lambda function). At this time, we think the best choice is to insert each line from our CSV in a DynamoDB and for each line launch our lambda functions.

What do you think of this ?

Ashan · Accepted Answer · 2017-12-26 17:19:04Z

2

There are multiple patterns to process large files with Lambda.

One approach is to use a Lambda function is to split the large file and delegate the parts to worker Lambda functions.
If the processing steps for parts are complex enough you can trigger multiple Step function workflows.

In your proposed approach, if each item processing is large enough it will make sense to process item by item, but generally its more efficient to process as batches.

answered Dec 26, 2017 at 17:19

Ashan

19.7k5 gold badges53 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Use AWS step functions for processing large amount of data?

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related