6

I have an application on an AWS EC2 instance that runs once daily. The application fetches some files from a web service, parses the files line by line, updates a database, updates S3 files based on changes in the database, sends notification emails to customers as well as a few other tasks.

This is a series of logical tasks that must take place in sequence, although some of the tasks can be thought of as sub-tasks that can be executed in parallel. All tasks are a combination of Perl scripts and Java programs, with a single Perl script acting as the manager that executes each in turn. Some tasks can take as long as 45 minutes to complete, and the whole process can take up to 3 hours in total.

I'd like to make this whole process serverless. My initial idea was to use AWS Lambda, whereby each task would execute as a Lambda function, until I discovered Lambda functions impose a 5 minute execution timeout. It seems like the AWS Step Functions service is actually a better fit for my use case, but my understanding is that this service is backed by Lambda, so the tasks will still have the 5 min execution limitation.

(I'm also aware that I would have to re-write my Perl scripts to a language supported by Lambda).

I assume that I can work around the execution time limit by refactoring my code into smaller functions that will guarantee to complete in under 5 minutes. In my particular situation though, this seems inefficient.

Currently the database update task processes lines from a file one at a time. For this to work with Lambda, a Lambda function would need to handle only a single line from the file (or a very small number of lines) in order to guarantee not spilling over 5 minutes execution time. This would involve opening and closing a connection with the database on every invocation of the Lambda function. Also, each line processed should result in an entry written to a file, to be stored in S3. Right now, I just keep a file handle in memory and write the file to S3 when all lines are processed, but with Lambda I would need to keep reading the file, updating it and writing it back to S3.

What I'm asking is:

  • Is my use case a bad fit for AWS Lambda and/or AWS Step Functions?
  • Have I misunderstood how these services work?
  • Is there another AWS service that would be a better fit for my use case?

After further research, I think AWS Batch might be a good idea.

Improve this question

3 Answers 3

Reset to default
2

What you want are called Activity Workers. Tl;dr: You register "activities" and each gets an ARN. Then you can put that ARN in the resource field of Task states and then you run some code (the "worker") somewhere (in a Lambda, on EC2, in your basement, wherever) that polls for tasks identified by that ARN, then calls back to report success or failure. Activity Workers can run for up to a year.

Step-by-step details at the AWS docs

In response to RTF's comment, here's a deeper dive: Suppose you have code to color turtles in color_turtles.pl. So what you do is call the CreateActivity API - see http://docs.aws.amazon.com/step-functions/latest/apireference/API_CreateActivity.html - giving the name "ColorTurtles" and it'll give you back an ARN, a string beginning arn:aws... Then in your state machine you make a Task state with that ARN as the value of the resource field. Then you add code to color_turtles.pl to poll the service with http://docs.aws.amazon.com/step-functions/latest/apireference/API_GetActivityTask.html - whenever a machine you're running gets to that task, it'll go look for activity workers polling. It'll give your polling worker the input for the task, then you process the input and generate some output, and call SendTaskSuccess or SendTaskFailure. All these are just REST HTTP calls, so you can run them anywhere and I mean anywhere; in a Lambda, on an EC2 instance, or on some computer anywhere on the Internet.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

Oh, your Activity Workers can be in Perl :) But you might want to change anyhow.
I've been up and down those docs for a few hours, and I don't understand where the task code itself is supposed to be located. For example, if I define a state machine (SM) and execute it from the console, my SM definition will indicate a state (task) to start at. That task is associated with an ARN, which is an activity that I created for the task. But how do I define the code to run for that task, or tell it where the code is?
Activity workers are all fine IF you don't have any specialist environment requirements, for example if you're running sphinxsearch etc etc - good catch though if he's just running code it'll work nicely.
Mrk Fldig, I think I disagree. Your activity worker can run anywhere that can make an HTTP connection, so It's not obvious how running any particular package would get in the way. Am I missing something?
It seems like this solution would work best if I create docker containers for each of the individual steps/tasks in my application and register the dockers with EC2 Container Service. Apparently that's possible, but I still don't understand how I get Step Functions to run/launch the code in the docker image e.g. associate the docker with an activity
|
2

So to answer your questions:

1) Yeah, if you've got something that'll run for around 45 minutes, whilst you could engineer it with Lambda/Step functions you're probably better off getting a EC2 micro instance.

2)Nope you've pretty much got it.

3) As above you want to go with EC2 for this, there's a good article on using Data Pipelines to start / stop an EC2 instance here that way by starting instance only when you need it the cost(if any) is negligible.

I have jobs that run in this fashion normally you can get away with with a t2.micro instance which is free tier eligible.

You can also run your perl scripts on an EC2 instance so no need to rewrite them!

Improve this answer

4 Comments

The data pipelines solution sounds interesting. How would you launch an application on your EC2 instance from the pipeline? Do you have to configure the instance to run your code when it's started by the pipeline?
Well you could I suppose, or just setup a cron job on the instance but use Data Pipelines to start it a bit before the cron is supposed to begin.
Yeah, I would have to configure a user-data script or cloud-init etc. because I need the ability to run the pipeline on demand if necessary e.g. if the whole workflow failed and needed to be re-tried.
It turns out Step Functions has what he needs, they're called Activity Workers.
0

I will start with that it seems you are looking for workflow solutions on AWS. SWF and Step functions are the two most popular ones. Steps function is more recent offering and encouraged by AWS more than SWF.

SWF has native capability to handle long-running tasks, the downside is that you have to provide your own execution environment for deciders (can't use lambda).

With step functions, you can do this in two different ways. One of the approaches is suggested by Tim in his answer. There is an alternative way to achieve the same which is using job poller in step functions. Job pollers have the ability to call (poll) your resource and find out if the task is done and if not you can send execution in wait mode for the specified time. As mentioned above maximum execution time allowed currently for any workflow is 1 year. In case you have tasks which may take longer than 1 year, you can't use step functions in its current form.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.