Threading strategies for a webservice implementation

Question

If I am building a webservice/Web API for servicing requests, there are 2 threading strategies I can think of.

I will explain this in terms of Java (though the question may be relevant for any other language also)

1. In the main thread, I accept the request and pass it to a thread. The thread does the processing.

2. In the main thread, I accept the request and pass it to a thread. This thread just puts the request into a job queue and exits. You have a separate thread pool which processes requests in the queue and processes them.

How do I pick of these methods?

In the first method, I can run as many run as many threads as my OS/memory allows me and each request starts getting processed as soon as it is received. However, things may slow down if there are too many threads. In the 2nd method, if I have a fixed number of worker threads, then if all threads are occupied, then new requests may have to wait before the processing starts.

There is an SLA that there will be an average of so many requests per hour and each webservice call will have be processed within x seconds.

Is there any theory about how to figure out how best to do this. A lot of testing will be done to compare but is there already some theory for this? What factors need to be considered?

This is a Java program running on Tomcat.

Answer 1

Your first option doesn't make sense. There is no need to switch to new thread, you just process request in the thread you receive it and let container to manage threads.

Second option is useful when you do not need to wait until request is processed. You throw it in queue and forget about it. It allows you to process long request without HTTP timeout. It also help with uneven load.

There is no universal simple rule when to use one strategy over another.