Multiple Concurrent Operations in Node.js

Question

I'm wondering what the best way to handle multiple concurrent operation in node.js. For example, let's say a given HTTP request needs to make 3 database queries before it can render the response, but each of the requests are independent. My current naive implementation works like this (in pseudo code)

doRequest1(query, function (response1) {
   doRequest2(someQuery, function (response2) {
      doRequest3(someQuery, function (response3) {
          renderHTML(response1,response2,response3);
      });
   });
});

Is there a better way to do this? It seems wasteful to wait for the response to each of the queries to start the next one.

Answer 1

You should use a flow control library, like async

https://github.com/caolan/async

Or step

https://github.com/creationix/step

Edit: added an example with async

async.parallel([
    function(callback){
        doRequest1(query, callback);
    },
    function(callback){
        doRequest2(query2, callback);
    },
    function(callback){
        doRequest3(query3, callback);
    },
],
function(err, results){
    if (err) {
        renderError(err);
    } else {
        renderHTML(results[0], results[1], results[2]);
    }
});

If you want to make each function run as series (wait for the previous function to complete), you may use async.series instead of async.parallel. If queries depend on the result of previous queries, you may use async.waterfall. If there is a complex dependency graph between queries, you may use async.auto.

Answer 2

Gruikya's answer states what you should do, here's how it works behind the scene:

var unfinished = 3
var noop = function() {}
var data1, data2, data3
doRequest1(query1, function(err, results) {
  if (err) return cb(err), cb = noop
  data1 = results
  if (--unfinished === 0) done()
}
doRequest2(query2, function(err, results) {
  if (err) return cb(err), cb = noop
  data2 = results
  if (--unfinished === 0) done()
}
doRequest3(query3, function(err, results) {
  if (err) return cb(err), cb = noop
  data3 = results
  if (--unfinished === 0) done()
}
function done() {
  // do stuff with the data here
  cb(null) // everything ok
}

Answer 3

Update using ES6 features

You can use async/await syntax and the Promise.all method to get the same result in a single line without needing callbacks:

const allRequests = [doRequest1(query), doRequest2(someQuery), doRequest3(someQuery)]
const [response1, response2, response3] = await Promise.all(allRequests)