Sequential async operations for each in Node JS

Question

I am working on a database migration. This requires querying one DB, getting an array of records and perform a set of async operations to insert the data in the new DB. In order to keep data consistency, I want to insert the records one at the time so I want each operation to run sequentially. The only way I have found to do these is using recursion.

Is there a cleaner way of doing this same thing? I know there is a library called async https://caolan.github.io/async/v3/ which I never tried before.

The recursive method I have written looks like this:

const insertItem = async (data) => {
    let item = data[0];

    if (!item) {
        //I am done return
        return;
    }

    try {
        //Do multiple await calls to insert record into new database
    } catch (e) {
        //Recover from error (DB rollbacks, etc)
    } finally {
        //Remove inserted or failed item from collection.
        data.shift();
        await insertItem(data);
    }
};

//Query original database
getInfo().then((data) => insertItem(data));

Why not just loop through the data? If you want it sync it sequentially just wait for promise to resolve before moving to next record — Shubham Srivastava
– Shubham Srivastava, Commented Aug 27, 2020 at 5:46

xMayank · Accepted Answer · 2020-08-27 05:50:47Z

1

You can use sync loop for...of it will wait for HTTP response.

const dataArr = ['data1', 'data2', 'data3'];

async function processItems(arr){
  for(const el of arr) {
    const response = await insertData(el);    
    // add some code here to process the response.
  }
};

processItems(dataArr);

answered Aug 27, 2020 at 5:50

xMayank

1,9852 gold badges8 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

traktor · Accepted Answer · 2020-08-27 07:03:44Z

The posted code achieves iterating through the data collection (an array) retrieved from the first data base by using data.shift to mutate the argument array before calling the single function handling everything recursively.

To clean this up, remove the shift and recursive calls by splitting the data processing function into two:

one function to step through data retrieved from the first data base,
a second function to insert records in the second data base as required.

This removes the need for the .finally clause and leaves the structure of the code looking more like

async function insertData(data) {
    for( let index = 0 ; index < data.length; ++index) {
        await insertItem( data[index]);
    }
}

async function insertItem( data) {
    try {
        //Do multiple await calls to insert record into new database
    } catch (e) {
        //Recover from error (DB rollbacks, etc)
        // throwing an error here aborts the caller, insertData()
    }
}

getInfo().then( insertData).catch( /*... handle fatal error ...*/);

Depending on preferred style, insertItem could be declared as nested function within insertData to keep it looking neat, and insertData could be written as an anonymous function argument of the then call after getInfo().

It is possible, of course, to perform asynchronous operations sequentially be other means, but using await inside an async function is perhaps the simplest coding method.

Thank you @traktor53 . It does make sense and it is far better than my original code. However I believe what xMayank posted looks simpler.Thank you both!!!

Collectives™ on Stack Overflow

Sequential async operations for each in Node JS

2 Answers 2

Comments

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Related