node js script exits prematurely

Question

Promise newbie here.

I'm trying to retrieve icon_name field from asset database, Equipment table in mongodb
and update icon_id field in equipments database, equipments table in mysql.

I have about 12,000 records with icon_name field in Equipment.

The script runs successfully however it doesn't seem to go through all the records. When I check the equipments table there are only about 3,000 records updated.

I tried running the script several times and it appears to update a few more records each time.

My suspicion is the database connection is close before all the queries are finished but since I use Promise.all I don't know why it happened.

Here is the script

const _ = require('lodash'),
    debug = require('debug')('update'),
    Promise = require('bluebird')

const asset = require('../models/asset'),
    equipments = require('../models/equipments')

const Equipment = asset.getEquipment(),
    my_equipments = equipments.get_equipments(),
    icons = equipments.get_icons()

Promise.resolve()
.then(() => {

    debug('Retrieve asset equipments, icons')
    return Promise.all([
        icons.findAll(),
        Equipment.find({ icon_name: { $ne: null } })
    ])
})
.then(([my_icons, asset_equipments]) => {
    debug('Update equipments')
    const updates = []

    console.log(asset_equipments.length)
    asset_equipments.forEach((aeq, i) => {

        const icon_id = my_icons.find(icon => icon.name === aeq.icon_name).id
            up = my_equipments.update(
                { icon_id },
                { where: { code: aeq.eq_id } }
            )

        updates.push(up)
    })


    return Promise.all(updates)
})
.then(() => {
    debug('Success: all done')
    asset.close()
    equipments.close()
})
.catch(err => {
    debug('Error:', err)
    asset.close()
    equipments.close()
})

Thanks in advance.

Answer 1

Code looks fine but spawning 12000 promises in parallel might cause some trouble on the database connection level. I would suggest to batch the concurrent requests and limit them to let's say 100. You could use batch-promises (https://www.npmjs.com/package/batch-promises)

Basically something like

return batchPromises(100, asset_equipments, aeq => {
            const icon_id = my_icons.find(icon => icon.name === aeq.icon_name).id;
      return my_equipments.update({ icon_id }, { where: { code: aeq.eq_id } });
        });