Making the while loop synchronous

Question

I've got a following piece of code

var page = 2;
var last_page = 100;
while(page <= last_page) {
  request("http://some_json_server.com/data?page=" + page, function (error, response, body) {
    if (!error && response.statusCode == 200) {
      store_data(body)
    }
    page++;
  });                
}

I've done the following, but it is actually not retrieving anything. Am I doing this correctly?

var page = 2;
var last_page = 100;
while(page <= last_page) {
var async_arr = [];
async_arr.push(
  function(next) {
    request("http://some_api_url?page=" + page, function (error, response, body) {
      if (!error && response.statusCode == 200) {
        store_data(body);
      }
    });
  }
);

async.series(
  async_arr, done
);

Regarding your edit, you have to call next() in the request callback — Andreas Hultgren
– Andreas Hultgren, Commented Apr 9, 2013 at 13:03

katspaugh · Accepted Answer · 2013-04-09 13:12:31Z

55

With while you get a busy loop, which is counter-purpose in Node.

Make it a recursive function instead. Each call will be done in a separate tick.

var page = 2;
var last_page = 100;

(function loop() {
    if (page <= last_page) {
        request("/data?page=" + page, function (error, response, body) {
            if (!error && response.statusCode == 200) {
                store_data(body)
            }
            page++;
            loop();
        });
    }
}());

answered Apr 9, 2013 at 13:12

katspaugh

18k12 gold badges69 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

theGrayFox Over a year ago

+1 What if you didn't know the number of pages? How would you handle this @katspaugh

katspaugh Over a year ago

@TheGrayFox, the response should somehow indicate either the number of pages, or just send you no data so you know you shouldn't make the next request.

Eric Uldall Over a year ago

A recursive function will eventually eat up all of your memory and you will exceed your call stack size.

Eric Uldall Over a year ago

@katspaugh assuming the use case never exceeds page 100 it will probably not cause any problems and its recursion only occurs inside of the request callback it will probably not cause issue. Just something good to note if you plan on using this pattern.

Eric Uldall Over a year ago

I think you should look into using the http agent. nodejs.org/api/http.html#http_new_agent_options I've found that you can get far better control over large number of http requests by using the agent as a custom request pool.

|

Andreas Hultgren · Accepted Answer · 2014-09-04 06:25:23Z

You're looking for async.whilst(). This solution is assuming you actually want to do each request after the other. As @UpTheCreek mentions (edit: the comment I referred to was edited) it would likely be possible to do it asynchronously and keep track of each result using async.parallel.

var page = 2,
    lastPage = 100;

async.whilst(function () {
  return page <= lastPage;
},
function (next) {
  request("http://some_json_server.com/data?page=" + page, function (error, response, body) {
    if (!error && response.statusCode == 200) {
      store_data(body)
    }
    page++;
    next();
  });
},
function (err) {
  // All things are done!
});

Damn, I wish you had answered my question I just asked just now. I think this is exactly what I'm looking for.

seanbehan · Accepted Answer · 2018-12-15 16:23:52Z

You can also wrap your while loop in async and break after your promise resolves/conditions have been met...

    const request = require("request")

    ;(async()=>{

    let results = []
    while(true){
      await new Promise(resolve => {
        request('http://www.seanbehan.com/', (err, resp, body)=>{
          console.log(new Date, 'Downloading..')
          results.push(body)
          resolve(body)
        })
      })

      if(results.length >= 5){
        break
      }
    }

    console.log(results)
  })()

Collectives™ on Stack Overflow

Making the while loop synchronous

3 Answers 3

7 Comments

1 Comment

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

7 Comments

1 Comment

Comments

Linked

Related