Fetch data using fetched data?

Question

I'm trying to fetch new products from a supplier website. On their new products page is a bunch of URLs and thumbnails, and clicking a URL takes you to it's product page.

I'm fetching all of the URLs from the page, then want to fetch data from each of those URLs. This works, but it's very slow and i'm not sure how to rewrite it so it's cleaner and faster.

import fetch from "isomorphic-fetch";
import cheerio from "cheerio";

export const fetchNewProducts = async (page) => {
  try {
    const req = await fetch(
      "https://www.supplier.co.uk/newarrivals/?setPerPage=25&search_direction=asc&pageID=" +
        page
    );
    const html = await req.text();
    const $ = cheerio.load(html);

    let newProducts = [];

    for (let i = 1; i < 26; i++) {
      let pageSrc = $(
        `#product_listing > tbody > #_${i} > td:nth-child(2) > a`
      ).attr("href");

      pageSrc = "https://www.supplier.co.uk" + pageSrc;

      const req2 = await fetch(pageSrc);
      const html2 = await req2.text();
      const $2 = cheerio.load(html2);

      let imageSrc = $2(
        "#product-main-image .main-image-inner:first-child img"
      ).attr("src");
      const name = $2("#product-details dd:nth-child(2)")
        .text();
      const brand = $2("#product-details dd:nth-child(4)")
        .text();
      const price = $2("#product-details dd:nth-child(6)")
        .text();

      newProducts.push({
        name,
        imageSrc,
        brand,
        price,
        pageSrc,
      });
    }

    return newProducts;
  } catch (err) {}
};

module.exports = {
  fetchNewProducts,
};

sid · Accepted Answer · 2021-08-18 12:04:39Z

2

I'd suggest you to unleash the power of Promises. It's much more than just async/await.

I've created an example for you that does exact same thing; to fetch data from http requests simultaneously instead of a loop.

Demo: https://codesandbox.io/s/parallel-http-requests-vvrcv (click on console there to view the output)

Useful link to learn how parellel requests work using Promises: https://javascript.info/promise-api

Happy coding & good luck!

edited Aug 18, 2021 at 12:04

answered Aug 18, 2021 at 11:56

sid

2,10710 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Liiaam93 Over a year ago

Awesome ! I'll bookmark those for when my brain is working hah. I had been meaning to learn about Promises, but wasn't sure how or why i'd need to. Thanks!

Tobias S. · Accepted Answer · 2021-08-18 11:48:27Z

You can fetch all of the URLs in parallel and wait for the result using Promise.all().

const pageSrcs = []

for (let i = 1; i < 26; i++) {
   let pageSrc = $(
     `#product_listing > tbody > #_${i} > td:nth-child(2) > a`
   ).attr("href");

   pageSrc = "https://www.supplier.co.uk" + pageSrc;
   pageSrcs.push(pageSrc);
}

await Promise.all(pageSrcs.map(pageSrc => fetch(pageSrc)
 .then(res => res.text())
 .then(html2 => {
    const $2 = cheerio.load(html2);

    let imageSrc = $2(
      "#product-main-image .main-image-inner:first-child img"
    ).attr("src");
    const name = $2("#product-details dd:nth-child(2)")
      .text();
    const brand = $2("#product-details dd:nth-child(4)")
      .text();
    const price = $2("#product-details dd:nth-child(6)")
      .text();

    newProducts.push({
      name,
      imageSrc,
      brand,
      price,
      pageSrc,
    });
})))

return newProducts;

Amazing thanks! I'm still relatively new to JS, kinda jumped into the deep end, whilst i feel like i've learned a lot, there's still so much i'm not sure about.

Collectives™ on Stack Overflow

Fetch data using fetched data?

2 Answers 2

1 Comment

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Related