Web crawler Java

5 Aug 2025 | 7 min read

The web crawler is basically a program that is mainly used for navigating to the web and finding new or updated pages for indexing. The crawler begins with a wide range of seed websites or popular URLs and searches depth and breadth to extract hyperlinks.

The web crawler should be kind and robust. Here, kindness means that it respects the rules set by robots.txt and avoids frequent website visits. The robust means the ability to avoid spider webs and other malicious behavior.

These are the following steps to create a web crawler:

In the first step, we first pick a URL from the frontier.
Fetch the HTML code of that URL.
Get the links to the other URLs by parsing the HTML code.
Check whether the URL is already crawled before or not. We also check whether we have seen the same content before or not. If both the condition doesn't match, we add them to the index.
For each extracted URL, verify that whether they agree to be checked(robots.txt, crawling frequency)

We use jsoup, i.e., Java HTML parsing library by adding the following dependency in our POM.xml file.

Let's start with the basic code of a web crawler and understand how it works:

WebCrawlerExample.java

Output:

Let's do some modifications to the above code by setting the link of depth extraction. The only difference between the code of the previous one and the current one is that it crawl the URL until a specified depth. The getPageLink() method takes an integer argument that represents the depth of the link.

WebCrawlerExampleWithDepth.java

Output:

Difference Between Data Crawling and Data Scraping

Data crawling and Data scrapping both are two important concepts of data processing.

Data crawling means dealing with large data sets where we develop our own crawler that crawl to the deepest of web pages.

Data scrapping means retrieving data/information from any source.

Data Scraping	Data Crawling
Data scrapping extracts data not only from the web but also from any source.	Data crawling extracts data only from the web.
In data scrapping, duplication is not necessarily a part.	In data crawling, duplication is an essential part.
It can be done at any scale, i.e., small or large.	It is mostly done on a large scale.
It requires both the crawl parser and agent.	It requires only a crawl agent.

Let's take one more example to crawl articles using a web crawler in Java.

ExtractArticlesExample.java

Output: