DEV Community

Cover image for 🕸️ Web Scraping in Python: A Practical Guide for Data Scientists
Vikas Gulia
Vikas Gulia

Posted on

🕸️ Web Scraping in Python: A Practical Guide for Data Scientists

#programming #datascience #webscraping #beginners

"Data is the new oil, and web scraping is one of the drills."

Whether you’re gathering financial data, tracking competitor prices, or building datasets for machine learning projects, web scraping is a powerful tool to extract information from websites automatically.

In this blog post, we’ll explore:

  • What web scraping is
  • How it works
  • Legal and ethical considerations
  • Key Python tools for scraping
  • A complete scraping project using requests, BeautifulSoup, and pandas
  • Bonus: Scraping dynamic websites using Selenium

✅ What is Web Scraping?

Web scraping is the automated process of extracting data from websites. Think of it as teaching Python to browse the web, read pages, and pick out the data you're interested in.

⚖️ Is Web Scraping Legal?

Scraping publicly available data for personal, educational, or research purposes is usually okay. However:

  • Always check the website’s robots.txt file (www.example.com/robots.txt)
  • Read the Terms of Service
  • Avoid overloading servers with too many requests (use time delays)
  • Never scrape private or paywalled content without permission

🧰 Popular Python Libraries for Web Scraping

Library Purpose
requests To send HTTP requests
BeautifulSoup To parse and extract data from HTML
lxml A fast HTML/XML parser
pandas To organize and analyze scraped data
Selenium For dynamic websites with JavaScript
playwright Modern alternative to Selenium

🧪 Step-by-Step Web Scraping Example

Let’s scrape quotes from http://quotes.toscrape.com — a beginner-friendly practice site.

🛠️ Step 1: Install Required Libraries 

pip install requests beautifulsoup4 pandas
Enter fullscreen mode Exit fullscreen mode

🧾 Step 2: Send a Request and Parse HTML 

import requests
from bs4 import BeautifulSoup

URL = "http://quotes.toscrape.com/page/1/"
response = requests.get(URL)
soup = BeautifulSoup(response.text, "html.parser")

print(soup.title.text)  # Output: Quotes to Scrape
Enter fullscreen mode Exit fullscreen mode

🧮 Step 3: Extract the Quotes and Authors 

quotes = []
authors = []

for quote in soup.find_all("div", class_="quote"):
    text = quote.find("span", class_="text").text.strip()
    author = quote.find("small", class_="author").text.strip()

    quotes.append(text)
    authors.append(author)

# Print sample
for i in range(3):
    print(f"{quotes[i]} — {authors[i]}")
Enter fullscreen mode Exit fullscreen mode

📊 Step 4: Store Data Using pandas 

import pandas as pd

df = pd.DataFrame({
    "Quote": quotes,
    "Author": authors
})

print(df.head())

# Optional: Save to CSV
df.to_csv("quotes.csv", index=False)
Enter fullscreen mode Exit fullscreen mode

🔁 Scrape Multiple Pages 

all_quotes = []
all_authors = []

for page in range(1, 6):  # First 5 pages
    url = f"http://quotes.toscrape.com/page/{page}/"
    res = requests.get(url)
    soup = BeautifulSoup(res.text, "html.parser")

    for quote in soup.find_all("div", class_="quote"):
        all_quotes.append(quote.find("span", class_="text").text.strip())
        all_authors.append(quote.find("small", class_="author").text.strip())

df = pd.DataFrame({"Quote": all_quotes, "Author": all_authors})
df.to_csv("all_quotes.csv", index=False)
Enter fullscreen mode Exit fullscreen mode

🔄 Bonus: Scraping JavaScript-Rendered Sites using Selenium

Some sites load data dynamically with JavaScript, so requests won't work.

🛠️ Install Selenium & WebDriver 

pip install selenium
Enter fullscreen mode Exit fullscreen mode

Download the appropriate ChromeDriver from https://chromedriver.chromium.org/downloads and add it to your system path.

🌐 Selenium Example 

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from bs4 import BeautifulSoup
import time

service = Service("chromedriver")  # Path to your ChromeDriver
driver = webdriver.Chrome(service=service)

driver.get("https://quotes.toscrape.com/js/")
time.sleep(2)  # Wait for JS to load

soup = BeautifulSoup(driver.page_source, "html.parser")
driver.quit()

for quote in soup.find_all("div", class_="quote"):
    print(quote.find("span", class_="text").text.strip())
Enter fullscreen mode Exit fullscreen mode

🧠 Best Practices for Web Scraping

  • ✅ Use headers to mimic a browser: 
headers = {"User-Agent": "Mozilla/5.0"}
requests.get(url, headers=headers)
Enter fullscreen mode Exit fullscreen mode
  • ✅ Add delays between requests using time.sleep()
  • ✅ Handle exceptions and errors gracefully
  • ✅ Respect robots.txt and terms of use
  • ✅ Use proxies or rotate IPs for large-scale scraping

📦 Real-World Use Cases

  • 📰 News Monitoring (e.g., scraping articles for sentiment analysis)
  • 🛒 E-commerce Price Tracking
  • 📊 Competitor Research
  • 🧠 Training Datasets for NLP/ML projects
  • 🏢 Job Listings and Market Analysis

📌 Final Thoughts

Web scraping is a foundational tool in a data scientist’s arsenal. Mastering it opens up endless possibilities — from building custom datasets to powering AI models with real-world information.

“If data is fuel, then web scraping is how you build your own pipeline.”

Top comments (0)