Scrapy project

Repositories

• scrapy

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  python crawler framework scraping crawling
  Python 8,778 38,193 447 (34 issues need help) 289 Updated Sep 5, 2020

• itemloaders

  Library to populate items using XPath and CSS with a convenient API
  Python BSD-3-Clause 2 17 2 2 Updated Sep 3, 2020

• itemadapter

  Common interface for data container classes

  python metadata python3 scrapy python-dataclasses python-attrs
  Python BSD-3-Clause 2 10 3 5 Updated Sep 1, 2020

• queuelib

  Collection of persistent (disk-based) queues
  Python 45 197 2 5 Updated Aug 26, 2020

• booksbot

  Forked from stummjr/books_crawler

  A crawler for http://books.toscrape.com
  Python 637 27 0 2 Updated Aug 8, 2020

• scrapy.org

  The scrapy.org website

  html
  HTML 97 38 3 5 Updated Aug 4, 2020

• scrapyd

  A service daemon to run Scrapy spiders
  Python 510 2,128 100 24 Updated Jul 31, 2020

• scrapy-bench

  A CLI for benchmarking Scrapy.

  python web-crawler scrapy command-line-tool benchmark-suite scrapy-bench
  Python MIT 14 26 6 1 Updated Jul 27, 2020

• quotesbot

  This is a sample Scrapy project for educational purposes
  Python MIT 634 936 0 6 Updated Jul 13, 2020

• parsel

  Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

  css python xml scraping selectors xpath lxml
  Python 89 564 21 (1 issue needs help) 10 Updated Jun 18, 2020

• w3lib

  Python library of web-related functions

  python
  Python 83 309 12 11 Updated May 13, 2020

• scrapy-itemloader Archived

  [Archived] Library to populate Scrapy items using XPath and CSS with a convenient API
  Python BSD-3-Clause 7 5 2 0 Updated May 5, 2020

• cssselect

  CSS Selectors for Python

  css python selectors
  Python 44 225 14 12 Updated Apr 28, 2020

• protego

  A pure-Python robots.txt parser with support for modern conventions.

  python robots-txt robots-parser
  DIGITAL Command Language BSD-3-Clause 10 13 0 0 Updated Apr 6, 2020

• scrapyd-client

  Command line client for Scrapyd server
  Python 102 548 18 9 Updated Dec 6, 2019

• scrapely

  A pure-python HTML screen-scraping library
  HTML 250 1,700 25 5 Updated Nov 28, 2019

• loginform

  Fill HTML login forms automatically
  Python 67 232 9 2 Updated Oct 18, 2019

• scurl

  Performance-focused replacement for Python urllib

  python cython chromium gurl urlparse
  Python Apache-2.0 6 16 10 (1 issue needs help) 1 Updated Oct 2, 2018

• url-chromium

  url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url

  chromium gurl
  C++ 2 0 0 0 Updated Aug 7, 2018

• base-chromium

  base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/
  C++ 3 1 0 0 Updated Jul 31, 2018

• dirbot

  Scrapy project to scrape public web directories (educational) [DEPRECATED]
  Python 1,132 1,597 0 0 Updated Oct 27, 2017

• scrapy-bench-speedcenter

  Forked from Parth-Vader/scrapy-bench-speedcenter

  Codespeed for scrapy-bench
  Python 2 2 0 0 Updated Aug 28, 2017

• pypydispatcher

  A fork of http://pydispatcher.sourceforge.net/ with PyPy support
  Python 3 12 1 0 Updated Jul 3, 2017

• slybot
  60 221 5 0 Updated Apr 28, 2015

• gsoc2014-integration-tests

  GSoC2014 - Scrapy Integration tests project
  Shell 3 3 0 0 Updated Mar 18, 2014