Scrapinghub

Repositories

autoextract-poet

web-poet definitions for AutoExtract

Python 0 1 0 0 Updated Aug 3, 2020

spidermon

Scrapy Extension for monitoring spiders execution.

testing monitoring scraping crawling spiders monitoring-tool scrapinghub

Python BSD-3-Clause 55 265 34 (1 issue needs help) 12 Updated Aug 3, 2020

dateparser

python parser for human readable dates

Python BSD-3-Clause 309 1,527 131 (2 issues need help) 49 Updated Aug 3, 2020

scrapy-poet

Page Object pattern for Scrapy

Python BSD-3-Clause 9 24 3 3 Updated Jul 29, 2020

varanus

A command line spider monitoring tool

spider monitoring python36

Python 5 6 2 1 Updated Jul 28, 2020

scrapy-autounit

Automatic unit test generation for Scrapy.

Python BSD-3-Clause 12 28 11 5 Updated Jul 28, 2020

js2xml

Convert Javascript code to an XML document

Python MIT 16 112 1 0 Updated Jul 28, 2020

extruct

Extract embedded metadata from HTML markup

microformats semantic-web rdfa json-ld microdata opengraph

Python 76 495 15 (2 issues need help) 13 Updated Jul 28, 2020

scrapinghub-autoextract

Python clients for Scrapinghub AutoExtract API

Python BSD-3-Clause 3 14 4 0 Updated Jul 24, 2020

python-crfsuite

A python binding for crfsuite

data-science crf crfsuite

Python MIT 189 646 42 3 Updated Jul 22, 2020

web-poet

Web scraping Page Objects core library

python web-scraping page-objects

Python BSD-3-Clause 3 17 0 1 Updated Jul 18, 2020

splash

Lightweight, scriptable browser as a service with an HTTP API

Python 412 3,046 291 24 Updated Jul 17, 2020

article-extraction-benchmark

Article extraction benchmark: dataset and evaluation scripts

Python MIT 6 32 1 0 Updated Jul 16, 2020

shub

Scrapinghub Command Line Client

Python 56 95 33 (6 issues need help) 7 Updated Jul 15, 2020

crawlera-headless-proxy

A complimentary proxy to help to use Crawlera with headless browsers

crawler proxy scraping

Go MIT 14 55 2 (1 issue needs help) 0 Updated Jul 10, 2020

autoextract-spiders

Pre-built Scrapy spiders for AutoExtract

Python BSD-3-Clause 5 8 2 3 Updated Jul 8, 2020

epmdless_dist

Erlang 0 2 0 0 Updated Jul 3, 2020

woodpecker

Forked from laszlocph/woodpecker

An opinionated fork of the Drone CI system

Go 18 0 0 0 Updated Jul 1, 2020

webstruct-demo

HTTP demo for https://github.com/scrapinghub/webstruct

Python MIT 2 3 0 2 Updated Jun 30, 2020

frontera

A scalable frontier for web crawlers

Python 203 1,100 76 (10 issues need help) 22 Updated Jun 24, 2020

python-scrapinghub

A client interface for Scrapinghub's API

Python 49 155 18 2 Updated Jun 22, 2020

shub-workflow

Python BSD-3-Clause 7 5 1 1 Updated Jun 17, 2020

baseimage-docker

Forked from phusion/baseimage-docker

A minimal Ubuntu base image modified for Docker-friendliness

Python MIT 1,033 1 0 1 Updated Jun 9, 2020

docker-erlang-otp

Forked from erlang/docker-erlang-otp

the Official Erlang OTP image on Docker Hub

Dockerfile Apache-2.0 45 0 0 2 Updated Jun 9, 2020

scrapinghub-stack-scrapy

Software stack with latest Scrapy and updated deps

Dockerfile BSD-3-Clause 8 36 1 0 Updated May 25, 2020

marathon-apps-collectd-plugin

Forked from jsargiot/marathon-apps-collectd-plugin

marathon-apps-collectd-plugin

Python GPL-2.0 84 2 0 0 Updated May 25, 2020

mochiweb

Forked from shaneaevans/mochiweb

MochiWeb is an Erlang library for building lightweight HTTP servers.

Erlang 470 0 0 1 Updated May 20, 2020

docker-images

Dockerfile 9 28 0 3 Updated May 9, 2020

portia

Visual scraping for Scrapy

Python 1,265 7,838 104 9 Updated May 9, 2020

kafka-docker

Forked from wurstmeister/kafka-docker

Shell Apache-2.0 2,069 0 0 1 Updated May 6, 2020