A little like that j-thing, only in Go.
Go 10.8k 810
Polite, slim and concurrent web crawler.
Go 2k 192
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Go 762 98
tiny Go library to normalize URLs
Go 404 59
Package rehttp implements a Go HTTP transport that handles retries.
Go 183 14
Proper URL escaping as per RFC3986
Go 23 14
Loading…