curl https://getsquirrel.schollz.com | bash
Downloading the web can be cumbersome if you end up with thousands or millions of files. This tool allows you to download websites directly into a file-based database in SQLite, since SQlite performs faster than a filesystem for reading and writing.
Install
Download the latest release for your system, or install a release from the command-line:
$ curl https://getsquirrel.schollz.com | bash
On macOS you can install the latest release with Homebrew:
$ brew install schollz/tap/squirrel
Or, you can install Go and build from source (requires Go 1.11+):
$ go get -v github.com/schollz/squirrel
Usage
Basic usage
It should be compatible with Firefox's "Copy as cURL", just replace curl with squirrel. By default it will save the data in a database, urls.db.
$ squirrel "https://www.sqlite.org/fasterthanfs.html" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H "Accept-Language: en-US,en;q=0.5" --compressed -H "Referer: https://www.google.com/" -H "Connection: keep-alive" -H "Upgrade-Insecure-Requests: 1" -H "If-Modified-Since: Thu, 02 May 2019 16:25:06 +0000" -H "If-None-Match: ""m5ccb19e2s6076""" -H "Cache-Control: max-age=0"
Contributing
Pull requests are welcome. Feel free to...
- Revise documentation
- Add new features
- Fix bugs
- Suggest improvements
Thanks
Thanks Dr. H for the idea.
License
MIT