Summary

Usage of HttpCompressionMiddleware needs to be relfected in Scrapy stats.

Motivation

In order to estimate scrapy memory usage efficiency and prevent.. memory leaks like this.
I will need to know:

1. number of request/response objects that can be active (can be achieved by using [trackref](https://docs.scrapy.org/en/latest/topi

If you're using proxies with requests-html and rendering JS sites is all good. Once you render a website pyppeteer don't know about this proxies and will expose your IP. This is an undesired behavior when scraping with proxies.

The idea is that whenever someone passes in proxies to the session object or any method call, make pyppeteer also use these proxies. #265

Add .nextElementSibling and .previousElementSibling properties to DOM Element that point to sibling elements.

• nextElementSibling
• previousElementSibling

Main examples at Apify SDK webpage, Github repo and CLI templates should demonstrate how to manipulate with DOM and retrieve data from it.

Also add one example of scraping with Apify SDK + jQuery to https://sdk.apify.com/docs/examples/basiccrawler

Feedback from: https://medium.com/better-programming/do-i-need-python-scrapy-to-build-a-web-scraper-7cc7cac2081d

I lost an hour trying to make

My project have routing based on hosts. But web driver make request to http://127.0.0.1:9080.
How can i change host?

In #549, we added unittests for dispatch AtCoder and Codeforces.
To increasing test coverage, we should add unittests for other contest (HackerRank, CS Academy, ...).

Not only tests for all other contests, tests for the part of contests are welcome.

Is your feature request related to a problem? Please describe.

I'm worried that the format function is brittle.

Describe the solution you'd like

It should always return html. And never fail. I could use some help writing more tests for this function (that is run on every repr and str call).