Using certificates in urllib3

Question

I'm a Python newbie. I'm using urllib3 to talk to an api. The reason I'm using this and not requests is that I'd like to host my app on GAE. My app uses certicates. When I post data, I get the following error:

TypeError: __init__() got an unexpected keyword argument 'cert_reqs'

How can I include certs in my urlopen call? A snippet of code follows

CA_CERTS = ('client-2048.crt', 'client-2048.key')

http = urllib3.PoolManager()
r = http.urlopen('POST', url, body=payload, headers={'X-Application': '???', 'Content-Type': 'application/x-www-form-urlencoded'}, cert_reqs='REQUIRED', ca_certs=CA_CERTS)
print r.status, r.data

Answer 1

You can drop down to the HTTPSConnectionPool level which you may do directly:

from urllib3.connectionpool import HTTPSConnectionPool
conn = HTTPSConnectionPool('httpbin.org', ca_certs='/etc/pki/tls/cert.pem', cert_reqs='REQUIRED')

Or, more simply or via the connection_from_url() helper function:

conn = urllib3.connection_from_url('https://httpbin.org', ca_certs='/etc/pki/tls/cert.pem', cert_reqs='REQUIRED')

Note that ca_certs is the file name of a certificate bundle used to validate the remote server's certificate. Use cert_file and key_file to present your client certificate to the remote server:

conn = urllib3.connection_from_url('https://httpbin.org', cert_file='client-2048.crt', key_file='client-2048.key', ca_certs='/etc/pki/tls/cert.pem', cert_reqs='REQUIRED')

Then issue your request:

response = conn.request('POST', 'https://httpbin.org/post', fields={'field1':1234, 'field2':'blah'})
>>> print response.data
{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "field1": "1234",
    "field2": "blah"
  },
  "headers": {
    "Accept-Encoding": "identity",
    "Connection": "close",
    "Content-Length": "220",
    "Content-Type": "multipart/form-data; boundary=048b02ad15274fc485c2cb2b6a280034",
    "Host": "httpbin.org",
    "X-Request-Id": "92fbc1da-d83e-439c-9468-65d27492664f"
  },
  "json": null,
  "origin": "220.233.14.203",
  "url": "http://httpbin.org/post"
}

Answer 2

you should pass the cert_reqs='REQUIRED' and ca_certs=CA_CERTS args to the PoolManager() instantiation directly.

So the original example can be changed to this:

CA_CERTS = ('client-2048.crt', 'client-2048.key')

http = urllib3.PoolManager(cert_reqs='REQUIRED', ca_certs=CA_CERTS)

r = http.urlopen('POST', url, body=payload, headers={'X-Application': '???', 'Content-Type': 'application/x-www-form-urlencoded'})
print r.status, r.data

Answer 3

Passing User-Agent header seemed to help in my case, on top of the other answers.

Not sure if this is a common behaviour, but my server would return 403 - Access denied error when performing a HTTPS request using self-signed certificates and without User-Agent.

When ignoring certificates (ie. using an empty ssl.SSLContext) the User-Agent header wasn't required and the request would succeed. Only when passing a self-signed certificate using the ca_certs parameter, I needed to include the User-Agent

http = urllib3.PoolManager(cert_reqs='REQUIRED', ca_certs='/path/to/cacert.pem')
r = http.urlopen('GET', url, headers={'User-Agent': 'myapp/v0.1.0'})
print(r.data)

I cannot find any source indicating why User-Agent may be required when using a self-signed certificate. Any clarification on that point most welcome.

Read more about User-Agent header here.