24

I'm running a Python script that uses the requests package for making web requests. However, the web requests go through a proxy with a self-signed cert. As such, requests raise the following Exception:

requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

I know that SSL validation can be disabled in my own code by passing verify=False, e.g.: requests.get("https://www.google.com", verify=False). I also know that if I had the certificate bundle, I could set the REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE environment variables to point to those files. However, I do not have the certificate bundle available.

How can I disable SSL validation for external modules without editing their code?

Improve this question

3 Answers 3

Reset to default
57

Note: This solution is a complete hack. And works only for requests < 2.28 (https://github.com/psf/requests/pull/6074)

Short answer: Set the CURL_CA_BUNDLE environment variable to an empty string.

Before:

$ python
import requests
requests.get('http://www.google.com')
<Response [200]>

requests.get('https://www.google.com')
...
File "/usr/local/lib/python2.7/site-packages/requests-2.17.3-py2.7.egg/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

After:

$ CURL_CA_BUNDLE="" python
import requests
requests.get('http://www.google.com')
<Response [200]>

requests.get('https://www.google.com')
/usr/local/lib/python2.7/site-packages/urllib3-1.21.1-py2.7.egg/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning)
<Response [200]>

How it works

This solution works because Python requests overwrites the default value for verify from the environment variables CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE, as can be seen here:

if verify is True or verify is None:
verify = (os.environ.get('REQUESTS_CA_BUNDLE') or
    os.environ.get('CURL_CA_BUNDLE'))

The environment variables are meant to specify the path to the certificate file or CA_BUNDLE and are copied into verify. However, by setting CURL_CA_BUNDLE to an empty string, the empty string is copied into verify and in Python, an empty string evaluates to False.

Note that this hack only works with the CURL_CA_BUNDLE environment variable - it does not work with the REQUESTS_CA_BUNDLE. This is because verify is set with the following statement:

verify = (os.environ.get('REQUESTS_CA_BUNDLE') or os.environ.get('CURL_CA_BUNDLE'))

It only works with CURL_CA_BUNDLE because '' or None is not the same as None or '', as can be seen below:

print repr(None or "")
# Prints: ''
print repr("" or None )
# Prints: None
Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

I've successfully used this technique on Unix, but cannot get it to work on Windows
Also with set CURL_CA_BUNDLE="" not working on Windows for me
This is amazing in its simplicity and hackiness :) Confirming this works as described on macOS 10.14 with Python 3.7/3.8.
@David: That's how variables work in Bash and similar shells. If you just do CURL_CA_BUNDLE="", that only sets the variable within the context of that specific shell or for the command that follows on the same line. If you use export CURL_CA_BUNDLE="", then it will also apply to any new processes started from that shell, like Python (but still not ones in completely separate processes).
This no longer works with requests 2.28.0 or newer, since it was considered a bug, see github.com/psf/requests/issues/6071...
|
4

I saw this hack only due to some trouble with my private CA.

The given hack with CURL_CA_BUNDLE='' will not longer work in 2022 with next minor release of requests (that means 2.28 ?).

Please refer to GH issue 6071 which was fixed 6 days before in Feb. 2022

Improve this answer

3 Comments

Yeah... so are there any other hacks other than pinning the version of requests?
Use requests 2.27.0 (pip install requests==2.27.0)
pip install requests==2.27.1 because hvac 2.1.0 requires requests<3.0.0,>=2.27.1, but you have requests 2.27.0 which is incompatible.
4

My problem was that a third party package sent the request, so that I had no chance to influence the verify = True.

What helped me:

pip install pip-system-certs

And then:

import pip_system_certs #import as first package
Improve this answer

1 Comment

This is the only solution that worked for me in Windows.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.