Python socket limit (1024) error

Question

Any idea how to solve this kind of problem with Python 2.7.8 ? As seen, there are used some site-packages, e.g. gevent, but AFAIK the issue is the limit error: IOError: cannot watch more than 1024 sockets

  File "C:\Python27\lib\httplib.py", line 806, in send
    self.connect()
  File "C:\Python27\lib\site-packages\requests-2.3.0-py2.7.egg\requests\packages\urllib3\connection.py", line 106, in connect
    conn = self._new_conn()
  File "C:\Python27\lib\site-packages\requests-2.3.0-py2.7.egg\requests\packages\urllib3\connection.py", line 90, in _new_conn
    (self.host, self.port), self.timeout, *extra_args)
  File "C:\Python27\lib\site-packages\gevent\socket.py", line 574, in create_connection
    sock = socket(af, socktype, proto)
  File "C:\Python27\lib\site-packages\gevent\socket.py", line 242, in __init__
    self._read_event = io(fileno, 1)
  File "core.pyx", line 466, in gevent.core.loop.io (gevent/gevent.core.c:8305)
  File "core.pyx", line 804, in gevent.core.io.__init__ (gevent/gevent.core.c:14393)

IOError: cannot watch more than 1024 sockets

Any idea?

Answer 1

You probably just need to increase the open file descriptor limit on your system. For more, see here: increase ulimit for # of file descriptors - the default is usually 1024 on Linux.

Answer 2

If you are using grequests to make the requests, you can use a session

sess = requests.Session()
reqs = [grequests.get(u, /* params */, session=sess) for u in urls]
results = grequests.map(reqs)