For context, I'm writing an application in Python that needs to poll many hosts continuously, so I create a large number of sockets to communicate with those hosts. However, I can only create 511 sockets - when I try to create 512, I get a ValueError: too many file descriptors in select(). I thought this error was referencing the maximum amount of file descriptors that a process can have open at any given time, but when I try increasing that maximum with Python's win32file._setmaxstdio(), it has no effect - no matter what I set the limit to, I can only create 511 sockets. I even tried setting the limit to a value lower than 512 just to see if it would change anything, but I could still create 511 sockets! So as far as I can tell, the limits referenced by _setmaxstdio() and _getmaxstdio() are completely unrelated to the limit to how many sockets/file descriptors select() can handle.
I tried investigating Python's select module to see if I could find where select()'s maximum is defined, or how to increase it. Python's documentation for the select.select() function doesn't mention either of those things, but it does mention that select() comes from Windows' Winsock library. So I checked Microsoft's documentation of the select() function:
Four macros are defined in the header file Winsock2.h for manipulating and checking the descriptor sets. The variable FD_SETSIZE determines the maximum number of descriptors in a set. (The default value of FD_SETSIZE is 64, which can be modified by defining FD_SETSIZE to another value before including Winsock2.h.)
I read this to mean "select() can handle 64 sockets by default, but you can change that by altering the value of FD_SETSIZE before you include the header file". So I assume Python sets it to 512 before including the Winsock2 header file? Or is select()'s limit set some other way?
I just want to know where the select() function's limit is defined, how I can check it, and if it can be increased from within Python, but I'm clearly missing something fundamental here. select() can handle some number of file descriptors, and _setmaxstdio() is used to "[set] a maximum for the number of simultaneously open files at the stream I/O level", but changing the limit with _setmaxstdio() doesn't affect the limit for select(). Why not? If select() isn't limited by the maximum amount of file descriptors you're allowed to have, then what is it limited by?