ValueError: filedescriptor out of range in select on macOS

[peteboothroyd]

When using KazooClient (on macOS) I occasionally get the error: ValueError: filedescriptor out of range in select thrown from the _select method of handlers/threading.py (specifically the line return select.select(*args, **kwargs)).

I believe this occurs when a file descriptor has a value which is too large, which is not necessarily when there are too many open files (ie the max file descriptor number > number of open files). This has been fixed by using epoll for Linux machines but not on macOS where epoll is not available. Please could you provide an equivalent method which uses kqueue when available?

[peteboothroyd]

Using the python selectors module might be a good platform agnostic way to avoid this problem.

[StephenSorriaux]

Hi,

Thanks for this issue. I might not have any time on my side to investigate this problem, but any PR are very welcomed.

[peteboothroyd]

It's a bit tricky for me to be able to contribute back because of my company's rules (not impossible though).
I have a fairly simple solution which utilises the selectors standard library package, but have a few questions:

• Is the exception watch list xlist actually used anywhere? It is a parameter and return argument for the current implementations, but as far as I can tell it is never used. The implementation I have does not return any exception list, so I wonder if this is a problem or not.
• What is the stance on python2 compatibility? I (sadly) cannot see that you've signed the python3 statement. selectors was introduced in python 3.4 and would require a back port for earlier versions

[peteboothroyd]

For anybody else who comes across this. Replacing the select method in threading.py with the following worked for me:

def select(self, rlist, wlist, _, timeout=None):
    selector = selectors.DefaultSelector()
    fileobj_event_masks = defaultdict(int)
    for read_fileobj in rlist:
        fileobj_event_masks[read_fileobj] |= selectors.EVENT_READ
    for write_fileobj in wlist:
        fileobj_event_masks[write_fileobj] |= selectors.EVENT_WRITE

    for fileobj, event_masks in fileobj_event_masks.items():
        selector.register(fileobj, event_masks)
        
    key_events_ready = selector.select(timeout)
    revents = [key.fileobj for key, event in key_events_ready if event & selectors.EVENT_READ]
    wevents = [key.fileobj for key, event in key_events_ready if event & selectors.EVENT_WRITE]

    return revents, wevents, []

[peteboothroyd]

@StephenSorriaux any update?

[StephenSorriaux]

@peteboothroyd

• you are right, xlist is not used anywhere. Maybe we should consider it.

• since selectors is available on Python 3.4 and beyond, and that Kazoo is compatible with Python 3, I suggest to use the selectors module when its available and keep the current way if not