BlockingIOError exception ignored; should I be concerned?

user_5031373 0
Viewed 5041

I'm writing a script that is 20,000 tasks that each make a subprocess call and a TCP call or two. To make this not take all day, I'm using Python's new asyncio.

However, I'm concerned about these errors Python prints out while my script runs:

Exception ignored when trying to write to the signal wakeup fd:
BlockingIOError: [Errno 11] Resource temporarily unavailable

It will print a bunch of them, but not raise any exceptions. I did get OSErrors about Too many open files and dropped connections with the servers before, but I used semaphores to only allow 100 connections to each server at a time and only 700 connections total.

Since Python's not raising any exceptions, I can't catch the errors. But it doesn't seem to affect the script.

Are these errors something I should be concerned about? If so, what do I need to do to get rid of them? If not, how do I get rid of them so they're not in my program's output?

Also, if these errors are serious, why does Python ignore them instead of raising an exception?

python-asyncio python-3.x linux python
1 Answers

It looks like the limiting factor is running lots of short-lived subprocesses. From the Python bug tracker:

"Exception ignored when trying to write to the signal wakeup fd" message comes from the signal handler in Modules/signalmodule.c. The problem is that Python gets a lot of SIGCHLD signals (the test scripts creates +300 processes per second on my computer). The producer (signal handler writing the signal number into the "self" pipe) is faster than the consumer (BaseSelectorEventLoop._read_from_self callback).

With the patch, I start getting messages with 140 concurrent processes, which is much better :-) IMO more than 100 concurrent processes is crazy, don't do that at home :-) I mean processes with a very short lifetime. The limit is the number of SIGCHLD per second, so the number of processes which end at the same second.

I changed my code to restrict how many create_subprocess_execes could run at once. I stopped seeing the errors when I got below 35, though I'll probably set it to 20 just to be sure. Your mileage may vary.

async def myTask(stuff, semaphore, loop):
    with semaphore:
        process = await asyncio.create_subprocess_exec('short_program', loop=loop)

def taskRunner(stuffs):
    loop = asyncio.get_event_loop()
    semaphore = asyncio.Semaphore(20)  # limit how many can run at a time
    tasks = [
        asyncio.ensure_future(myTask(semaphore, loop))
        for i in range(20000)
    ]
    loop.run_until_complete(asyncio.gather(*tasks))
    loop.close()
user_50313730
Related