Thursday, February 9, 2012

python | head == IOError: [Errno 32] Broken pipe

I spent way more time today than I could afford fighting against a python script that, when piped to head, generated the following message:

IOError: [Errno 32] Broken pipe

Now had this been a simple program I could have just caught the IOError and managed, but that didn't work. The program was using Popen from the subprocess module and and when I "handled" the exception I got a weird exception:

tail: stdout: Broken pipe
close failed in file object destructor:
Error in sys.excepthook: 
Original exception was:

Note, the original exception was empty...

I am a Python neophyte, and of course it took me a long time to figure out how to properly handle the broken pipe. The meat of the matter is that Python does not ignore SIGPIPE, it creates an exception for it. My googling and stackoverflowing lead me to more exception handling including replacing the sys.excepthook and other hackery.

The solution was of course much simpler and eventually found on the comp.lang.python archive. I needed to restore the signal handler for SIGPIPE doing the following:

from signal import signal, SIGPIPE, SIG_DFL 
#Ignore SIG_PIPE and don't throw exceptions on it... (
for x in range(100000):
        print x
The Python manual was a bit misleading the way that I parsed the following:

Python installs a small number of signal handlers by default: SIGPIPE is ignored (so write errors on pipes and sockets can be reported as ordinary Python exceptions) and SIGINT is translated into a KeyboardInterrupt exception. All of these can be overridden.

What seems to really happen is that SIGPIPE is translated to an IOError, at least in Python 2.6.1, and in order to make it not you need to use the SIG_DFL one.  Note that SIG_IGN doesn't work.


Sean said...

Thank you for this article. I was trying to write a quick replacement for shuf on my mac, and I couldn't figure out why it was breaking on pipes - this was the exact problem.

Unknown said...

This was a helpful article. I was trying to figure out how to prevent my python scripts from throwing the same error. The only issue still left here is how to catch that exception within the script itself; that would be nice so (for example) the script could have time to flush out a pickle file rather than just terminating.

Ryan Dietrich (dextius) said...

Thank you so much for this article. This needs to be the top result on every search engine for this problem. Great explanation.

Alwin Co Daan said...

Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Java developer learn from Java Training in Chennai. or learn thru Java Online Training in India . Nowadays Java has tons of job opportunities on various vertical industry.