Thursday, February 9, 2012

python | head == IOError: [Errno 32] Broken pipe

I spent way more time today than I could afford fighting against a python script that, when piped to head, generated the following message:

IOError: [Errno 32] Broken pipe

Now had this been a simple program I could have just caught the IOError and managed, but that didn't work. The program was using Popen from the subprocess module and and when I "handled" the exception I got a weird exception:


tail: stdout: Broken pipe
close failed in file object destructor:
Error in sys.excepthook: 
Original exception was:

Note, the original exception was empty...

I am a Python neophyte, and of course it took me a long time to figure out how to properly handle the broken pipe. The meat of the matter is that Python does not ignore SIGPIPE, it creates an exception for it. My googling and stackoverflowing lead me to more exception handling including replacing the sys.excepthook and other hackery.

The solution was of course much simpler and eventually found on the comp.lang.python archive. I needed to restore the signal handler for SIGPIPE doing the following:


from signal import signal, SIGPIPE, SIG_DFL 
#Ignore SIG_PIPE and don't throw exceptions on it... (http://docs.python.org/library/signal.html)
signal(SIGPIPE,SIG_DFL) 
for x in range(100000):
        print x
The Python manual was a bit misleading the way that I parsed the following:

Python installs a small number of signal handlers by default: SIGPIPE is ignored (so write errors on pipes and sockets can be reported as ordinary Python exceptions) and SIGINT is translated into a KeyboardInterrupt exception. All of these can be overridden.

What seems to really happen is that SIGPIPE is translated to an IOError, at least in Python 2.6.1, and in order to make it not you need to use the SIG_DFL one.  Note that SIG_IGN doesn't work.





Monday, September 29, 2008

Stack Overflow During WPF Model Generation Crashing WASCE

We ran into a NASTY bug this morning in Portlet Factory that took all morning to recover from. On opening a model the WASCE server window would close and the model would show the somewhat usual "could not generate metadata" messages.

After about 30 minutes I caught the WASCE window with the following text at the bottom of it.

Server started.
Unrecoverable Stack Overflow: addr=7c90edde, ee=5f313e0, er=b052d18
JVMDG217: Dump Handler is Processing Signal 11 - Please Wait.
JVMDG303: JVM Requesting Java core file
It didn't matter what model got opened. If the WASCE server was running and a model was opened it it crashed. I rolled back to yesterday, I removed almost all of my models and tried to use a simple test case to no avail.

Then I figured it had been about 3 months and there might be a fix pack for WPF. There it was version 6.1.0.1! Go get it NOW!

The bug that was causing the issue had to do with the Use Parent Profiling option:
Ref 32338 In the Imported Model builder, "Use Parent Profiling" causes a recursion errorParent profiling can now be used in the Imported Model builder without producing a Stack Overflow error.
There are several more that look helpful as well, but if you are using Imported Models you need to have this fix.

Things appear to be back to normal. Let's hope for a better afternoon.

Sunday, September 28, 2008

Portlet Factory Java Expressions

I found something in Portlet Factory that I have wondered about for a few months now. When you pick an item from the Java category of the reference/value picker there are some useful values for the java expressions. USERID is one I use all the time. I have been typing the ${Java/Boolean.TRUE} reference quite a few times lately, and I really wanted to add it and others to the list.

In order to add to the list you need to add items to the WebContent/WEB-INF/config/IndirectJavaExpressions.config file. It is a properties file format, and therefore pretty easy to figure out.

Here is what I added:
  • TRUE=Boolean.TRUE
  • FALSE=Boolean.FALSE
  • NULL=null

I use those almost every time I use the tool. I am sure I will figure out more as I go. Now I just need to remember to take .cvsignore away for that file.

Sunday, August 3, 2008

At least 150 miles to go

It looks like I am running out of time to train for my first century ride next weekend. I will be riding 105 miles on Saturday, August 9th and at least 45 miles on Sunday.

It has all lead up to this. Over 500 miles in July and August, almost bi-weekly spin classes, as many long rides as I could fit in and a 3/4 mile 12.5% grade climb the end of July. All I can do at this point is hope I am ready.

I will be riding as a member of Sylvia's Dream Team, and we hope to have 14 people riding this year. Not a bad growth rate from Mike, Dwight and Me last year!

My goal is to raise $1000 and ride at least 150 miles. That works out to about $6.67 a mile.

I would love your support, both financial and your thoughts! Visit my participant center to track how I am doing

Sunday, July 6, 2008

What a 4th of July weekend!

I have to say that this was probably one of the best 4th of July weekends I can remember. Certainly one of the best since I was a kid. The weather was gorgeous, and we spent the weekend at home together. I got out for a couple hours on the bike, but otherwise we were together, and that is what made it great.

We got to go for a ride together on the bike path with a trail-a-bike. I was amazed at how Ayden and I talked the whole way. He made it 9 miles (with more than a few stops) on the bike path, and I can't remember when I have had a better time! He even peddled most of the way, which does help.

I sat on the deck yesterday and watched Ayden swing on the swingset for all he was worth. The joy he gets from such a simple thing is inspiring.

It is amazing how much you can learn from an (almost) 5 year old when you take the time to look and listen. I need to remember to do that more often.

Sunday, June 29, 2008

WebSphere Portlet Factory -- Introductions

I have been working in WebSphere Portlet Factory lately and I am beginning to get a feel for how it can make web development easier, especially when building portlet applications. Learning to use it involves a different way of thinking about building web applications. The process of building applications using it initially warped my mind the same way that learning LISP did after many years of C.

I have yet to find a good series of tutorials that teach portlet factory using the "hello world" style. Two starting points for the newbie are the wiki and the redbook. The wiki has some good content, but not a good starting from scratch set of tutorials. The redbook does do starting from scratch, but in a grandiose way that forces you to ignore errors that are displayed until you finish things later... It is a mammoth work, some 730 pages in all. It is a little intimidating when you are just starting out.

Another recommended area to get support is the forums (9 in all). The 4 most relevant to the beginner are:
The problem is that there are 9 of them and 4 are useful, and you are never really sure where to search for the answer to your question. I haven't found a way to search across only these 4 forums and frequently have to search across everything at IBM, which is not exactly helpful...

One thing to know is that the tool is far from free. If you own WebSphere Portal (6 and above) you are entitled to one seat of the designer, and Eclipse plugin that allows you to build portlet factory applications. You are also entitled to run those applications inside of the WebSphere Portal environment. Things get more complicated if you want to run outside of WebSphere Portal (more on that another day), and the pricing for the designer standalone is similar to that of IBM's Rational Application Developer product.

That being said, I think that the cost might be justified when stacked against the complexity of building applications in the J2EE, even with the support of frameworks like Spring. It really is remarkable how quickly you can put a web application together that displays data or does CRUD.

You can download the Portlet Factory 6.1 Beta from the WebSphere Portal 6.1 Beta Site.

If I can find my blog legs I will be doing more on this topic, and may even take a shot at the "hello world" style tutorials.

Sunday, December 31, 2006

Getting into the (Spring Web) Flow

I spend my days doing web application development in Struts, and lately have been spending more and more time developing portlet applications for WebSphere Portal. I recently been using a new framework that eases some of the problems that I have recently had using Struts to build flow based web and portal applications.

Problem 1
One of the things I consistently find difficult to do in Struts is restricting a user's navigation to a well defined path, especially when users start tampering with URLs. In order to ensure that we are in a valid state we:
  1. Use subclassing of actions, where the parent action defines an abstract execute method for the child to implement and checks state for us before delegating to the child to handle the request. This usually leads to very tight coupling, and a headache when something needs to be changed.
  2. We duplicate code in each action to make sure we are in the correct state. Maybe we use helpers, but the code still smells.
Problem 2
Lately, I have been working with Struts newbies. I have seen that the separation between action mappings in the Struts configuration file and the action calls embedded in the views makes it difficult to comprehend the "flow" of an application. To see what is happening you have to process all of the views, looking for forms, links and javascript looking for submissions to action url mappings in the struts config file.

Problem 3
As I try to bring Struts to bear in portal development I have found it even harder to keep things coherent (IMHO, Struts has more than a bit of an impedance mismatch with portal development). A consultant that I worked with introduced me to the portlet state pattern, which is a simple solution that works well for portlet applications. The general idea is that navigation is controlled by the state that the application is currently in, the current state is stored in the session, and actions modify the current state. The controller can ensure that only actions which the current state allows are executed. One shortcoming is that the transitions are encoded in separate Java classes, meaning that the documentation value that a Struts config file would have provided is gone. Given the time I might have come up with an externalized configuration solution but I never quite got around to it.

A better way?
In building the next generation of a login/registration/self-help front end for an existing portal I quickly realized that I didn't want to write bunches of code to make sure a user had not changed a url and navigated past part of the login or registration flow (see Problem 1). What I wanted was the portlet state pattern for a Struts web application. I looked at solutions for Struts and found a few:
Then I found Spring Web Flow.

In less than an hour I had the existing main flow for the application defined with stub pages, and was adding the new states. I ended up breaking some of it down later into sub-flows, as well as playing around with the different types of states more than I probably should have, but it was fun! What I found after a day of work was a well defined, mature, testable, self-documenting framework for defining and enforcing flow in web applications. There are certainly non-flow based web applications out there, but I do know that the vast majority of the web applications that I have written in my life need some way of enforcing a known set of transitions between states.

One of the things that surprised me was how easy it was to "get something working" with mocked up pages and "fake" actions but the navigation that I would expect between pages, and then drop in real actions/data/views one at a time, having a "working" application at each step of the way.

SWF is that it isn't coupled to a web application framework (to be fair Commons Chain isn't either). This means that it should work great when I am wearing my portlet development hat, and that it is testable without needing to mock up a servlet environment. To accomplish this SWF implements an abstraction layer over the Servlet API. This means that it can be tested outside of a servlet container, and that it can work with other APIs (like the portlet API). In reality it is an engine that other tools and frameworks call into. For more information look at the documentation for ExternalContext. It works with Struts, Spring MVC and JSF out of the box.

For those "afraid" of XML configuration, there is a fully functional programatic API for building your flow. I found the XML configuration more practical, and it has the benefit of being able to be manipulated using XSLT to generate documentation for the flow (that my boss, and the customer, could read). I even hacked together an XSLT template to generate a dot file that can be processed by graphviz to generate a visual representation of the flow (This won't be necessary when the Spring IDE WebFlow Editor is complete, hopefully someday soon).

A couple of stumbling blocks I hit along the way:
  1. Many of the tutorials that are out in the wild are for a pre-1.0 release of the framework and therefore might not be a 1:1 match to the current version. The project has been around for quite a while (some history). The API, and configuration syntax has changed for the 1.0 release and I did hit a couple of snags trying to follow the examples that I found out on the net. There is a whole set of examples bundled with the framework that were a good introduction. The manual for the framework is a good technical reference, but is far to low level. One thing I think that the project needs is a good quick start tutorial that is up to date with the 1.0 API.
  2. The configuration uses OGNL to define expressions which are used to retrieve values from the contexts and pass to underlying objects. The error messages that are generated when the OGNL expression can't be parsed or bound are sometimes "less than helpful". I don't know if more work on error handling from the parser would help in this case, or if there is not enough contextual information being passed back by the parser.
The SWF framework addresses several of the weaknesses of using Struts to build flow based applications. It provides the ability to force a navigation path, has a single point of reference for the possible states and transitions that a flow can be in, and works across web frameworks. It even supports portal development out of the box.

I will be using Spring Web Flow more in the near future, and can't wait to get to try it on some of the more complicated portlets that I have been working on. I know it would have made the last search/display/edit/create portlet that I did a lot easier. I wonder if I should rewrite it using SWF...

Note: One interesting project that looks similar to SWF is Commons SCXML, which is based on a W3C specification. The format of the configuration files is similar to the SWF configuration file, and it will be interesting to see how the two technologies grow with/against one another.


Links: