Getting a Grip on the Small Things

One of the most effective ways to bring a software project to success is to aggressively pursue simplicity throughout the entire design. Simplicity, as Rich Hickey and the dictionary define it, means not intertwined. It’s a piece of criteria than can be objectively assessed. The best designers have a multitude of different techniques at their disposal to achieve this goal at every layer of a system.

At the highest level, usage of queues, transactions, commutativity, and immutable data structures help us to architect beautiful systems compromised of multiple programs. Further down the stack, pure functions, communicating sequential processes, and software transactional memory guide us to the promised land of decomplected. Perhaps even a little below this level lies the challenge of effectively designing for error handling, logging, monitoring, metrics, and conditions. What constructs does the masterful designer turn to for fending off this potential complexity?

I think the answer to this question is that many of us have simply given up. I’ve seen too many software projects littered with functions like the following:

It’s my contention that the small things often end up drowning us in complexity. This hypothetical function, while only 10 lines, is hopelessly complected in at least 7 ways. Aside from actually executing a query and returning its results, it also performs logging, precondition validation, records metrics, logs error states, handles error conditions, and determines error values.

In my opinion, most developers choose to ignore the littering that ensues from this approach because aspects like logging and metrics tend to usually not affect the output of the function. I’m not saying this is actually the case, because excessive logging can fill up the local disk, or metric calls can raise socket errors. But the vast majority of the time, less skilled developers are operating in the closed mode and care only about the fact that the output is typically as expected.

We know better.

"But no matter what technology you use …, the complexity will eventually kill you. It will kill you in a way that will make every sprint accomplish less - most sprints be about completely redoing things you’ve already done. And the net effect is that you’re not moving forward in any significant way." - Rich Hickey

To these challenging aspects, I have devised a solution that dramatically reduces complexity. About a year and a half ago, I invented and open sourced Dire. Dire provides decomplected, ad-hoc error handling, conditions, and a few other helpful constructs. Dire was, in my opinion, a step in the right direction.

Perhaps the most interesting thing one can do with Dire is conditional loading of function modifiers. This technique pleasantly lets you compose loggers, exception handlers, conditions, and other common aspects. The recent API addition for removing these function modifiers means that runtime decisions can be made to add or drop aspects.

This technique was criticized by others. With the flexibility of runtime composition, we lost temporal control over who, and when, these aspects were being added or removed. I spent a couple of months in search of a remedy to this incidental complexity. I never came up with an answer.

Sometime later, I was happy to hear that Stuart Sierra opened sourced a library called Component. Stuart’s library provides just enough structure to control the stateful parts of a program in a way that’s particularly easy to understand. Conceptual components can be composed, dependencies made explicit, and set up/tear down convenient from the REPL. Stuart unknowingly finished the puzzle that tortured me.

The following Gist shows how we can capture each aspect as a Component record (it’s on GitHub too). This allows for aspects to be composed and returns the temporal control that we previously lost. I’ve used this technique of Dire in combination with Component many times over the last few months with great success. In particular, this makes a stellar way to slot Riemann metrics reporting into your program without mucking up your application logic.

I hope this helps you build genuinely simpler programs.

My name is Michael Drogalis. I’m an independent software engineering consultant and contractor. Interested in working with me? Send a tweet over to @MichaelDrogalis or an email to mjd3089 at rit dot edu. I’d be delighted to hear from you.

Tags: clojure