After the Honeymoon

Keeping the theory alive

As a software consultant I'm often hired to get involved in the big-bang development process, to help get projects over the delivery line. Delivery can be hard, but I've found that the most difficult part is often what happens after, when the software needs to be transitioned into long term care.

In reality this transition often involves a change in personnel. The project may have sourced external developers on a temporary basis for the initial build, or key developers will simply rotate out, mirroring the typically short developer/project lifespan.

For developers it's a common experience to inherit a codebase from others who have since moved on. With limited time and delivery pressure from the business, it can be a real challenge to understand why a system was built in a particular way, so that we avoid our changes compromising whatever graceful simplicity the system may once have had. We may resort to surviving on minimally invasive coding, despite the need for engagement with the system at a more fundamental level.

Theory Building

I've been pondering this for a while, but a nerve has been hit by reading Peter Naur's essay on 'Programming as Theory Building'. Here Naur describes software development as theory building, where developers build up a logical reasoning about how the real world problem can be best reached by code. This 'theory' - the accumulated reasoning that underpins the actual output of code - largely lives in the heads of individuals, and cannot be adequately expressed in documentation.

This might sound obvious, but it's a key understanding of the problems we face in transferring ownership of software to others. When developing a theory, as programmers we choose what parts of the real world to represent and what parts to exclude. We make compromises as to create an architecture that has the right amount of flexibility, abstraction and pragmatism. We will have a vision of how the software might evolve in the future, and we will have an internal risk-register of where the rabbit holes might be. Despite our best intentions it's not practical or feasible to write all of this down for others.

To get into the mindset of the original developer and to capture their insights is tantalizingly difficult. There are many definitions for what legacy software is out there, but Naur gives us the most compelling: software is dead 'when the programmer team possessing its theory is dissolved'. If we're simply passing on bespoke software with minimal downstream involvement, then we're just handing them dead software.

Influence on Technical Approaches

Without always explicitly stating it, many modern practices are established to try and to alleviate this problem. Microservices is one example, where we proactively split up a codebase as to avoid the underlying theory in the software being too sophisticated and onerous to capture. It's like instead of wanting to manage a sprawling garden, we move our plants and vegetables into pots. The problem is that the pots will then require more collective maintenance, but one hopes that the overall picture is more understandable and simpler.

A large body of conventional wisdom is to use automated tests to hold the theory, to serve as living documentation. This can include advanced testing DSLs such as BDD1 and Fit2 alongside various lower level mocking frameworks that try to describe what the system does, to tell the story of design. The tightrope is to avoid bringing in complexity to manage complexity.

Modern Functional Programming Languages

As a functional coder, I'm more interested in the impact that functional programming has on theory building and conveyance, and I'm hopeful the story is a good one. I've seen that the code produced by languages based on immutable data is simpler3. Without the havoc wreaked by large object models and mutable state, simplicity is surely a virtue in helping us to pass on our theories.

But is there a flip-side? We can typically see that the theory building is packed into fewer, denser lines of code, and that the entire code-base is consequently managed by fewer developers. First this needs to be celebrated, as developers are usually happier in smaller teams and more productive, and able to fit more of the system into their heads4. The opposing negative is that the theory building becomes compacted into smaller and tighter knowledge silos, and the perceived risk of the overall project is increased.

At the code level it's arguable that after the heyday of TDD5 we are seeing a reduction in the quantity of test-code. A substantial amount of theory building is now conducted via REPL development, and through iterative live code reloading in the view tier. This means we are writing less test code to assist the design process, which leads to less overall documentation (the tests that remain become more targeted towards regression).

And what of 'bottom-up programming languages' such as LISP (i.e. Clojure)? In bottom up programming we grow a DSL through function composition that allows us to mirror the business domain (a language within a language6). This investment brings about efficiencies, but I've seen on large projects that developers can tend to hover around the 'top' of the language with only a subset persevering all the way down.

Using modern functional languages does help us to produce smaller and simpler code-bases, and this is undoubtedly beneficial for delivery and what comes after. Still, there is no free lunch. It's only in parts of the industry where the clouds are clearing - where we're building software that is simpler and can be shipped faster - that we're coming up against the kind of problem that Naur was engaged with thirty years ago.

Against a background of change

I've been guilty of focusing far more on getting a project 'over the line' than with what comes afterwards. I've found that as a consultant, clients can be rightly skeptical when we leave it too late to enumerate our concerns about disengaging. At best it looks like we're changing the brief - 'you said you could cleanly deliver us a system' - and worst it looks like we're chasing our commercial agenda in light of a closing contract.

Once the main thrust of delivery has been successfully completed, projects will change. In full flow, a delivery project can be high tempo, and positively anarchistic as developers intuitively and seamlessly move around the project as part of a collective flow.

This isn't sustainable for the long term. The high tempo will need to calm down, and the business will want a more formalised process for getting their features delivered. Stakeholders will also want to de-risk the IT capability, asking the more experienced developers to mentor junior hires, with roles and responsibilities assigned.

Against a typical rush to make structural changes to a project, it can be difficult to keep championing the underlying software theory. This is especially the case if we lack a consciousness and the language to articulate our concerns.

Raising Consciousness

As Naur says we need to stop thinking about software as a production process. Software is more of a living and breathing entity, kept alive through the deep mental engagement by developers who thoroughly understand the how and the why of it.

Learning from this, in our company we stress the importance of managing rotation. Planned rotation with overlap is the best way to ensure that the baton is passed smoothly, rather than us being on the back foot with sudden departures. We need to give time for the theory to be passed on through collaboration, where the developers holding the theory are present to answer a barrage of questions and to advise on future direction.

Where possible we strive to keep original developers in the picture and with the ability to consult, even if they are no longer engaged with the day-to-day development activities. When drawing up plans for engagement with a client, how we manage the handover needs to be given priority and clarity, not left until when it happens.

We also believe that the choice of technology does make a huge difference. It's essential to choose a stack that avoids a software program becoming too bloated - consisting of hundreds thousands lines of code - when there are simpler choices. Smaller and leaner code-bases allow for future re-writes, and for subsequent developers to impose their own theories as to keep pace with the evolution of the real world problem.

Finally we must combat the idea that software should necessarily 'slow down' post delivery. Rather than slowing down, developers need to be truly craving ownership of the system and to gain intimacy with it, to develop and iterate their own theories, whilst improving the overall quality.