A distributed, diverse company

Puppet make open-source software configuration and infrastructure automation software. They are a 475 person strong company with people spread all over the world including the US, Canada, the Czech Republic, Sweden, the UK and Australia. Diversity is a key element of Puppet Lab’s success, bringing together developers to form a community based on different backgrounds, ethnicity and gender.

Originally Puppet was a Ruby development house. About 4-5 years they transitioned their remote service layer to Clojure, creating some high-profile and popular open-source products as a result, including Trapperkeeper and more recently Comidi.

I had a chat with Tech leads Scott Walker and Pieter Loubser about the Clojure journey at Puppet. Scott works out of the Belfast, UK office and Pieter works remotely from London.

The history of Clojure at Puppet

Jon Pither: How did Clojure get started at Puppet?

Scott Walker: About four years ago our CTO Deepak Giridharagopal introduced it. We wanted to transition to the JVM, and from there Clojure seemed like a better way of doing things rather than going with Java.

Pieter Loubser: Puppet DB was our first big Clojure project, which is over 4 years old now.

Scott Walker: We then also migrated our Ruby based Puppet-Master tool to the JVM, the new incarnation being called Puppet Server. Puppet-Server makes extensive use of Clojure, using our familiar in-house stack based on Trapperkeeper.

When we ported our Ruby app to the JVM we saw massive performance benefits. We got true concurrency; horizontal scaling on a single node using JVM threads.

And of course once it was on the JVM, we were no longer tied to Ruby and could use Clojure.

Open Source

Jon Pither: Let’s jump in to what open source Clojure offerings Puppet has built. In particular, please tell me about Trapperkeeper.

Scott Walker: Trapperkeeper provides building blocks for Clojure applications - a service lifecycle framework. It provides configuration out of the box, everything a service needs such as monitoring and metrics gathering. It saves developers from having to write their own time-consuming boilerplate.

There are components for Jetty, WebSockets, a scheduler and more. Trapperkeeper takes Java APIs such as the Apache Http Asynchronous Client and makes them accessible.

Pieter Loubser: It’s like a filo-fax - a binder for Clojure production ready code. The idea of a binder is where the name came from.

Scott Walker: Trapperkeeper has lots of users now and there’s a community built around it, so it continues to evolve.

Jon Pither: Any other prominent open-source Clojure libraries?

Scott Walker: There is Comidi allowing you to use a Compojure styled syntax to integrate with the Bidi routing library. It marries the strengths of both libraries - the expressive syntax-sugar of Compojure with the bidirectional simple data-structure approach of Bidi.

Pieter Loubser: We also have clj-kitchensink which is a library full of our commonly used utilities.

Clojure

Jon Pither: Let’s dig into Clojure at Puppet, what’s it been like adopting Clojure?

Scott Walker: I come from a computer science background. I’ve done a lot of Java, but I found that the kind of Java code I was writing was evolving to become more functional - i.e. using less mutable state where possible. Then Clojure came along and it was a perfect fit for me. With Clojure you naturally adhere to good design principles, it leads you to a different and better way of thinking.

It’s a good experience to learn a new language every once in a while and go through that process of coding up Hello World. When you freshen up your skill-set and gain new experiences it helps you to write better code overall.

Pieter Loubser: For me the Java interop of Clojure is a big deal. When I picked up Clojure I needed to spike something big, and being able to use existing Java libraries and tools helped with this.

Scott Walker: At Puppet we believe that with Clojure we can write better code in less time that is more expressive. Clojure has worked well from the get-go and people enjoy using it.

Hiring

Jon Pither: How have you found the hiring process?

Scott Walker: Clojure was one of the reasons I was attracted to Puppet. I think it was a real statement of intent by Puppet to adopt Clojure in the way they that have - especially around four years ago when it was still considered new. I was impressed and it summed the company up as being technically astute and forward looking.

Jon Pither: What about searching for Clojure developers?

Scott Walker: We don’t hire specifically for Clojure developers, rather we look for great engineers. Most of the people we hire are typically a high calibre and excited about learning Clojure.

Training

Jon Pither: What has the up-skill process being like for getting other devs into Clojure?

Scott Walker: We have a strong emphasis on self learning, where developers take responsibility as engineers to up-skill themselves. We make sure developers have the resources they need, including being able to talk the most experienced people in one particular area. Our senior developers spend lots of time coaching and advising others.

We use GitHub here and we are very tight on the code-base, ensuring that all commits have had more than a single pair of eyes on them.

Also Puppet has been involved at all the major Clojure conferences. Going to conferences is fantastic for sparking creativity and creating seeds of enthusiasm.

State of Clojure

Jon Pither: What do you feel the state of Clojure is now?

Pieter Loubser: Clojure was originally a bit hipster with people questioning why they should go off and learn a Lisp. This seems to have been reversed now with people being more open minded and this is healthy.

Jon Pither: How is Clojure doing in Belfast in particular?

Scott Walker: There’s a functional meetup called Functional Kats. They have these in Dublin and Belfast, with Clojure often being one of the languages people want to play with.

I’d say there is a core of people using Clojure in Belfast and it’s growing slowly. I think on the whole there’s a generational shift happening. Middle management is seeing what the newer generation of programmers want to use and is adapting. Opinion is shifting.

Jon Pither: Any recently developments in the Clojuresphere that excite you?

Pieter Loubser: Spec looks cool, I’m sure everyone at the moment would answer this question with Spec.

Scott Walker: It’s liberating to choose when to use tools like Spec, as oppose to with static types where you are forced down a certain road.

Technologies

Jon Pither: Any technologies you’d like to give a shout out to?

Pieter Loubser: Trapperkeeper! (See a Clojure/West video Intro to Trapperkeeper video)

Scott Walker: I’d also say have a look at PuppetDB as an example of a fairly long lived and mature Clojure project. There should be lots of idiomatic, well-composed Clojure code in there!