International Betting
Sky Betting and Gaming is a gambling company that is entirely online. They run a UK operation from Leeds and an international operation from Sheffield, started as a greenfields project two years ago.
They are ranked in the Sunday Times top 100 best places to work in 2016 and again in 2017.
I chatted with Principal Software Engineers Ed Bowler and Paul Brabban to understand this Clojure adoption story.
Background
Jon: Could you give me an overview of the project?
Ed: Sky Betting and Gaming recently decided to expand from the UK into international markets. The international arm started in Sheffield so as to not distract from the incredibly successful UK operation, based in Leeds.
Paul: We went live in Italy last year and we’re seeing even better growth than we hoped. We’re all really happy with an increasing share of the market there and our mobile app shot to the top bookmaker spot within weeks!
Ed: We have about 1,100 people now across all of Sky Betting and Gaming. There’s around 50 of us in Sheffield for the International Tribe. The tech comprises mainly Clojure, Scala, Java, Node.js, and React.js on the front end.
The Clojure aspect is our team. The teams here are divided up by function, for example we have a sportsbook team, a casino game team, an accounts team, and ours, a data team.
It’s a nice place to work because we have the freedom to choose the technology we use and we have autonomy within teams.
Clojure Journey
Jon: How did Clojure get started at Sky Betting and Gaming?
Ed: When I started I was given the remit of working out how to generate financial reports and to do it in a serverless way. Having used Clojure in the past, I picked Clojure for building a proof of concept. We needed a low impact way of generating reports; integrating with 3rd parties such as CRM systems, marketing systems, and real time analytics.
We picked AWS Lambda to crunch the data and to output it to Amazon RDS. We took an Event Sourcing approach; as we developed the platform we collected as much data as we could and figured out how to report on it later.
Jon: What was your journey to Clojure before Sky Betting and Gaming?
Ed: My road to Lisp started in 2007 using Common Lisp. In 2009 I discovered Clojure and in 2010 I created a startup that actually used Clojure. We went for five years until we ran out of money.
I joined Sky Betting and Gaming as a Principal Java developer, and I’ve since written just 5 lines of Java.
Jon: How has Clojure been received?
Ed: 99% of people are interested and looking around. A good number of those are really interested and are picking it up and learning it. Some Microservices have been started outside the team.
The people I’ve forced to use Clojure have picked it up incredibly well.
Jon: Is it a matter of time before the whole ship becomes Clojure?
Ed: If I had my way!
We started an FP guild following the the spotify model of cross cutting guilds. Every fortnight there’s an hour long guild meetup with cake (I’ve put on a stone). I started the guild to spread the idea and to get the good news out.
We played with 4Clojure and talked about concepts of the Clojure language.
IDES
Jon: What IDEs do you use?
Paul: Emacs.
Ed: I’ve been using Emacs since 1999. Even when I was using Java I was using Emacs.
We have about a few Clojure developers. Two are using Emacs, one is using Cursive, and another is using Proto-Repl with Atom.
Hiring
Jon: Do you worry about being able to hire in Sheffield?
Ed: You need people that are smart and get things done.
All languages have pluses and minuses. With Clojure you need to get past the brackets, which should take about an hour. Any smart person can pick it up, especially if they are guided. Paul had a 2-3 month gap before joining; in his own time he picked up Emacs and Clojure in one go and became immediately productive. This was seriously impressive and it vindicated the whole approach. Pick up smart people up and they will learn this stuff.
Paul: I did Java for a few years. After that, I worked with a team to rewrite a Java monolith over to NodeJS microservices in a fairly functional style. When I interviewed here I’d had fun with a couple of functional languages and been through the SICP book. The opportunity to try something completely different with a Lisp that was properly functional was too big an opportunity for me to miss.
Lambdas:
Jon: What is your stack?
Paul: AWS. The majority of our code runs on AWS Lambda. We originally had a single large repository, but we’ve been breaking it down and extending it into lots of small code repos. The downside to Clojure is that each Lambda Function ships with several megabytes of the JVM Clojure runtime as well as the AWS SDK. That gives us quite slow cold startup times.
Jon: Have you looked at ClojureScript for Lambda?
Ed: Not initially as I had more experience with the JVM and more confidence with it. However relatively large artifacts mean the startup and deployment times do have an impact. Recently we have started trying to use ClojureScript for a faster startup.
Our Lambdas tend to stay running; polling and chewing through records. The startup times are much less of an issue than they might be in, say, an API endpoint.
Paul: We typically have Kinesis queues in front of the Lambdas, which is essentially a simplified, hosted Kafka.
Jon: How do you provision the Lambdas?
Paul: We use AWS CloudFormation - we build them with Clojure using Crucible and use Spec to validate. Our Continuous Delivery pipeline then provisions using the templates. We’ve made over 600 deployments since we started using Travis last summer.
Tech shout out
Jon: What technologies / libraries do you use?
Ed: We use Amazonica quite a bit; it makes it so fast to get up and running with the AWS tech.
Paul: The Amazonica maintainer, Michael Cohen is really responsive and super on the ball.
Ed: We use Cheshire for JSON fast parsing, HTTP kit the client library is useful for neat handling of async invocation and connection pooling.
Paul: Leiningen and a private Travis CI subscription for builds. We just introduced scheduled builds using Travis’ new feature to run dependency-related checks. The folks maintaining lein-dependency-check took some PRs to bring it up to date, and of course lein-ancient.
Ed: - Clojure.test and Timbre.
Spec is worth a mention. We’ve seen what Spec has to offer, but it will take a while before we know how to effectively use it.
Clojure
Jon: Is Clojure giving you actual business value, rather than just adding the cool factor?
Ed: Data processing is what it’s all about. With Clojure we’re already a level up. Paul Graham talks about building the language and API up to the business. At Sky Betting and Gaming we are getting there but we’re not that far along; it’s been an interesting journey to take.
The benefits to the business are that we have a very small team being very productive. We have one of the best burn-down rates. You could maybe argue that the people are just superstars, but you could also point to the tech. It’s a combination of good people and good tech.
Paul: There are three developers on the data tech side, and a testing specialist who’s taking the chance to learn Clojure for test automation too. The team would have made it work whatever… it’s a pros and cons thing. The choice to use Clojure makes us very attractive to other developers; we are getting a lot of interest from people with an interest in Functional Programming which is potentially useful.
Ed: Ostensibly Clojure is a practical language and is all about getting stuff done.
Paul It’s very quick to get things up and moving, especially when I think back to what I was doing with Java.
Ed: The means to abstraction are more flexible in Clojure. You can keep the domain logic separate from the operational aspects such as handling errors etc. You get this separation far easier than in a language like Java where you have objects.
Paul: In Clojure I can get on with solving the business problem and then add the safety features in afterwards. Clojure is very tolerant of change compared with other languages that demand a strict contract. I was only discussing that with a colleague earlier this week!
Ed: This is exactly what Spec is designed for; qualifying only the data that we care about.
Paul: With Clojure on the JVM we have rock solid reliability and code that is easy to understand.
Training
Jon: How you found training?
Paul: There’s a learning curve, especially for those just with a background in Java or JavaScript, for example. Once the necessary skills are learnt though it does pay off nicely.
Ed: You get a lot with Clojure. You have a bag of solutions and it’s your job to cross the gap of knowing how to apply those tools and to apply the learning.
Last things
Jon: Any last words?
Ed: The company is always looking out for talent, wanting to recruit good people.
Paul: I’ve found the Clojurians Slack invaluable for helping me out and sharing the Clojure philosophy. Having easy access to people like Alex Miller is really helpful!
Ed: There is (def shef), a monthly Sheffield functional programming meetup. Clojure has featured a number of times.
