Gresham is a financial technology business, with headquarters in London and offices across the UK, North America and Asia. In 2019, Gresham announced a partnership with a major bank to deliver a new dynamic cash management solution and banking platform.
We caught up with Colin Reid (Development Manager) and Mark Tinsley (Principal Engineer) at Gresham, in their delightful office in Bristol’s Brew House, to discuss why Clojure, Kafka, and event-sourcing are key ingredients of their new product.
Joe: So what does Gresham do?
Colin: We help financial firms overcome common data, reconciliation, connectivity, and reporting challenges. That’s the core business. We help them be compliant with regulations by implementing things like the reconciliation process across many different systems, and the regulatory reporting processes. We give these institutions control over their data.
In banks there’s often a sprawl of different systems with different messaging formats, and none of it is cross-referenced. Reconciliation is really important here. Originally it was all on-premise, but we have cloud services now too.
Another major offering is our new cash management solution. Existing banks will typically have some slow, legacy core banking systems and in corporate banking particularly banks have been slow to re-new and upgrade – largely because of the complexity of existing platforms and their customer requirements. With our solution banks can keep core banking logic and accounts in those existing systems while quickly building a faster, more flexible corporate banking platform. The solution enables our customers to deliver account management solutions across cash and digital currencies, embedded bank accounts, escrow and regulated monies – there are many use cases. The key point is the bank can focus on their specific priorities without the major risk of decommissioning legacy core systems until they are ready
Joe: You have quite a few different product streams. What tech stacks are being used?
Colin: Traditionally it has been Java, Spring. We have used Gigaspaces in the past for smart, distributed caching. We also have some products on the .Net/Microsoft stack that came from an acquisition, but we’re definitely aligning towards the JVM. We use a lot of Kafka, and we’re also seeing most of our products gradually aligning on Kafka. We use a lot of AWS for our own cloud services. Many banking clients are still not ready to put their data into the public cloud, so we’ll support them running our systems on their own on-premise or private cloud environments.
Clojure
Mark: Our new cash management solution runs on GCP and this is where we use Clojure. All our backend services for the cash management solution are written in Clojure. The frontend is TypeScript and Javascript. We have a team in Sydney using Java and Spring Integration to talk directly to the bank’s services, and when their services talk to us they use Kafka.
We’ve also started using Babashka to replace Bash for any scripts we need. It’s really nice, as work can be shared much more easily around the team.
Joe: Yes, having a team that can use a single language for everything can be really powerful. You mentioned you’re using Clojure on the backend, what form does this take?
Mark: Our backend is made up of many microservices. These microservices are deployed to Kubernetes, and the services talk to each other using Kafka. Each service uses Kafka Streams internally. We tend to build a single container with many Kafka Streams topologies within it. We also have a service that exposes a GraphQL API.
Joe: How did you come to select Clojure? Was this the first time you had used Clojure on a project?
Colin: That’s a long time ago now! Our lead architect. Both him and I had quite a strong background in functional programming. We’d done some functional programming during our degrees many years earlier but never really had a chance to adopt it. We worked up slowly, watched a lot of Rich Hickey’s videos.
We’d reached a general point of dissatisfaction with how large and clumsy Spring was becoming. Everything was annotation-based, and just trying to navigate a codebase was becoming really quite painful
. The amount of code we had to have for the amount of functionality we needed just seemed out of proportion.
We spent a couple of years just watching, listening, and seeing what was happening in that space, before we really bit the bullet and decided, “Yeah, we should go for it. It is viable, we could approach this next project using Clojure”.
The other thing we did was speak with existing developers about how we should staff the project, and who would be curious enough that they wanted to get involved. This was great as we had a good take-up from people who were keen to adopt and try something different. There was quite a lot of risk analysis up front actually - How are we going to staff it? What are the risks?
So we had initially started talking/thinking about using Clojure around 2014/15. We felt it was easier to start using Clojure on a ‘green field’ project rather than try to integrate it into something that was still very much a monolith on some of the other projects.
We had quite a few conversations with JUXT at this stage too, we ran some initial days of training and got some review of the project and the architecture. That gave us a bit of confidence to actually get started, know that we were on the right track, and think “How do we move forward now?”. That was 2018.
Joe: Were there any particular reasons that you felt this project was a good fit for Clojure?
Colin: Actually, yes. Firstly, the fact that everything is immutable. We wanted to know that once we had made a decision on some data, that it was ‘truthful’, then we weren’t going to have other parts of the system changing it randomly. Consistency was really important. We were also attracted to something that was simple, we were keen on small, composable functions that were really testable.
Being able to rationalise on something, because when using Clojure the amount of code you end up with is less. That’s something I’m really quite pleased about.
If you compare the amount of code we had a few years ago with what I saw when I looked a few weeks ago, I think we’ve got the same amount of code now as we did then! We’ve got much more functionality, but I don’t think the net amount of code has actually changed.
Joe: It just gets distilled and refined!
Colin: Yes, which I think is fantastic. Also, I think we have around three times the amount of test code as we do actual business logic, which I think is a good ratio.
Most of our code is test code. This has turned out well.
Event-driven Banking
Joe: So one of the other key characteristics of the cash management solution is that it’s a fully event-driven system, and event-sourcing is a key pattern of the architecture. Why did you adopt that approach?
Mark: With finance systems you need to understand how you got to the position that you’re currently in. With Kafka you can save all events, so you have an audit history to the beginning of time if you want it. So Kafka lends itself well to finance. With a topic you’re saying, “These are the facts that have happened”, and if your program code is ever wrong you can correct it and re-consume those facts to make up a different ‘model’ or ‘current view’ of what the world should look like.
The event-sourcing model with Kafka gives you audibility, and this feeds in really well to things like reconciliation. Being able to prove where you have actually put money
. We may have one account, but when the customer looks at it they might want to see that account as many different ledgers. We need to prove where the money has gone and what has happened. We have this power from the events.
Each message represents a fact about something that has happened. Re-reading these facts allows you to rebuild your internal model. This means that you can re-consume all the messages (facts) about the system and generate a new internal model. This is very different to how SQL databases operate, in SQL you mutate the current system state to the new state, in the process you lose the previous state (say you drop a column) and are limited to what the new system state can be, as this might involve complex logic over and above SQL update operations.
Joe: So provenance - being able to trace back exactly how any value, like a balance, was arrived at - is very important for you.
Colin: It’s quite complementary to our approach because we wanted to make sure that everything was double-entry. Every transaction we did was with a double-entry accounting principle. This fits really nicely with event sourcing as you don’t ever rewrite history (it’s happened, you’ve got the record of it).
Mark: You can use a single event log and process it in different ways to arrive at a different model for use. If you only care about the balance, or only about credits or debits, you can process the same event log in different ways.
Clojure Ecosystem
Joe: Which parts of the Clojure ecosystem are you using?
Mark: We’re using Jackdaw for our Kafka Streams topologies. We actually maintain our own wrapper for Jackdaw internally because we’ve had to extend some things. Jackdaw hasn’t had a lot of updates recently, although now there are some updates starting to come through and PRs are getting merged. We’ve introduced our own concepts that don’t exist natively in Kafka Streams, such as the eventual join, and we’ve added these to our own internal wrapper. We’re debating now whether to remove Jackdaw entirely and interact more directly with the Streams API, and get greater control over upgrades. For now though, Jackdaw remains our primary way of defining these topologies.
We moved away from Java logging frameworks to Timbre logging. We’ve extended our logging quite a bit to improve exception handling - we walk through the chain of exception and gather exception-info and the associated data. You get a single, consolidated map that you can understand - one log that has all the information you need. JSON logs work well with the cloud providers.
We used to use Spec to validate all our events, but we’ve now switched to Avro and we’re using the Kafka Schema Registry. It makes it a lot easier for us to share schemas with the Java team that we integrate with, and they can consume Avro without having to learn about EDN or Nippy. Avro gives us a schema in one place for documentation. Spec is also quite loose (for instance, you can send a map that has extra keys, and someone might become dependent on it. There are some libraries that can work around this, but Avro lets us say “No, these topics matter” and everyone can guarantee that the data on them will be exactly as expected. We generate our Avro schemas using abracad, and the team that we integrate with can then consume the schemas directly as they need to generate POJOs.
All of our services use Integrant and Aero. This has been there since the beginning and it makes it very easy for us to create a test config and set up the whole service locally
.
We use Ring and compojure-api for REST APIs. We also use Lacinia for our GraphQL API.
Joe: How about dependency management?
Mark: We use lein on our projects. We have a monorepo, and we use lein-parent to control dependency versions in a parent project.clj. In your project you can pull in a library but the version management is done at the top level. This is easier for us and works well with the monorepo style.
We solved some interesting problems with our project build because we don’t want to send the whole monorepo to the Docker daemon. We ended up creating a Docker build where we add all the dependencies, then use that as our base image when building the individual service.
[Check out lein-metajar - Ed]
Side-effects
Joe: How do you deal with side-effects in your system? What’s your preferred way to represent this in your code?
Mark: My preference is to try to push those side-effecting parts as far to the edges as possible. You have as much of the system as possible represented as pure functions, then you have a namespace (typically called ‘core’) where you stitch together the pure and side-effecting parts. Pushing those side-effecting parts out to the edges makes things easier to test.
That’s how I like a Clojure project to be. That’s not to say ours is like that, but I prefer that approach.
On the cash management solution it’s slightly different. Each of the things that Integrant controls wraps around a Kafka Streams topology. So that’s our unit of test, the topology.
In some areas we can pull out enough parts to usefully test something in isolation. The problem with this kind of test is that when events change, or schemas change, the test won’t break because you have fixed whatever data you are passing in. For this reason the value of this kind of test is reduced. That’s why we need those overall tests, that test the topology, and if the schema changes then these will break. I know that if something about the schema changes, those tests will break, and those are the tests that really matter.
Obviously there’s a trade-off between these two types of test, and it depends how much things are changing.
Challenges
Joe: So the majority of the backend system is implemented as Kafka Streams topologies. What are some of the challenges you face?
Mark: Streams is a framework and you have to fit in with it. The powers of testing that you currently have are not great. The standard tools don’t deal with all the scenarios that we care about. We want to be able to test more extensively how Streams is working with internal topics, if there are multiple partitions then have tests where things are consumed out of order. One of our engineers has done a lot of work to extend the TopologyTestDriver, and we’ve spoken a lot with Confluent about this (see DanielStephens/ktest).
With Streams when you work with the TopologyTestDriver it’s a very different way of testing. You say, “I’ve consumed these messages, so what do I get out” but with Clojure we always like to execute a pure function and what is the result? It’s similar, but the volume of quite complex code you’re going through is large. You have to pass in specific Java types, like a KStream, which needs to be set up. Your event has to be well structured, as we’re using Avro. All this can mean the test becomes quite large. Ideally we’d like to be able to test our business logic and ignore the framework, but there’s no good support for this right now.
Joe: Ideally you need the TopologyTestDriver to explore all possible edge cases, for instance, to test different possible orderings, given the steps defined in your topology.
Mark: Yes, exactly. Also, Kafka is our source of truth, for the whole system. We don’t have a database that we write to. We often query the Streams stores directly, but the TestDriver assumes a single partition and that you will have all data in the local store. In reality, you may not. We use a lot of Java interop to query the stores but this is mostly hidden away in our own libraries.
Joe: How do you package and deploy your Clojure applications?
Mark: So we deploy as Docker images, to Kubernetes. We can actually deploy the whole of the cash management platform locally using minikube. We use CircleCI and as part of our release we build jars, create Docker images, update YAML files using Kustomize and then produce a final, versioned YAML file. We then use a Babashka script to deploy this to an environment.
Each of our microservices is its own uberjar, and we can have many Kafka Streams topologies started by a single microservice. We find this helps us to reduce the resources that we use. There’s nothing shared between the topologies though, they have their own separate stores, and they only communicate via Kafka topics.
Joe: What challenges have you had in using Clojure for the cash management platform?
Colin: I think getting developers is quite interesting. It can be hard to find developers, but I don’t know if this is the language or whether it’s just that finding good developers is hard. You might advertise for a Clojure Developer, but are you going to find that person? We’ve had to be quite flexible on that. We really look for attitude and aptitude, curiosity and passion.
Mark: There’s also tooling around Java for things like static analysis. It just isn’t quite there for Clojure yet.
Colin: It’s the nature of finance and the industry we’re in that we need to take security very seriously. The nature of Clojure means there’s not a lot of static analysis tooling out there, for instance. So we look to close this gap in other ways, with some other mitigations. We’re using IAST [interactive application security testing] tooling instead, so this is interaction analysis using a tool called Seeker. It doesn’t understand the Clojure but we get stack traces and it shows us the flow of data through the system. We also do the normal NVD checks. With these analysis tools you’re often looking for something that does Java, and so may be useful for Clojure even if it doesn’t give you the same insight.
[Agree. Useful tools in this space include clj-kondo, kibit, eastwood, carve, yagni, bikeshed, clj-holmes, clj-watson, nvd-clojure - Ed]
Joe: Yes, sometimes it’s not enough to explain why certain kinds of vulnerabilities are not possible. For compliance reasons you need to produce a report that’s the result of a scan.
Onboarding
Joe: You mentioned hiring for attitude and aptitude. How do you bring engineers that don’t have Clojure experience into the project?
Mark: If it’s graduates, they’re going to need help on a lot of things. There’s Clojure, but also Kafka, and the world of distributed computing, which is really complicated. So even some of the Clojure developers we’ve hired, if someone doesn’t have experience with Kafka it’s going to take them a while to understand those things.
Each individual is different really, we haven’t got a set programme for exactly how we’ll train people. Some like to work with someone and some like to work more on their own. This is fine, and we try to start people on smaller tasks. This can be hard as you quickly run out of small tasks. I really try to make sure that I’m not just talking at people all day, as this is really exhausting for them. You can see it in their eyes, when they’re starting to glaze over! I really like to get people to take a project and just play with it, try to make things fail, come back to me when things don’t make sense.
One of our engineers, Dan Stephens, has done amazing work on our READMEs. He’s laid out exactly how to get started on your first day working with the system, how the code is laid out, how you can set up your machine, how you can get minikube working, if you want to. It can be a lot of work but really useful when we get a few graduates in at one time.
Colin: Every new person will go through that and they’ll fix issues for you. If things have moved forward or been pulled out as a library and we’ve not updated the README, they will catch that. There’s always something that’s moved on.
We’ll usually have a task in mind for new joiners. They’ll be working in the team, but we’ll have a task carved out for them, fairly standalone, that gives them exposure to a narrow part of the system. The actual change might be really simple, even just a one-line change, but going through the process of setting everything up and getting a working local environment is really useful. It gives people confidence in the first week.
Mark: I always try to say to any developer joining, “Get a PR up fast”. Quick feedback is the key thing. PRs are helping me understand what you understand - we’re sharing this idea. I want to see how you approach things and discuss. Rather than wait a week and then think, “Well, we could have iterated on this a bit”. It’s always good to get things out fast and get feedback fast.
We have a daily stand-up as well, so we can use that to check that people are on track.
Joe: If new starters are new to Clojure, they will need to get more comfortable with the language itself and the core APIs. Is there anything you commonly use there?
Colin: We tend to give a couple of books to each new starter. Clojure for the Brave and True and Clojure Applied. It depends what our current favorites are. We also find 4Clojure exercises are useful. We buddy people up, so someone would do an exercise and then discuss with their buddy. The buddy might say, “Well, actually you could do it this way, or this way” and exploring these different approaches and alternatives is very useful.
We also have a lot of special-interest channels in Slack, not just for Clojure actually. People will post interesting articles there. We have channels for Kafka, Kubernetes, Clojure. There’s not a lot of traffic, but occasionally somebody will find a good article, or watch an interesting talk, or find a good reference. The Kafka one is pretty noisy.
Pre-pandemic we used to have weekly open learning sessions where we’d pick an interesting presentation from the last month or two and play it in our stand-up room, then chat about it afterwards.
Mark: In the past we’ve also had regular knowledge sharing sessions. It depends on where people are, so when you have lots of new starters you regularly have people saying, “I don’t understand that”. In our team right now we have a new starter so we’re starting to run these sessions again.
Future
Joe: Where would you like to see Clojure go next? What would you like to see in the future, that would make you keep choosing Clojure for new projects and use Clojure more widely at Gresham?
Mark: I’m really happy with Clojure. Most of my gripes are with Streams and Kafka! I spend a lot of my time concentrating on those things. The solution for how to upgrade from one Streams version to the next is not a solved problem, and we’ve had to do a lot of work around that (and continue to work on that).
Things like Babashka are helping a lot. Previously there would be one or two people that know Bash really well, and they write all the scripts. It’s quite isolating though. Now with Babashka, more people are doing it. Having Clojure running in every possible environment helps so much.
Joe: So making Clojure that universal language for teams, so that there’s good Clojure support in whatever environment they need?
Mark: Yes exactly. I doubt we will be using ClojureScript as we have a team of JavaScript engineers that are happy with the tools they use.
Colin: It would be great to have more developers available, so wider adoption and popularity would always help. It’s always hard to apply a new language to an existing product, but for a new greenfield project I’d be keen to use Clojure. It also does depend on the available developers in our team, as a lot of developers are quite passionate about the one language they love.
Mark: We’ve had engineers across into the Clojure team, who had no previous experience but had an appetite to learn. More cross-pollination between different teams would be a good way for us to drive adoption.
Editors
Joe: What IDEs or editors do you use in the team?
Mark: People use whatever they prefer. I use Spacemacs, others use plain Emacs, Intellij is used a lot and is very, very good.
Colin: Yes, quite a few using Cursive, and then we have some using VS Code with Calva. I don’t write any code but that’s my preferred choice.
It’s funny because you think the editor thing is really a solved problem, then VS Code has come along and is huge. It’s just a good product.
Mark: If you’ve customised your key bindings it can get hard to pair with others or use their machine, you realize that you’re stuck without your own setup. What’s changed now though is that people are remote, so you can do the ping-pong pairing thing and say, “Okay you drive for a while” and there’s no need to use someone else’s setup. Sometimes you’ll write a test and push it to a branch, and the other person can pull and make it pass.
REPL-Driven Development
Joe: Do you tend to work with the REPL a lot when working on Gresham projects? Is it a REPL-driven development experience?
Mark: Yes. If you’re working directly with Streams topologies and Kafka interop, you can get Kafka running very easily. Straight away, in the REPL, you can start Kafka (Great!) then I’ll create an admin client to see what’s there, see how to check offsets, etc. The key thing with the REPL is quick feedback, that’s what I want it for.
I tend to write a test first, but still, with the REPL you’re working out the shape, structure, and how you want to do it. You get very quick feedback to see, “Oh, that’s not going to work. We need to rethink this”. This is something to learn as part of the Clojure adoption process.
Colin: Yes, this is possibly a bigger challenge than learning the language itself, perhaps.
Joe: Yes, you almost need to continually make sure that you’re not introducing barriers to that quick REPL feedback. It’s very easy for things to creep in, say a test that you need to stop the world to run. You have to have a mindset of, “We need to maintain this REPL experience”.
Mark: Yes, definitely. We’ve had those things in the past, where there have been some fixtures or tests setup that must always be executed in a certain way. When we see that we always tend to break it apart, make a comment block up if you need to with parts you can run easily in the REPL.
If you break functions up so that they just do one very small thing, you can run that directly in the REPL. Make sure you have ways to switch things on and off, bring Kafka up and down.
Colin: Yes, that whole thing of having to keep on top of it is true. We had a PR just recently where there was a need to always run this two step process to exercise the code. After some feedback, we got this back to just a good REPL experience.
