Managing a Fanbase

BandSquare is a general purpose marketing platform helping artists and big organizations to manage and extract data from their audiences and fanbases. The platform uses campaigns, social network data, smaller tailored websites and polls to provide a audience platform that can pull and push data. An example use case is fanbases requesting that artists visit certain cities.

Once data has been collected it is then amalgamated to provide a unified view of an audience.

BandSquare has Clojure and Datomic at the heart of the system. I caught up with Val Waeselynck to find out more.

Technology Story

Jon: What is the tech story here?

Val: We used to be an Angular.js SPA (Single Page Application) backed by Node.js and MongoDb.

As the information model evolved and became more sophisticated, the complexity grew. MongoDB in particular became a big pain point as the querying power is very limited. We had lots of bugs and troubles evolving the platform.

We wanted it to be a data platform and we were stuck. We wanted features, to go faster, and to have better quality. We could not add man power so we needed a technical shift to change the dynamics.

Jon: How did the Clojure journey begin?

Val: At the time I knew Clojure. I had very little experience with Datomic; I’d played with it a bit.

The value proposition of Clojure is a good fit. I could use the interactive development story of the REPL to raise productivity and quality. Datomic was a good fit too that didn’t have the problems of classic databases.

Datomic is very good for querying for data in a very expressive way. It’s also great for testing; we wanted a really good testing story to increase quality. We were migrating the whole back-end from Node.js and we wanted to test that everything still worked.

Jon: Any regrets?

Val: No. The migration to Clojure/Datomic made us so much more productive and we have less bugs.

Mongo/Angular is not bad tech, just a not good fit for the level of sophistication we needed. We still use Angular on the front end.

Jon: What about using ClojureScript on the front end?

Val: I wish I could - an architecture based on React.js would be good as well - but not right now.

Datomic

Jon: Any challenges with Datomic?

Val: I wouldn’t advise a team to use it if they don’t have the architectural initiative to embrace a new paradigm. You have to figure out how to architect your system to get the most leverage, but this can be really rewarding.

For example we rely on a lot on a special feature called speculative writes (aka db.with()) where you can pretend that a write is made locally. This has enabled us to have an architecture where we can rewrite code and perform testing using this ability. We can fork databases and experiment with no runtime cost.

You can also mock and clone connections to debug locally data that is in production; this gives amazing leverage and opportunities. This stuff is extremely powerful, but the challenge is you have to model your architecture to leverage it.

Tech Stack

Jon: What does your whole stack look like?

Val: We use AWS, EBS. PostgreSQL, S3, Redis for caching, Google Analytics, Datomic on DynamoDB, and Netlify which is good for hosting SPAs.

Jon: What Clojure libraries do you use?

Val: I’m a big fan of Specter - I don’t use nearly as much of it as I would like.

The data manipulation capabilities of Datomic are almost better than Specter; it’s also a library that has a Datalog querying engine.

We also use DataScript; we store a representation of the data model in DataScript, like an annotated UML diagram. We derive the Datomic schema from data in DataScript.

Jon: Wow, sounds like an interesting approach. What’s the motivation?

Val: The domain model was duplicated in code in various places: in the Datomic schema, in the Plumatic Schema validation area, in data transformation code (Datomic to JSON). We decided to have one source of truth that removes the duplication; a single declarative place where the information model is described and that different representations of the data model can be derived from. Some people are opposed to this approach, but it’s a different philosophy.

Jon: What else do you use?

Plumatic Plumbing, especially the Graph API, which lets you express complex computations as graphs of simple steps. Each computational step is small and depends on previous steps in the graph. It’s very good for code reuse of the small, expressive steps.

Jon: How much code do you have?

Val: About 26K lines outside tests, 35K tests including. Our platform does a lot; information extraction, talking to other services and integrating data etc.

Jon: How do you feel about the state of your current codebase?

Val: The codebase is clean, it’s just big because it has lots of functionality.

One of the biggest changes we could do is to move to a GraphQL like architecture, such as using Om Next or Falcor. This would be a better way of exchanging data and removing the accidental complexity of lots of REST endpoints exchanging JSON in different formats.

Paris?

Jon: What’s the state of Clojure adoption in Paris?

Val: A few companies are using it and beyond that there are many more people interested. You either find it in the departments of big companies, or the big startups.

Jon: Do you see it growing?

Val: Definitely; more and more people are interested. CTOs are reluctant right now to change, but the culture itself is shifting.

Resources

See Val’s Blog, and in particular this post where he discusses Datomic techniques such as speculative writes.