Hot on the heels of our last Crux
dev diary we have
facilitating users to quickly get going without committing to a Kafka
deployment model. As wrote about in the diary, the JDBC form of Crux
is aimed at that sweet spot, where users will already likely have a
relational data-store setup in place, and the volumes of data may not be
high enough to warrant the up-front Kafka investment.
This said, we’re still loving Kafka - Confluent’s managed cloud solution is worth considering as a good stepping stone, or even an end solution.
Kafka gives you a more authentic separation of concerns that you need with the database inside out approach, to handle vast amounts of data at scale. It is much more than just a message bus used for systems integration - it can be a golden store record of truth that is highly available and easily replayable.
To that end, the Crux team has been cooking up a Crux Kafka Connector for ease of plugging Crux into an existing Kafka installation and for data to be streamed in to produce subsequent bitemporal graph queries. More details to follow.
It’s been great to see so many people playing around with Crux and the odd case-study emerging.
Crux is unbundled and open source, and so we always hoped that people would start building out their own modules and different ways of using it. So far, we’ve seen Crux built to use Active Objects as the event-log (mirroring the recent JDBC effort), Xodus as the node-local K/V store, and there have been experiments such as running on top of AWS SQS and using Pathom/Fulcro. There have been many other conversations about what underlying tools and platforms could support a running Crux setup and so hopefully more community contributions will continue to blossom out.
To help with this - as part of the most recent release - we have changed the API and have introduced the concept of node topologies so that Crux is more easily configurable.
Before, the api had functions such as
start-jdbc-node, but now there’s a
start-node entry function that takes an options map, one of
those options being
crux.node/topology. Officially supported
topologies are currently:
A topology specifies a map of modules that are needed to construct a topology. A module can now specify what options it takes, the default values, validation functions, and parse functions, if Crux is to be run from the CLI. Configuration options can also be used to override modules within a topology.
Our hope is that this configuration approach makes it easier to extend Crux and to play about with new variants of Crux topologies, embracing the unbundled approach.
Below is an example of a properties file for a Kafka node:
crux.node/topology = crux.kafka/topology crux.node/kv-store = crux.kv.rocksdb/kv crux.standalone/event-log-dir = data/eventlog crux.kv/db-dir = data/db-dir crux.kafka/bootstrap-servers = localhost:9092 crux.kafka/tx-topic = crux-transaction-log crux.kafka/doc-topic = crux-docs crux.kafka/create-topics = true crux.kafka/doc-partitions = 6 crux.kafka/replication-factor = 3
We’ve gone on tour to talk about Crux. Håkan gave a talk at ClojuTRE on the Design and Implementaton of a Bitemporal Database. The last time he had visited Helsinki was for Assembly94.
Johanna gave a Crux tutorial at ClojuTRE, gracefully added to the schedule by our friends at Metosin.
Jeremy and I talked about Event Streaming Architectures and Temporal Databases at Strange Loop. Jeremy also ran an unsession that was well attended.
The JUXT team have run training workshops at Heart of Clojure and we gave a talk to our friends at 'Scala in the City' hosted by Revolut at Canary Wharf.
It’s been a fun journey since launch, but our thoughts now are turning to the medium and long term future.
We are growing the Crux team with some exciting full-time additions and we are creating a new London based development center, that will be where future versions of Crux core will be built from. When we formally start the BETA program in the next 2-3 months, we will do so with a global support capability including North America, Europe and Australia.
We have been looking long and hard at the roadmap and we’ve got exciting plans for Crux and extending its range of functionality.
First though we will be entering a formal BETA program and we will inviting partners to join. As part of the BETA we want to see and understand how Crux is performing in more production installations and of what improvements we will need to make.
Thanks for reading this journal. Come along and say hi: