Open Time Store™
Copyright © JUXT LTD 2018-2019

Systems

To start a Crux system, use the Java API or the Clojure crux.api.

There are two main Crux systems: cluster node vs standalone. Use a cluster node system when horizontal scalability is required or when you want the guarantees that Kafka offers in terms of resiliency, availabilty, and retention of data.

Multiple instances of a cluster node systems may participate in a cluster using Kafka as the primary store and as the central means of coordination.

The Standalone system is where a single Crux instance has everything it needs locally. This is good for experimenting with Crux and for small to medium sized deployments, where running a single instance is permissable.

Table 1. Crux Systems
Name Transaction Log Java API

Cluster Node

Uses Kafka

Crux.startClusterNode options;

Standalone

Uses local event log

Crux.startStandaloneSystem options;

Crux systems implement the ICruxAPI interface and are the starting point for making use of Crux. Systems also implement java.io.Closeable and can therefore be lifecycle managed.

Note
To see an example of a Crux System being integrated into a Clojure lifecycle management framework such as Integrant, see the example Integrant system.

Properties

Crux Systems will use the following properties:

Table 2. Systems Configuration
System Component Property Description

Both

kv-store

kv-backend

Sets the K/V Store to use, e.g. "crux.kv.rocksdb.RocksKv"

Both

kv-store

db-dir

Sets the data-directory to use, e.g. "data/db-dir-1"

Standalone

tx-log

bootstrap-servers

Kafka Servers to connect to e.g. "localhost:9092"

Standalone

http-server

server-port

Port for Crux HTTP Server e.g. 8080

Standalone System

Using the Crux standalone system is the best way to get started. Once you’ve started a standalone Crux instance as described below, you can then follow the getting started example.

Project Dependency

To add Crux as a project dependency:

  juxt/crux {:mvn/version "19.04-1.0.3-alpha"}

Start the standalone system:

(require '[crux.api :as crux])
(import (crux.api ICruxAPI))

(def ^crux.api.ICruxAPI system
  (crux/start-standalone-system {:kv-backend "crux.kv.memdb.MemKv"
                                 :db-dir "data/db-dir-1"}))

For the purposes of this Hello World, we are using the simplest configuration of Crux, where all of the pluggable components are in-memory. There is no Kafka or RocksDB to worry about.

You can later stop the system if you wish:

(.close system)

Cluster Node System

When using Crux at scale it is recommended to use Crux as nodes connecting to a Kafka cluster.

Add Crux as a project dependency:

  juxt/crux {:mvn/version "19.04-1.0.3-alpha"}

And Kafka:

  org.apache.kafka/kafka_2.11 {:mvn/version "2.1.0"}

Use the API to start a cluster node system, configuring it with the bootstrap-servers property in order to connect to Kafka:

(def ^crux.api.ICruxAPI system
  (crux/start-cluster-node {:kv-backend "crux.kv.memdb.MemKv"
                           :bootstrap-servers "localhost:29092"}))
Note
If you don’t specify kv-backend then by default the cluster-node system will use RocksDB. You will need to add RocksDB to your list of project dependencies.

You can later stop the system if you wish:

(.close system)

RocksDB

Project Dependency

Add RocksDB as a project dependency:

  org.rocksdb/rocksdbjni {:mvn/version "5.17.2"}

Configuration

Set the following properties when configuring Crux systems:

Table 3. Rocks Configuration
Property Value

kv-backend

"crux.kv.rocksdb.RocksKv"

db-dir

i.e. "data/db-dir-1"

For example when constructing the standalone system:

(def ^crux.api.ICruxAPI system
  (crux/start-standalone-system {:kv-backend "crux.kv.rocksdb.RocksKv"
                                :db-dir "data/db-dir-1"}))

Http

Crux can be used programmatically as a library, but Crux also ships with an embedded HTTP server, that allows clients to use the API remotely via REST.

Set the server-port configuration property on a Crux system to expose a HTTP port that will accept REST requests.

Visit the guide on using the REST api for examples of how to interact with Crux over HTTP.

Extras

Embedded Kafka

Crux has an embedded Kafka that can be used.

Add Kafka and Zookeeper to your project dependencies:

  org.apache.kafka/kafka_2.11 {:mvn/version "2.1.0"}
  org.apache.zookeeper/zookeeper {:mvn/version "3.4.13"
                                  :exclusions [io.netty/netty
                                               jline
                                               org.apache.yetus/audience-annotations
                                               org.slf4j/slf4j-log4j12
                                               log4j]}

See the below example code:

(require '[crux.kafka.embedded :as ek])

(def storage-dir "dev-storage")
(def embedded-kafka-options
  {:crux.kafka.embedded/zookeeper-data-dir (str storage-dir "/zookeeper")
   :crux.kafka.embedded/kafka-log-dir (str storage-dir "/kafka-log")
   :crux.kafka.embedded/kafka-port 9092})

(def embedded-kafka (ek/start-embedded-kafka embedded-kafka-options))

You can later stop the Embedded Kafka if you wish:

(.close embedded-kafka)