Open Time Store™
Copyright © JUXT LTD 2018-2019

Nodes

To start a Crux node, use the Java API or the Clojure crux.api.

There are different configurations of a Crux node:

Table 1. Crux Nodes
Name Transaction Log Java API

Cluster

Uses Kafka

Crux.startClusterNode options;

JDBC

Uses JDBC event log

Crux.startJDBCNode options;

Standalone

Uses local event log

Crux.startStandaloneNode options;

Use a cluster node when horizontal scalability is required or when you want the guarantees that Kafka offers in terms of resiliency, availability, and retention of data.

Multiple cluster nodes participate in a cluster using Kafka as the primary store and as the central means of coordination.

The JDBC node is for when you don’t want the overhead of maintaining a Kafka cluster. Read more about the motivations of this setup here.

The Standalone node is where a single Crux instance has everything it needs locally. This is good for experimenting with Crux and for small to medium sized deployments, where running a single instance is permissible.

Crux nodes implement the ICruxAPI interface and are the starting point for making use of Crux. Nodes also implement java.io.Closeable and can therefore be lifecycle managed.

Note
To see an example of a Crux Node being integrated into a Clojure lifecycle management framework such as Integrant, see the example Integrant node.

Properties

The follow properties are required for any type of Crux Node:

Table 2. Nodes Configuration
Component Property Description

kv-store

kv-backend

Sets the K/V Store to use, e.g. "crux.kv.rocksdb.RocksKv"

kv-store

db-dir

Sets the data-directory to use, e.g. "data/db-dir-1"

tx-log

bootstrap-servers

Kafka Servers to connect to e.g. "localhost:9092"

http-server

server-port

Port for Crux HTTP Server e.g. 8080

Standalone Node

Using a Crux standalone node is the best way to get started. Once you’ve started a standalone Crux instance as described below, you can then follow the getting started example.

Local Standalone Mode
Table 3. Standalone Configuration
Component Property Description

tx-log

event-log-dir

where to store events locally, i.e. "data/eventlog-1"

Project Dependency

  juxt/crux-core {:mvn/version "19.09-1.4.0-alpha"}

Start the standalone node:

(require '[crux.api :as crux])
(import (crux.api ICruxAPI))

(def ^crux.api.ICruxAPI node
  (crux/start-standalone-node {:kv-backend "crux.kv.memdb.MemKv"
                               :db-dir "data/db-dir-1"
                               :event-log-dir "data/eventlog-1"}))

You can later stop the node if you wish:

(.close node)

Kafka Nodes

When using Crux at scale it is recommended to use multiple Crux nodes connected via a Kafka cluster.

Local Cluster Mode

The following properties are required for a Kafka Cluster Node:

Table 4. Cluster Node Configuration
Component Property Description

tx-log

bootstrap-servers

Kafka Servers to connect to e.g. "localhost:9092"

Project Dependencies

  juxt/crux-core {:mvn/version "19.09-1.4.0-alpha"}
  juxt/crux-kafka {:mvn/version "19.09-1.4.0-alpha"}

Use the API to start a cluster node, configuring it with the bootstrap-servers property in order to connect to Kafka:

(def ^crux.api.ICruxAPI node
  (crux/start-cluster-node {:kv-backend "crux.kv.memdb.MemKv"
                            :bootstrap-servers "localhost:29092"}))
Note
If you don’t specify kv-backend then by default the cluster node will use RocksDB. You will need to add RocksDB to your list of project dependencies.

You can later stop the node if you wish:

(.close node)

JDBC Nodes

JDBC Nodes use next.jdbc internally and pass through the relevant configuration options that you can find here.

Below is the minimal configuration you will need:

Table 5. JDBC Configuration
Property Description

dbtype

One of: postgresql, oracle, mysql, h2, sqlite

dbname

Database Name

host

Database Host

user

Database Username

password

Database Password

db-dir

For h2 and sqlite

Add crux-jdbc to your project dependencies:

  juxt/crux-jdbc {:mvn/version "19.09-1.4.0-alpha"}

Use the API to start a JDBC node, configuring it with the required parameters:

(def ^crux.api.ICruxAPI node
  (crux/start-jdbc-node {:dbtype "postgresql"
                         :dbname "cruxdb"
                         :host "<host>"
                         :user "<user>"
                         :password "<password>"}))

RocksDB

Project Dependency

Add RocksDB as a project dependency:

  juxt/crux-rocksdb {:mvn/version "19.09-1.4.0-alpha"}

Configuration

Set the following properties when configuring RocksDB in Crux:

Table 6. Rocks Configuration
Property Value

kv-backend

"crux.kv.rocksdb.RocksKv"

db-dir

i.e. "data/db-dir-1"

For example when constructing the standalone node:

(def ^crux.api.ICruxAPI node
  (crux/start-standalone-node {:kv-backend "crux.kv.rocksdb.RocksKv"
                                 :db-dir "data/db-dir-1"
                                 :event-log-dir "data/eventlog-1"}))

Http

Crux can be used programmatically as a library, but Crux also ships with an embedded HTTP server, that allows clients to use the API remotely via REST.

Remote Cluster Mode

Set the server-port configuration property on a Crux node to expose a HTTP port that will accept REST requests:

Table 7. HTTP Nodes Configuration
Component Property Description

http-server

server-port

Port for Crux HTTP Server e.g. 8080

Visit the guide on using the REST api for examples of how to interact with Crux over HTTP.

Docker

If you want to experiment with Crux using a demo Docker container from Docker Hub (no JVM/JDK/Clojure install required, only Docker!) then please see the standalone web service example. You can also use this self-contained demonstration image to experiment with the REST API.

Backup and Restore

Crux provides utility APIs for local backup and restore when you are using the standalone mode. For an example of usage, see the standalone web service example.

An additional example of backup and restore is provided that only applies to a stopped standalone node here.

In a clustered deployment, only Kafka’s official backup and restore functionality should be relied on to provide safe durability. The standalone mode’s backup and restore operations can instead be used for creating operational snapshots of a node’s indexes for scaling purposes.

Extras

Embedded Kafka

Crux is ready to work with an embedded Kafka for when you don’t have a indepently running Kafka available to connect to (such as during development).

Add crux-kafka-embedded to your project dependencies:

  juxt/crux-kafka-embedded {:mvn/version "19.09-1.4.0-alpha"}

See the below example code:

(require '[crux.kafka.embedded :as ek])

(def storage-dir "dev-storage")
(def embedded-kafka-options
  {:crux.kafka.embedded/zookeeper-data-dir (str storage-dir "/zookeeper")
   :crux.kafka.embedded/kafka-log-dir (str storage-dir "/kafka-log")
   :crux.kafka.embedded/kafka-port 9092})

(def embedded-kafka (ek/start-embedded-kafka embedded-kafka-options))

You can later stop the Embedded Kafka if you wish:

(.close embedded-kafka)