Understanding an Organisation’s Event Driven Architecture
During the discovery or design of an Event Driven Architecture (EDA), it’s important to understand what EDA means to those involved. In the introduction, we alluded to the large scope of EDA being a socio-technical problem to overcome. Within a single large organisation, the term EDA can mean significantly different things - knowing which aspect of EDA is in play will help all involved.
Towards an Informal Taxonomy
We will use the below diagram to explore the various meanings of EDA. Ideally, you will be able to place your own position on the map. Hopefully, it will reveal, to the reader, the trade-offs made in comparison to other locations.
Underpinnings
Message-Driven Architecture
Whilst it is not technically necessary to use a message-driven architecture when building an EDA, in practice the majority of projects do.
A message-driven architecture decouples systems in both time and location: producers of messages need not know when and where a consumer processes a message [3].
Common technologies in this space include Kafka, SQS, Kinesis, and Pub/Sub. They provide other desirable features such as scalability, delivery semantics, and ordering guarantees.
If an EDA is centered around transferring events between systems it should come as no surprise a message-driven architecture is often reached for.
Event-stores
Event Stores refer to dedicated event-store implementations/frameworks that could exist independently from a particular project. Event-store Infrastructure can be used both independently and in conjunction with Message Driven Architecture. Oftentimes an event-store stack will also provide event-routing.
When using event-sourcing as a state persistence pattern for a DDD aggregate, an event-store is the repository to which these events are stored. However, some projects will not have evolved from a DDD foundation and the thinking is not in terms of aggregates but event streams.
Common technologies in this space include EventStoreDB, Axon, and MartenDB. They tend to include desirable features like strong consistency on append, optimistic concurrency, and global ordering. When Event-store Infrastructure is not used for Event Sourcing then these features are often replicated by developers themselves using bespoke code and the features of Message Driven Architecture implementation[14].
Design
Event Driven Architecture
The common denominator throughout Event Driven Architecture is the “event”. From Wikipedia’s entry on EDA [16][17][18]:
An event can be defined as “a significant change in state”
and
Events do not travel, they just occur. However, the term event is often used metonymically to denote the notification message itself, which may lead to some confusion.
For the taxonomy we will say
An event is a notification of something that has happened in the past
The amount of context that is included in the notification can vary. It is rare to use the notification to prompt the discovery of the actual change. Typically, some form of event-carried-state-transfer is used.
Unfortunately, on inherited software projects, we don’t get to declare what an event is. Fowler notes the trap of using events passive-aggressively as commands. And this trap is not uncommon - expect cerebral contortions to justify a project’s ingrained thinking e.g. “action events” or “the issuance of command is itself an event”. There are many articles on Commands vs Events [20][21]. It is a judgment call to either, bend to a project’s culture, or try to formally separate the concepts.
Message Driven Architectures and Event-stores are used to implement EDA. It is possible to combine these stacks (see the overlapping centre) or not use them at all. The next blog post will explore Event Driven Architecture and Domain Driven Design without using an event-store or a message-bus.
There is an impression that at certain points in EDA’s history rolling your own solution was an acceptable and even encouraged thing to do - take care here, it may be appropriate but more likely isn’t.
Pragmatically, any system using events can be labelled an EDA.
Domain Driven Design
DDD is built around programming a Domain Model - wherein business logic and data are placed together in objects encapsulating the rules and processes of a (business) domain [18].
In DDD, a catalogue of software patterns helps the programmer and business in both building and working with complex domain models.
The key concept of an aggregate is introduced (the next blog post will explore this in more depth). An aggregate is a collection of entities and values. An entity is something identified by a name and has a life-cycle e.g. a loan application, a robot, or an order. Crucially an aggregate defines a transaction boundary - changes to the entities and values within can only be done via the aggregate. This constraint allows the programmer to implement the business logic required e.g. an order can not have more than one hundred line items or a robot can not shoot its owner.
Aggregates need to be persisted and this can be done in a few ways: perhaps directly to a database, via an ORM layer, or using domain events and event-sourcing.
Domain Events
In DDD a domain-event is
something happened that domain experts care about
We can contrast this to the definition of an Event we used in the EDA section.
An event is a notification of something that has happened in the past
Typically aggregates are responsible for generating domain events.
In a project is the set of domain-events equal to all events? This can be a grey area and having a shared understanding of a project here is helpful. It boils down to the scope of a “domain expert”. Let’s imagine that domain experts don’t include the programmers of the system. Let’s say a programmer changes an aggregate to include some metadata (say cache retention) that a domain expert doesn’t care about. If we are using the event-sourcing pattern to persist an aggregates state we need to use an event to capture the cache retention change. The event-stream is a mixture of domain-events and events.
It is arguably easier to work on a project in which domain-events fully describe the complete event stream of an aggregate - there is no conceptual context switching required. This may come at the expense of the “purity” of the domain model. Domain-event’s typically high-valued semantics may be lowered [1]. It may also blur the boundaries between what may be published downstream - the event stream may be too fine-grained to publish in full so filtering is required. This introduces the idea of integration events, we either selectively publish domain-events or create entirely new events from event combinations in the event-stream.
Again, what’s important here is realising what is meant for the particular project in hand [15] [18].
Top Level
Event Streaming
With event streaming, there’s a continuous stream of data flowing between systems, with the data representing a new state of events.
Continuous is an important word in this definition. There is no request/response semantic. Systems that process event streams have to be capable of continually updating and responding to a potentially never-ending source of data.
Naturally, a message-driven architecture is an ideal mechanism for streaming event data. Event-store infrastructure may be a source of an event stream. One need not use either of these mechanisms but cases of this on EDA projects tend to be rare.
Crucially, event streaming says nothing about the persistence, durability and retention of data. An event-stream can be viewed as a pipe responsible for accepting and delivering transient event data. These properties tend to be configurable within a message-driven architecture technology stack.
To some [2] [5] [8] [9], the distinction between an event-stream as described here and event-sourcing is very important. To others, and within projects, the boundaries can become blurred e.g. persistence may be achieved by attaching the event-stream to an S3 connector, or a Kafka topic may be set to have infinite retention.
Another view is that Event Streaming deals with “data in motion” and that Event Sourcing is a subset of Event Streaming as it is “data at rest” [6]. Event Streaming is said to add connectivity to Event Sourcing - this is an alternate way of saying “publish events”. Personally, the subset claim seems weak, there are properties of Event Sourcing that don’t naturally belong in Event Streaming. It is a complementary relationship rather than a subsuming one. On the ground, this specific contention makes little pragmatic difference to an engineer’s day-to-day.
Finally, in some DDD articles, the domain-events saved out to an event-store are referred to as an event-stream. And within the event-store infrastructure frame-work an aggregate may be referred to as an event-stream.
Event Sourcing
Event Sourcing ensures that all changes to application state are stored as a sequence of events. We guarantee that all changes to the domain are initiated by the events themselves. [19]
Sometimes a project can use it to mean an append-only audit-log. And often a project will implement the audit-log on an event-stream supplied in a message-driven architecture - lamented by some as these systems are not foremost event-stores. Thus require the bolting on of consistency code or edge case configurations to achieve the goals of event-sourcing.
See the Domain Events section for a discussion on the mixing of domain-events and event-sourcing [1] [2][4]
See the Event Streaming section for a discussion on the positions of Event Streaming and Event Sourcing [5][6]
JUXT’s Experience
Kafka is very prevalent in today’s enterprise. The below diagram highlights where Kafka Kstream apps sit.
There are ideas from DDD but often projects we encounter are not bootstrapped off of DDD concepts. Projections of aggregates are often in bespoke DB’s.
As you can see from our experience the classification is not perfect but hopefully, you can frame your projects in terms of one or more locations - and this is some help in getting everyone on the same page.
You can read Matt’s follow-up post “Simple, Tactical Domain Driven Design in Clojure” here.
References
- Domain Events vs Event Sourcing
- Event Driven Architectures vs Event Sourcing Patterns
- Reactive Manifesto
- Event Sourcing vs Change Data Capture
- Event Streaming vs Event Sourcing
- Event Sourcing vs Event Streaming
- Event Sourcing is Hard
- Event Streaming is NOT Event Sourcing
- This is NOT Event Sourcing
- The Many Meanings of Event Driven Architecture
- Domain Events in Event Sourcing
- Event Sourcing Explained
- What is Event Sourcing
- Kafka Not Suitable as an Event Store
- Domain Events
- Event Driven Architecture
- What do you mean by Event Driven
- Domain Driven Design
- Event Sourcing
- What’s the difference between a command and an event
- Event Sourcing vs Command Sourcing
