Preface

State is everywhere. The world is moving and we need to keep up. We need our computers to help us stay up-to-date with the latest information, chats, trends, stock-prices, news and weather updates, and other important stuff.

The web is primarily a means to move state around. You have some state here, and you want it over there. Or it’s over there, but you want it over here.

For two decades or more, the pre-dominant model for web programming has ignored state, instead requiring developers to work at the level of the HTTP protocol itself.

For example, in Java:

public void handleRequest(HttpServletRequest request,
                          HttpServletResponse response)
{
    response.setStatus(200);
}

or in Clojure

(fn [request] {:status 200})

This programming model puts the HTTP request and response at centre stage. The concept of state is missing entirely - the resource is seen merely as an operation (or set of operations) available for remote invocation.

For years, the same RPC-centered approach has been copied by web frameworks in many languages, old and new (Python, Ruby, Go, Clojure, Rust…​). It has survived because it’s so flexible, as many low-level programming models are.

But there are significant downsides to this model too. HTTP is a big specification, and it’s unreasonable to expect developers to have the time to implement all the relevant pieces of it. What’s more, many developers tend to implement much the same code over and over again, for each and every operation they write.

A notable departure from this programming model can be found in Erlang’s WebMachine and Clojure’s Liberator. To a degree, these libraries ease the burden on the developer by orchestrating the steps required to build a response to a web request. However, developers are still required to understand the state transition diagram underlying this orchestration if they are to successfully exploit these libraries to the maximum extent. Fundamentally, the programming model is the same: the developer is still writing code with a view to forming a response at the protocol level.

While this model has served us well in the past, there are increasingly important reasons why we need an upgrade. Rather than mere playthings, HTTP-based APIs are becoming critical components in virtually every organisation. With supporting infrastructure such as proxies, API gateways and monitoring, there has never been a greater need to improve compatibility through better conformance with HTTP standards. Yet many APIs today at best ignored, and worst violate many parts of the HTTP standard. For ephemeral prototypes, this casual approach to HTTP is acceptable. Yet HTTP is designed for long-lived systems with lifetimes measured in decades, that must cross departmental and organisational boundaries, while adapting to ongoing changes in technology.

It’s time for a fresh approach. We need our libraries to do more work for us. For this to happen, we need to move from the de-facto operational view of web services to a strong data-oriented approach, focussing on what a web resource is really about: state.

Basics

1. Introduction

yada is a web library for Clojure that lets you define websites and web APIs using data.

Despite some differences, virtually all web libraries do a similar thing: call your code with some request data and ask you to return a response, leaving the bulk of the responsibililty for implementing the HTTP standards in your hands: responding with the correct status codes, response headers and ensuring semantics have been followed properly.

yada is different. Rather than leaving all the implementation details to you, it helps you out in doing the right thing (according to HTTP standards), thereby creating a better web [1].

It achieves this by providing you with a highly-configurable handler that is general enough to use in the vast majority of cases. Rather than coding the handler, the developer only has to configure one. This declarative approach provides greater scope for accuracy, consistency and re-use.

yada represents a break from the traditional yet stale method of building web backends, an approach that can trace its roots all the way back to the days of Common Gateway Interface. While this yada's data-oriented approach takes full advantage of the data-oriented philosophy of Clojure to the full, there’s no reason why this methodology cannot be implemented by new web libraries in other languages.

Using yada for your web project will allow you to finally exploit and benefit from the many features embodied in the outstanding design of HTTP, for scale, flexibility, security, longevity and interoperability.

1.1. Design goals

yada is designed to meet the following goals:

  • Be easy to use for intermediate Clojure developers

  • Comprehensive compliance with HTTP standards over pragmatism and performance

  • Increase productivity through re-use

  • Handle large workloads with reasonable performance

  • Support multiple architectural styles, including Hypermedia APIs (REST)

yada is not an experiment. It is designed for, and has been tested in, production environments.

yada is sufficiently quick-and-easy for quick prototype work but scales up when you need it to, to feature-rich secure services that can handle the most demanding workloads, while remaining faithful to the HTTP standards.

Some familiarity with HTTP will help you understand yada concepts quicker, but isn’t absolutely necessary. As you learn how to wield yada you will also discover and learn more about the HTTP standards as you go.

1.2. Say Hello! to yada

It’s quick to get started with yada without knowing how it works - it’s easy to get started even if you only have a basic knowledge of Clojure.

Let’s begin with a few examples. The obligatory Hello World! example is (yada/handler "Hello World!"), which responds with a message.

Perhaps you might want to serve a file? That’s (yada/handler (new java.io.File "index.html")).

Now you know how to serve a file, you know how to serve a directory. But perhaps you’ve got some resources on the classpath? (yada/handler (clojure.java.io/resource "talks/")).

What about (yada/handler nil)? Without knowing, can you guess what that might do? (That’s right, it produces a 404 Not Found response).

What about a quick dice generator? (yada/handler #(inc (rand-int 6))). Notice we use a function here, rather than a constant value, to vary numbers between rolls.

How about streaming those dice rolls as Server Sent Events? Put those dice rolls on a core.async channel, and return it with yada.

All these examples demonstrate the use of Clojure types that are converted on-the-fly into yada resources, and you can create your own types too.

Let’s delve a little deeper…

1.3. Resources

In yada, resources are defined by a plain-old Clojure map.

This has many benefits. While functions are opaque, data is open to inspection. Data structures are easy to generate, transform and query - chores that Clojure makes light work of.

Here’s an example of a resource:

(yada/resource
  {:properties {}
   :methods {:get {:response (fn [ctx] "Hello World!")}
             :put {}
             :brew {}}
 
})

There’s a lot of things you can do with a resource data model but perhaps the most obvious is to create a request handler from it to create responses from HTTP requests. That’s the role of a handler.

With yada, we transform a resource into a handler using the handler function.

(require '[yada.yada :as yada])

(yada/handler (yada/resource {}))

A handler can be called as a function, with a single argument representing an HTTP request. It returns a value representing the corresponding HTTP response.

A yada handler is an instance of the yada.handler/Handler record. Since this record satisfies clojure.lang.IFn, yada handlers behave just like normal Ring handlers and can be used wherever you might use a Ring handler.

1.4. Serving requests

To use yada to create real responses to real HTTP requests, you need to add yada to a web-server. The web server takes care of the networking and messages of HTTP (RFC 7230), while yada focuses on the semantics and content ([RFC7231] and upwards).

Currently, the only web server you can use is ztellman/aleph. This is because yada is built on an asynchronous abstraction provided by ztellman/manifold. However, there is no technical reason why, in future, other web servers can’t be wrapped by manifold.

In the meantime, Aleph (which wraps a much bigger library, Netty) provides a very capable and performant server.

To write real applications you also need a router that understands URIs, and yada has some features that are enabled when used with juxt/bidi, although there is nothing to stop you using yada with other routing libraries.

1.5. Conclusion

That’s yada in a nutshell, but the quickest way to to learn it is to set up an environment and play. That’s what we’ll do in the next chapter.

2. Getting Started

In this quick tutorial we’re going to run a real Clojure project, diving into the code to show how yada is used.

Our project is called Edge, a sample project from JUXT to show some of our libraries in action. It lives on GitHub.

We’ll clone it first, then build it, then run it, then browse the examples and even make modifications.

So let’s get going!

2.1. Clone

First let’s clone the project and change into its working directory.

git clone https://github.com/juxt/edge
cd edge

2.2. Build & Run

Next we build and run it, in development mode.

boot dev

This can take up to a couple of minutes to build and run from scratch so don’t worry if you have to wait a bit before you see anything.

Starting reload server on ws://localhost:43201
Writing boot_reload.cljs...
Writing boot_cljs_repl.cljs...
09:13:50.368 [clojure-agent-send-off-pool-0] INFO  edge.server - Started web-server on port 3000

Starting file watcher (CTRL-C to quit)...

Compiling {sass}... 3 changed files.

2.3. Browse

Fire up a browser and browse to http://localhost:3000/hello. You should see a simple Hello World message.

2.4. Connect a REPL

Let’s connect a REPL.

Start a new terminal and change into the project’s working directory again.

cd edge

If the project is still building, wait until you see the following line before trying to connect.

nREPL server started on port 5600 on host 127.0.0.1 - nrepl://127.0.0.1:5600

Connect a client REPL to this JVM like this:

boot repl

This should result in something similar to this:

REPL-y 0.3.7, nREPL 0.2.12
Clojure 1.8.0
OpenJDK 64-Bit Server VM 1.8.0_92-b14
        Exit: Control+D or (exit) or (quit)
    Commands: (user/help)
        Docs: (doc function-name-here)
              (find-doc "part-of-name-here")
Find by Name: (find-name "part-of-name-here")
      Source: (source function-name-here)
     Javadoc: (javadoc java-object-or-class-here)
    Examples from clojuredocs.org: [clojuredocs or cdoc]
              (user/clojuredocs name-here)
              (user/clojuredocs "ns-here" "name-here")
user=>

We’re going to start changing some of Edge’s source code soon, and when we do that we’ll type (reset) on our REPL. So let’s try that now.

user=> (reset)
:reloading (edge.web edge.api edge.server edge.system user)
:resumed
user=>

2.5. Test the service

Let’s send an HTTP request to the system to check it is working. We can use a browser to visit http://localhost:3000/hello or use curl if you have it installed on your system:

curl http://localhost:3000/hello

The result should be the same:

Hello World!

2.6. Locate the source code

Fire up an editor and load up the file src/edge/hello.clj.

Locate the function called hello-routes. This returns a simple route structure that matches on the URI paths of incoming HTTP requests.

(defn hello-routes [_]
  ["/hello" (yada/handler "Hello World!\n")])

Make a change to string "Hello World!", for example, change it to "Hello Wonderful World!".

2.7. Reset the system

Now we’ve made a change to Edge’s source code, we must tell the system to reset. The system will then detect all the code changes and necessary dependencies to reload.

user=> (reset)
:reloading (edge.web edge.api edge.server edge.system user)
:resumed
user=>

Let’s test the service again:

$ curl http://localhost:3000/hello

You should now see that the change has been made:

Hello Wonderful World!

Congratulations. You’re all up and running with a project built with yada. This will make a great lab to try out your own yada experiments and see what is possible.

Browse to http://localhost:3000 and have fun!

3. Hello World!

In this chapter we examine the "Hello World!" resource in depth. If you have followed the Getting Started chapter, you’ll be up and running and ready to try the examples yourself, but it’s up to you.

Let’s look at the route definition in the file src/edge/hello.clj (the same one we saw in the Getting Started chapter). This function returns a route structure. The simplest route structure is a pair of elements contained in a vector.

(defn hello-routes [deps]
  ["/hello" (1)
   (yada/handler "Hello World!\n") (2)
   ])
1 The first element is a path (or pattern) which matches on the URI’s path when handling HTTP requests.
2 The second element is often a handler function (but can be another route structure, recursively).

In this route structure, any incoming request with a path of /hello gets sent to the yada handler defined by (yada/handler "Hello World!\n").

Let’s focus on this handler. We could have used a standard Ring function (fn [req] {…}) here but instead we created one with yada/handler, which takes a resource and turns it into a handler function.

A resource is a Clojure map (or more accurately, a Clojure record) that completely describes the methods, properties, representations, security and other miscellaneous properties of a web resource, as data. The reason we can use a string ("Hello World") here is because yada contains logic to coerce a string into a resource.

There are a number of built-in coercions from various Clojure types and of course you can provide your own.

3.1. Examining the response

Let’s send a request which gets routed to our handler, which creates the response. Let’s examine this response in more detail, via curl.

$ curl -i http://localhost:3000/hello
The -i option to curl shows us the HTTP status and response headers as well as the body, which is very useful for debugging. If you ever need to see the request headers too, add -v.

The response should be something like following (note that headers may appear in a different order):

HTTP/1.1 200 OK
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Fri, 17 Jun 2016 16:44:23 GMT
Last-Modified: Fri, 17 Jun 2016 16:43:02 GMT
ETag: fa863bd7ff53786d286e4bb3c0134416
Content-Type: text/plain;charset=utf-8
Vary: accept-charset
Content-Length: 23
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff

The first three response headers are added by our webserver, Aleph.

Server: Aleph/0.4.0
Connection: Keep-Alive
Date: Fri, 17 Jun 2016 16:44:23 GMT

Next we have another date and a string known as the entity tag:

Last-Modified: Sun, 09 Aug 2015 07:25:10 GMT
ETag: fa863bd7ff53786d286e4bb3c0134416

The Last-Modified header shows when the string Hello World! was created, which happens to be the last time the system was started (or reset). Java strings are immutable, so yada is able to deduce that the string’s creation date is also the last time it could have been modified.

The entity tag is computed from the value of the Hello World! itself. Unlike the Last-Modified value, it can survive a reset.

Both Last-Modified and ETag are used to support HTTP conditional requests and conflict detection when uploading new versions of a resource.

Next we have a header telling us the media-type of the string’s representation.

Content-Type: text/plain;charset=utf-8

yada is able to determine that the media-type is text, but without more information it must default to text/plain. It can also tell us the charset, which defaults to the default charset of the JVM (almost always utf-8)

Vary: accept-charset

Since the Java platform can encode a string to a number of charsets, yada adds a Vary header to signal to the user-agent (and proxy caches in between) that the body can change if a request contained a different Accept-Charset header. Java installations support many different charsets, so yada does too.

Next we are given the length of the body.

Content-Length: 13

This value is in bytes, regardless of the charset. It includes the newline.

Finally we see our response body.

Hello World!

3.2. A conditional request

In HTTP, a conditional request is one where a user-agent (like a browser) can ask a server for the state of the resource but only if a particular condition holds.

A common condition is whether the resource has been modified since a particular date, usually because the user-agent already has a copy of the resource’s state which it can use if possible. If the resource hasn’t been modified since this date, the server can tell the user-agent that there is no new version of the state.

We can test this by setting the If-Modified-Since header in the request.

Here’s how we might do this using the curl command.

$ curl -i http://localhost:3000/hello -H "If-Modified-Since: Mon, 1 Jan 2525 00:00:00 GMT"

Of course, nobody will have modified the resource since the year 2525, so we should get a 304 response, telling us we can use our cached copy:

HTTP/1.1 304 Not Modified
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 0
Vary: accept-charset
ETag: fa863bd7ff53786d286e4bb3c0134416
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Wed, 22 Jun 2016 16:57:46 GMT

Notice we also get the same Vary and ETag headers. These help any proxies between the user-agent and the service properly cache content, and if they would have been produced in a 200 response, then they must also be produced in a 304. (This is the kind of thing that yada takes care of for you, unlike most libraries).

3.3. Content negotiation

The responses we have received back from our service all contain this curious header called Vary set to accept-charset. The server is telling us (and any proxies between us) that the representation might vary depending on the charset negotiated. Let’s see if we can get our "Hello World!" message returned in other charsets.

Let’s try getting the string in UTF16 by telling the server that’s the only charset we’ll accept:

curl -i http://localhost:3000/hello -H "Accept-Charset: UTF-16"

This returns the following:

HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 28
Content-Type: text/plain;charset=utf-16
Last-Modified: Sun, 26 Jun 2016 11:11:31 GMT
Vary: accept-charset
ETag: 43b1f79e8efe0fa97c32901fbd5746d6
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Mon, 27 Jun 2016 07:45:07 GMT

��Hello World!

The "Hello World!" message is prepended with 2 bytes called the Byte Order Mark (BOM). The length of the string (including the newline) is 13 characters. Since each character here is 2 bytes, that makes 26. The additional of the BOM makes it 28, which is what our Content-Length header reports.

A BOM indicates the order that the 2 bytes are transmitted in. In big endian form the most-significant byte is transmitted first. We can tell the service that we only want the big endian form with the following:

curl -i http://localhost:3000/hello -H "Accept-Charset: UTF-16BE"

This will now produce the message without the BOM, because it is unnecessary. This means our Content-Length will be exactly 13 * 2 = 26.

HTTP/1.1 200 OK
…
Content-Length: 26

Hello World!

If we were to use UTF-32, which defaults to big-endian, we’ll get a Content-Length of 13 * 4 = 52.

HTTP/1.1 200 OK
…
Content-Length: 52

Hello World!

Note also that different representations generate different ETag values. The entity tag is a way of managing a cache of representations, not a cache of resources. Think of the ETag value as the key you could use in a key/value store that stored a cache of representations.

The negotiation of charsets may be considered by some to be unnecessary given the dominance of UTF-8. That is certainly true for today’s modern browsers. However, there are many other types of devices that are being connected to the internet (under the banner Internet of Things). Many of these devices have very tight constraints on processing and memory which prevent them from supporting UTF-8. If we are building a web service, we may want to connect these devices to it in the future.

3.3.1. Languages

Of course it is not just charsets that can be negotiated. Another example is languages. Our "Hello World!" string is in English. Let’s provide support for simplified Chinese.

This calls for a different implementation:

(defn hello-language []
  ["/hello-language"
   (yada/resource (1)
    {:methods
     {:get (2)
      {:produces
       {:media-type "text/plain"
        :language #{"en" "zh-ch;q=0.9"}} (3)
       :response
       #(case (yada/language %) (4)
          "zh-ch" "你好世界\n"
          "en" "Hello World!\n")}}})])
1 Using the yada/resource function to create a custom resource
2 The resource has a single method, GET
3 English is preferred, but Simplified Chinese is available too
4 This is a function that is given a context as the first argument. The yada/language convenience function pulls out the negotiated language from this context

Let’s test this by providing a request header which indicates a preference for simplified Chinese:

$ curl -i http://localhost:3000/hello-language -H "Accept-Language: zh-CH"

We should get the following response:

HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 13
Content-Type: text/plain
Content-Language: zh-ch
Vary: accept-language
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Mon, 27 Jun 2016 08:20:59 GMT

你好世界

There’s a lot more to content negotiation than this simple example can show. It is covered in depth in subsequent chapters.

3.4. Mutation

Let’s try to overwrite the string by using a PUT.

$ curl -i http://localhost:3000/hello -X PUT -d "Hello Wonderful World!%0a"

The response is as follows:

HTTP/1.1 405 Method Not Allowed
Allow: GET, HEAD, OPTIONS
Content-Length: 284
Content-Type: application/json
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Mon, 27 Jun 2016 08:56:58 GMT

The response status is 405 Method Not Allowed, telling us that our request was unacceptable. There is also an Allow header, telling us which methods are allowed. One of these methods is OPTIONS, which we could have used to check whether PUT was available without actually attempting it.

$ curl -i http://localhost:3000/hello -X OPTIONS

The response should be:

HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Content-Length: 0
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Mon, 27 Jun 2016 09:00:27 GMT

Both the PUT and the OPTIONS response contain an Allow header which tells us that PUT isn’t possible. This makes sense, because we can’t mutate a Java string.

We could, however, wrap the Java string in a Clojure atom which could reference different Java strings at different times.

To demonstrate this, try the following with the identifier http://localhost:3000/hello-atom.

(yada/handler (atom "Hello World!\n"))

Let’s try a normal GET.

$ curl -i http://localhost:3000/hello-atom

We can now make another OPTIONS request to see whether PUT is available, before trying it.

$ curl -i http://localhost:3000/hello-atom -X OPTIONS
HTTP/1.1 200 OK
Allow: GET, DELETE, HEAD, OPTIONS, PUT
Content-Length: 0
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Tue, 05 Jul 2016 15:41:36 GMT

It is! So let’s try it.

$ curl -i http://localhost:3000/hello-atom -X PUT -d "value=Hello Wonderful World!%0a"

And now let’s see if we’ve managed to change the state of the resource.

$ curl -i http://localhost:3000/hello-atom
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 23
Content-Type: text/plain;charset=utf-8
Last-Modified: Tue, 05 Jul 2016 16:08:22 GMT
Vary: accept-charset
ETag: 3c3e0684be182b7185f6ad10b63f246a
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Tue, 05 Jul 2016 16:08:35 GMT

Hello Wonderful World!

As long as someone else hasn’t sneaked in a different state between your PUT and subsequent GET, you should see the new state of the resource is "Hello Wonderful World!". Great!

But what if someone did manage to PUT their change ahead of yours? Their version would now be overwritten. That might not be what you wanted. To ensure we don’t override someone’s change, we could have set the If-Match header using the ETag value.

Let’s test this now, using the ETag value we got before we sent our PUT request.

$ curl -i http://localhost:3000/hello-atom -X PUT -H "If-Match: fa863bd7ff53786d286e4bb3c0134416" -d "value=Hello Wonderful World!%0a"
HTTP/1.1 412 Precondition Failed
Content-Length: 196
Content-Type: application/json
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Tue, 05 Jul 2016 16:10:53 GMT

We get a 412, which means a pre-condition failed. The pre-condition in question relates to our If-Match header value not matching the current value of the atom. This is a very useful result, because it means we can ensure that we don’t overwrite someone else’s data.

3.5. A HEAD request

There was one more method indicated by the Allow header of our OPTIONS request, which was HEAD. Let’s try this now.

Use the option --head to curl to tell it to issue a HEAD request (and not to expect a request body).
$ curl -i --head http://localhost:3000/hello

The response does not have a body, but tells us the headers we would get if we were to try a GET request.

3.6. Parameters

Often, a resource’s state or behavior will depend on parameters in the request. Let’s say we want to pass a parameter to the resource, via a query parameter.

To show this, we’ll write some real code:

(require '[yada.yada :refer [yada resource]])

(defn say-hello [ctx]
  (str "Hello " (get-in ctx [:parameters :query :p]) "!\n"))

(def hello-parameters-resource
  (resource
    {:methods
      {:get
        {:parameters {:query {:p String}}
         :produces "text/plain"
         :response say-hello}}}))

(def handler (yada/handler hello-parameters-resource))

This declares a resource with a GET method, which responds with a plain-text message formed from the query parameter.

Let’s see this in action, but without a parameter:

$ curl -i http://localhost:3000/hello-parameter

Here we get a 400 response. This means we’ve done something wrong (we’ve forgotten to add the query parameter).

Now let’s add the query parameter to the URI:

$ curl -i http://localhost:3000/hello-parameter?p=Ken

This should now get the 200 response we wanted:

HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 11
Content-Type: text/plain
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Tue, 05 Jul 2016 16:23:26 GMT

Hello Ken!

Great!

As well as query parameters, yada supports path parameters, request headers, form data, cookies and request bodies. You can have optional parameters, in fact, anything that can be expressed in Plumatic Schema, and yada will even coerce parameters to a range of types. For more details, see the Parameters chapter.

3.7. Hello Swagger!

Now we have seen how to build a single web resource, let’s see how to build a Swagger description from a collection of web resources.

In your editor, switch to src/edge/server.clj. This file defines the overall route structure which includes our routes for "Hello World!". This has been included twice, both at the root and under the /api path.

This second version uses the Clojure threading macro -> which wraps the route structure with yada/swaggered and gives it a bidi tag (used for generating URIs, we’ll use this later).

[
    ;; Hello World!
    (hello-routes {})

    ["/api" (-> (hello-routes {})
                ;; Wrap this route structure in a Swagger
                ;; wrapper. This introspects the data model and
                ;; provides a swagger.json file, used by Swagger UI
                ;; and other tools.
                (yada/swaggered
                 {:info {:title "Edge API"
                         :version "1.0"
                         :description "An example API"}
                  :basePath "/api"})
                ;; Tag it so we can create an href to this API
                (tag :edge.resources/api))]]

The purpose of yada/swaggered is to augment the route structure given to it with a route to swagger.json, which responds with a Swagger description of the route structure in JSON. Since yada resources are data maps, this is a relatively simple data transformation of the route structure.

We can test the resource is available at its /api location with curl:

$ curl -i http://localhost:3000/api/hello

We can also query the Swagger description with curl:

$ curl -i http://localhost:3000/api/swagger.json

This time we get a JSON body returned:

HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 290
Content-Type: application/json
Last-Modified: Wed, 22 Jun 2016 15:45:16 GMT
Vary: accept-charset
ETag: 7833a69510d2b80f2a414c3c4ef2b4d4
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Wed, 22 Jun 2016 15:56:25 GMT

{"swagger":"2.0","info":{"title":"Edge API","version":"1.0","description":"An example API"},"produces":["application/json"],"consumes":["application/json"],"paths":{"/hello":{"get":{"produces":["text/plain"],"responses":{"default":{"description":""}}}}},"basePath":"/api","definitions":{}}

Notice we still get a Vary header telling us that multiple charsets are available. JSON bodies are available in UTF-16 and UTF-32. Compare this with Clojure’s EDN, which is specificed to be UTF-8 only. In fact, yada is happy to produce Swagger definitions in EDN too:

$ curl -i http://localhost:3000/api/swagger.edn

Note that this time we get no Vary header, since there are no charset alternatives.

HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 284
Content-Type: application/edn
Last-Modified: Wed, 22 Jun 2016 15:45:16 GMT
ETag: 3aa57341aa88d68108dbead14f5b462c
Server: Aleph/0.4.1
Connection: Keep-Alive
Date: Wed, 22 Jun 2016 15:59:26 GMT

{:swagger "2.0", :info {:title "Edge API", :version "1.0", :description "An example API"}, :produces ["application/json"], :consumes ["application/json"], :paths {"/hello" {:get {:produces ("text/plain"), :responses {:default {:description ""}}}}}, :basePath "/api", :definitions {}}

It’s these little details that yada takes care of for you. There is no trickery involved, it’s simply the result of an almost obsessive focus on the relevant web standards. There is nothing special about strings, yada applies the same logic for anything else you ask it to handle. We’ll see more in the next chapter.

By the way, if you want to see the Swagger UI, browse to http://localhost:3000/swagger/?url=http://localhost:3000/api/swagger.json

Swagger

3.8. Summary

This has been a long chapter, but we have only covered a simple "Hello World!" example.

You should now realise that implementing even a basic example that properly complies with the HTTP standard is a surprisingly tough undertaking. But this simple example demonstrated how a rich and functional HTTP resource can be created with a tiny amount of code, and none of the behaviour we have seen is hardcoded or contrived. We have only demonstrated a simple Java string, and yada includes similar support for many other basic types (atoms, Clojure collections, files, directories, Java resources…).

But the best thing is you can programmatically create your own resources and types to fit your particular requirements.

Without a library like yada we would need to read and understand hundreds of pages of the HTTP RFCs and spend a great deal of extra effort coding up its various aspects. Of course, nobody would bother doing that, but the consequence is that we miss out on the many architectural benefits of HTTP.

Rarely can client code make any assumptions that the HTTP API it is accessing complies with the text in the HTTP RFCs, and must therefore rely on detailed knowledge of how the API is written, either through documentation, Swagger definitions, close collaboration between development teams or some other means (trial-and-error). This is a problem because it causes rigidity between our systems.

By using yada, we are pushing the responsibility of implementing HTTP correctly away from the programmer and into a library.

4. Installation

yada is a Clojure library and if you are using it directly from a Clojure application with a Leiningen project.clj file, include the following in the file’s :dependencies section.

[yada "1.2.0"]
[aleph "0.4.1"]
[bidi "2.0.12"]

4.1. Setting up a development environment

The best way to learn is to experiment with yada, either on its own or by using it to build some service you have in mind. Either way you’ll want to set up a quick development environment.

4.2. The Easy Way: Clone edge

The quickest and perhaps easiest way to get started is to clone the yada branch of JUXT’s edge repository. This is a continuously improving development environment representing our firm’s best advice for building Clojure projects.

$ git clone git@github.com/juxt/edge

edge is opinionated and combines a number of practices that we’ve found useful on many projects at JUXT.

If you’re new to Clojure, we recommend learning yada with edge. Not only is it packed full of examples for you to explore, the project can provide a solid foundation for your own projects with an architecture that is proven to scale as your project grows.

4.3. The Simple Way: Construct your own

If you prefer to build your own development environment you’ll need a few pointers in order to integrate with yada.

4.3.1. Serving resources

To serve the resources you define with yada you’ll need to choose a port and start a web server. Currently, yada provides its own built-in web server (Aleph), but in the future other web servers will be supported. (For now, please be reassured that Aleph will support many thousands of concurrent connections. Aleph is built on Netty, a very capable platform used by Google, Apple, Facebook and many others to support extreme workloads.)

To start a web-server, just require listener from yada.yada and call it with a route structure and some configuration that includes the port number.

(require '[yada.yada :refer [listener resource as-resource]])

(def svr
  (listener
   ["/"
    [
     ["hello" (as-resource "Hello World!")]
     ["test" (resource {:produces "text/plain"
                        :response "This is a test!"})]
     [true (as-resource nil)]]]
   {:port 3000}))

The listener function returns a map. To shutdown the listener, call the 0-arity function contained in the :close entry.

((:close svr))

4.3.2. Creating resources

Resources can be created in the following ways:

  • From a resource model, calling resource with a map argument

  • Using as-resource on an existing type known to yada

  • A normal Ring handler

4.3.3. Routing to resources

A handler can be created from a resource by yada's handler function, and this handler can be used anywhere a Ring-compatible function is expected. This way, you can use any routing library you wish, including Compojure.

However, yada and bidi are designed to work together, and there are a number of features that are enabled when they are used together. That’s why you can put yada resources in a bidi route structure without turning them into handlers, since bidi knows how to do that already.

4.4. REPL and Testing

The response-for function is great for testing the Ring response that would be returned from a yada type or handler.

(require '[yada.yada :refer [response-for]])

(response-for "Hello World!")

5. Resources

In yada's terminology, a resource is the same as it is in HTTP:

The target of an HTTP request is called a "resource". HTTP does not limit the nature of a resource; it merely defines an interface that might be used to interact with resources. Each resource is identified by a Uniform Resource Identifier (URI),

— RFC 7231 Section 2

5.1. Resource models

We can describe a resource directly using a plain old Clojure map called a resource model.

Here’s an example:

(require '[yada.yada :refer [resource]])

(def my-resource
  (resource
    {:id :example
     :description "The description to this example resource"
     :summary "An example resource"
     :access-control 
     :properties 
     :parameters {:query  :path  :header }
     :produces 
     :consumes 
     :methods {:get  :put  :post  :delete  :patch }
     :responses {}
     :path-info? false
     :sub-resource 
     :logger 
     :interceptor-chain 
     :error-interceptor-chain 
     :custom/other }))

Resource models are constrained by a schema, ensuring they are valid. The purpose of the yada.yada/resource function is to check the resource model is valid. The schema will attempt to coerce invalid resource models into their valid equivalents wherever possible. An error will be thrown only after these coercions have been attempted.

The result of a call to yada.yada/resource is actually an instance of the Clojure record yada.resource/Resource but you can treat it just like a map.

The following sections describe the anatomy of a resource model in more depth.

You can augment a resource model with your own data if you like, but the keys you use must be namespaced keywords. Don’t use the yada namespace as that’s reserved for future use (in a future version of yada, all its keywords will be namespaced with yada).

5.1.1. Resource identity

The optional :id entry of the resource model gives the resource a unique identity. You can use whatever you like for the value, but it should be unique. A namespaced keyword is typical:

{:id :resources/user-profile}

The main reason for giving your resources an identity is for creating hyperlinks targeting your resource. For example, this is how you would create a URL to the resource.

(yada.yada/path-for ctx :resources/user-profile)

This feature is only available if your resources are declared in a bidi hierarchical route structure. Otherwise, the URL cannot be determined.

5.1.2. Resource description and summary

Optionally, a resource can contain a textual description. This should be used for any descriptive text that applies to the resource as a whole (rather than individual methods, which can contain their own descriptions).

{:description "<descriptive text here>"
 :summary "<summary here>"}

The description and summary values are used in generated Swagger descriptions and can be used for any other purpose you like.

5.1.3. Security & Access Control

The :access-control entry can be used to restrict access to a resource and provide security. It encompasses authentication and authorization, if necessary across multiple realms. It also determines the circumstances that the resource can be accessed from different origins for browser interaction (CORS) as well as defining other security protections.

Multiple authentication schemes are supported, such as Basic, JWT and OAuth2.

Access control and security are fully described in the security chapter.

5.1.4. Properties

You can define various properties on a resource. These can be thought of as a resource’s metadata, information about a resource (rather than the resource’s state).

It is possible to specify a complete map of constant properties, if they are all known prior to a request. This is rare, and usually it’s necessary to provide a function that will be called during the processing of a request.

{:properties (fn [ctx]
               {:exists? true
                :last-modified #inst "2016-07-25 16:00:00 Z"})}

Certain properties, such as :exists? and :last-modified are special and used by yada to determine responses.

For example, if you know how to determine the date that your resource was last modified, you should return this date in the :last-modified entry of a map containing your resources’s properties. Doing so will enable yada's logic for conditional requests, for instance, allowing it to return 304 Not Modified responses when appropriate.

More information can be found in Properties.

5.1.5. Parameters

Web requests can contain parameters that can influence the response and yada can capture these. This is especially useful when you are writing APIs.

There are different types of parameters, which you can mix-and-match:

  • Query parameters (part of the request URI’s query-string)

  • Path parameters (embedded in the request URI’s path)

  • Request headers

  • Form data

  • Request bodies

  • Cookies

There are benefits to declaring these parameters explicitly:

  • yada will check they exist, and return 400 (Malformed Request) errors on requests that don’t provide the ones you need for your logic

  • yada will coerce them to the types you want, so you can avoid writing loads of type-conversion logic in your code

  • yada and other tools can process your declarations independently of your request-processing code, e.g. to generate API documentation

Parameter declaration, validation and coercion is a big topic and fully covered in the parameters chapter.

5.1.6. Representations

Resources have physical forms called representations. A resource can declare all the representations it supports.

Typically, a representation will have a designated content type, such as text/html or application/json, which tells the receiver how to process it.

Example: The string "Hello World!" might have the type text/plain. But the string "<h1>Hello World!</h1>" might be given the type text/html to indicate that it should be rendered as HTML.

If the content type of a representation begins with text/, it might also have a given charset, indicating how the bytes transferred should be turned into text.

Some representations will also indicate whether the content is compressed (called the encoding) and maybe the language used.

It is often useful to distinguish between outward representations that can be produced and the inward representations that can be consumed.

The :produces entry in the resource model declares the representations of the resource that can be produced.

Where there is more than one representation that can be produced, yada negotiates which type, if any, is actually produced taking into account the declared preferences of the user agent. This process is known as content negotiation.

The :consumes entry declares the incoming representations that the resource is able to accept. Some HTTP methods allow requests to contain bodies. Here there is no content negotiation, since the user agent will tell the server the content type of the body it is sending.

More details can be found in the representations chapter.

5.1.7. Methods

The :methods entry is a map, where each key is a keyword that corresponds to an HTTP method.

{:methods
  {:get {}
   :post {}
   :brew {}}}

There is no restriction on the methods you can declare.

The value of each method entry is also a map, which has the following structure:

{:response (fn [ctx] )
 :parameters {}
 :produces {}
 :consumes {}
 :authorization {}
 :description ""
 :summary ""
 :responses {404 {:description "Not found"}}
 :custom/other }

Each method has a specific prescribed behaviour, called the method’s semantics, which usually described in a particular RFC document (but it’s also fine to define your own).

Method semantics for core HTTP methods are built-in to yada but it’s possible to add your own via a Clojure protocol.

Many method semantics will involve a call to the function you declare in the :response entry, which is responsible for constructing the response, but if you’re not sure you should check the description for the actual method you’re using in the methods chapter.

5.1.8. Responses

By default, yada will produce error messages and stack traces for various status codes. If you wish to override this behaviour, you must provide alternatives via the :responses entry of the resource map.

For example, perhaps you want to provide a particular response that is generated whenever there is a 404 Not Found error. Many websites like to do this, perhaps as a hint to the user to check the URL.

In the response map we would add something like this:

{:responses
  {404 {:produces #{"text/html"}}
        :response (fn [ctx] )}}

5.1.9. Path info

The path-info? entry is a boolean flag which indicates whether the resource expects a path-info.

Imagine your URI path is /dir/abc/foo.txt. You may want to partially match this path such that the resource is called for all URIs that begin with /dir/. In this case, abc/foo.txt would be set as the path-info in the request map.

The reason we might want to indicate this on the resource is to tell our router that a partial match is required, and to give us access to the remaining path.

5.1.10. Sub-resources

Sometimes we cannot know the properties of a given resource up-front. For example, imagine you are serving files from a file-system. It is impossible to determine which resources will be present when the request arrives, and therefore which properties and content attributes should be declared.

To support such dynamic resources, yada allows the declaration of a function, as the value of the :sub-resource key, that will be called when the request arrives. The return value of the sub-resource function must return the actual resource.

This feature is commonly used together with path-info to provide dynamic groups of related resources.

5.1.11. Logging

The :logger entry can declare a function which is called whenever a request is processed and the response is about to be returned to the web-server. This allows you to log all requests to a file, for instance.

5.1.12. Interceptor chains

yada is built on a chain of interceptors that are processed asynchronously. For most cases, the default interceptor chain will suffice, but sometimes it is necessary to add to this chain, or modify it in some way, on a resource-by-resource basis. This is achieved by providing an alternative interceptor chain via the :interceptor-chain and :error-interceptor-chain entries.

5.2. Resources as Ring handlers

Now we have introduced all the entries that a resource model can contain, let’s use our knowledge to re-create a basic "Hello World!" resource:

(require '[yada.yada :as yada])

(def my-resource
  (yada/resource
    {:produces {:media-type "text/plain"}
     :methods {:get
                {:response (fn [ctx] "Hello World!")}}}))

Now we have a valid resource, we can now use it for a range of purposes — one obvious one is to handle HTTP requests. We can create a Ring request handler from a resource with the yada.yada/handler function:

(def my-ring-handler
  (yada/handler my-resource))

We can now use this handler in a route.

For example, with Compojure:

(GET "/my-resource" [] my-ring-handler)

Or with bidi:

["/my-resource" my-ring-handler]

Note, since yada is aware of bidi’s bidi.ring.Ring protocol, resources can be used in bidi route structures directly:

["/my-resource" my-resource]

5.2.1. Responding to requests

The handler created by yada works by constructing a series of internal functions called interceptors.

When a request is received, the handler creates a new instance of an object known as the request context, and its idiomatic symbol is ctx.

Each interceptor is a single-arity function that takes this request context as an argument, returning the same request context or a modified copy.

Here’s an interceptor which adds some information into the request context:

(fn my-interceptor [ctx]
  (assoc ctx :film "Life Of Brian"))

On each request, the request context is threaded through a chain of interceptors, the result of each interceptor being used as the argument to the next.

One of the entries in the request context is :response, which contains the Ring response that will be returned to the web server. Any interceptor can modify this (or any other value) in the request context.

Here’s an example of a request context during the handling of a request:

{:request {:method :get :headers {}}
 :request-id #uuid "bf2c06e1-b4bd-49fb-aa74-05a17f4e9e9c"
 :method :get
 :response {:status 200 :headers {} :body "Hello!"}}

The request context is not just passed to interceptors, but to functions you can declare in your resource.

5.3. Resource types

A resource type is a Clojure type or record that can be automatically coerced into a resource model. These types must satisfy the yada.protocols.ResourceCoercion protocol, and any existing type or record may be extended to do so, using Clojure’s extend-protocol macro.

(extend-type datomic.api.Database
  yada.protocols/ResourceCoercion
  (as-resource [_]
    (resource
      {:properties
        {:last-modified }
       :methods
        {:get }}})))

The as-resource function must return a resource (by calling yada.resource/resource, not just a map).

6. Parameters

As we learned in Parameters, parameters declarations are useful to validate and coerce request parameters, produce 400 status responses on bad requests thereby providing some protection against bad input data.

Many requests embed parameters in their URIs. For example, let’s imagine a URI to access the transactions of a fictitious bank account.

https://bigbank.com/accounts/1234/transactions?since=tuesday

There could be 2 parameters here. The first, 1234, is contained in the path /accounts/1234/transactions. We call this a path parameter.

The second, tuesday, is embedded in the URI’s query-string (after the ? symbol). We call this a query parameter.

You can declare these parameters in the resource model.

{:parameters {:path {:entry Long}}
 :methods {:get {:parameters {:query {:since String}}}
           :post {:parameters {:body }}}

Parameters can be specified at resource-level or at method-level. Path parameters are usually declared at the resource-level because they form part of the URI that is independent of the request’s method. In contrast, query parameters usually apply to GET requests, so it’s common to define this parameter at the method-level, and it’s only visible if the method we declare it with matches the request method.

We declare parameter values using the syntax of Prismatic's schema library. This allows us to get quite sophisticated in how we define parameters.

(require [schema.core :refer (defschema)]

(defschema Transaction
  {:payee String
   :description String
   :amount Double}

{:parameters {:path {:entry Long}}
 :methods {:get {:parameters {:query {:since String}}}
           :post {:parameters {:body Transaction}}}

6.1. Capturing multi-value parameters

Occasionally, you may have multiple values associated with a given parameter. Query strings and HTML form data both allow for the same parameter to be specified multiple times.

/search?accno=1234&accno=1235

To capture all values in a vector, declare your parameter type as a vector type:

{:parameters {:query {:accno [Long]}}}}

6.2. Capturing large request bodies

Sometimes, request bodies are very large or even unlimited. To ensure you don’t run out of memory receiving this request data, you can specify more suitable containers, such as files, database blobs, Amazon S3 buckets or your own extensions.

All data produced and received from yada is handled efficiently and asynchronously, ensuring that even with very large data streams your service continues to work.

{:parameters {:form {:video java.io.File}}}

7. Properties

Properties tell us about the current state of a resource, such as whether the resource exists, or when the resource was last modified. Properties allow us to determine whether the user agent’s cache of a resource’s state is up-to-date.

Sometimes all a resource’s properties are constant and can be known when the resource is defined. More likely the resource’s properties have to be determined by some logic, and often this logic involves I/O.

Also, if the resource has declared parameters, it can be that the resource’s properties depend in some way on these parameters. For example, the properties of account A may well be different from the properties of account B.

A resource’s properties may also depend on who is making the request. Your bank account details should only exist if you’re the one accessing them. If I tried to access your bank account details, you’d want the service to behave differently.

For this reason, a resource’s properties declaration in the resource-model points to a single-arity function that is called by yada after the request’s parameters have been parsed and the credentials of a caller have been established.

In many cases, it will be necessary to query a database, internal web-service or equivalent operation involving I/O.

If you use a properties function, anything you return will be placed in the :properties entry of the request-context. Since the request-context is available when the full response body is created, you may choose to return the entire state of the resource, in addition to its properties. This may be sensible if it helps avoid a second trip to the database.

8. Methods

Methods are defined in the methods entry of a resource’s resource-model.

Only methods that are known to yada can appear in a resource’s definition. Each method corresponds to a type that extends the yada.methods.Method protocol. This design also makes it possible to add new methods to yada, as required.

The responsibility of each method type is to encode the semantics of the corresponding method as it is defined in HTTP standards, such as responding with the correct HTTP status codes (most other web frameworks delegate this responsibility to developers).

Some methods can be implemented entirely by yada itself (HEAD, OPTIONS, TRACE etc.). Most methods, however, delegate to some function or functions declared in the method’s declaration in the resource-model.

8.1. Method semantics, by method

Each HTTP method has defined semantics. Often these semantics are defined in the HTTP standards, other RFCs or, in the case of custom methods, by you.

These semantics are important because they allow other web agents, such as browsers and proxies, to inter-operate with your site.

Below is an explanation of the semantics for every method yada currently supports and your responsibilities should you choose to provide the method for a resource.

8.2. GET

Specify a function in :response that will be called during the GET method processing.

If the resource exists, the single-arity function will be called with the request context as its only argument.

It should return the response’s body, which should satisfy yada.body.MessageBody determining how exactly the response’s body will be returned.

8.3. PUT

(coming soon)

8.4. POST

(coming soon)

8.5. DELETE

(coming soon)

(coming soon)

8.7. OPTIONS

(coming soon)

8.8. TRACE

(coming soon)

8.9. PATCH

(coming soon)

8.10. Handling all methods

If you want to handle all request methods, or a complex set of them, you can specify the special keyword :* in the methods section of your resource model.

{:methods
  {:*
    {:response (fn [ctx] )}}}

8.11. Custom methods

Custom methods can be added by defining new types that extend the yada.methods.Method protocol.

8.12. BREW

BREW is an example of a custom method you might want to create, especially if you are building a networked coffee maker compliant with RFC.

(require '[yada.methods Method])

(deftype BrewMethod [])

(extend-protocol Method
  BrewMethod
  (keyword-binding [_] :brew)
  (safe? [_] false)
  (idempotent? [_] false)
  (request [this ctx] ))

9. Representations

Resources have state, but when this state needs to be transferred from one host to another, we use one of a number of formats to represent it. We call these formats representations.

A given resource may have a large number of actual or possible representations.

Representation may differ in a number of respects, including:

  • the media-type (file format)

  • if textual, the character set used to encode it into octets

  • the (human) language used (if textual)

  • whether and how the content is compressed

Whenever a user-agent requests the state from a resource, a particular representation is chosen, either by the server (proactive) or client (reactive). The process of choosing which representation is the most suitable is known as content negotiation.

9.1. Producing content

Content negotiation is an important feature of HTTP, allowing clients and servers to agree on how a resource can be represented to best meet the availability, compatibility and preferences of both parties. It is a key factor in the survival of services over time, since both new and legacy media-types can be supported concurrently. (It is also the mechanism by which new versions of media-types can be introduced, even media-types that define hypermedia interactions, more on this later.)

9.2. Proactive negotiation

There are 2 types of content negotiation. The first is termed proactive negotiation where the server determines the type of representation from requirements sent in the request headers. These are the headers beginning with Accept.

For any resource, the available representations that can be produced by a resource, and those that it consumers, are declared in the resource model. Every resource that allows a GET method should declare at least one representation that it is able to produce.

Let’s start with a simple web-page example.

{:produces "text/html"}

This is a short-hand for writing […​]

(missing text here)

clojure {:produces [{:media-type "text/plain" :language #{"en" "zh-ch"} :charset "UTF-8"} {:media-type "text/plain" :language "zh-ch" :charset "Shift_JIS;q=0.9"}]

( todo - languages )

$ curl -i http://localhost:8090/hello-languages -H "Accept-Charset: Shift_JIS" -H
"Accept: text/plain" -H "Accept-Language: zh-CH"
HTTP/1.1 200 OK
Content-Type: text/plain;charset=shift_jis
Vary: accept-charset, accept-language, accept
Server: Aleph/0.4.0
Connection: Keep-Alive
Date: Mon, 27 Jul 2015 18:38:01 GMT
Content-Length: 9

?�D���E!

9.3. Reactive negotiation

The second type of negotiation is termed reactive negotiation where the agent chooses from a list of representations provided by the server.

(Currently, yada does not yet support reactive negotiation but it is definitely on the road-map.)

9.4. The Vary response header

(coming soon)

9.5. Body coercion

Where necessary, and according to the semantics of the HTTP method, yada will coerce the result of a method into an entity body of the negotiated content type.

For example, consider the following resource-map.

clojure {:produces "application/json" :methods {:get {:response (fn [_] {:greeting "Hello"})}}}

On receiving a GET request, yada will automatically convert the map {:greeting "Hello"} into the JSON body {"greeting":"Hello"}.

However, the key phrase here is where necessary. If a string is returned from a method, and the content type is application/json, there is some ambiguity between whether the resource developer intends for yada to encode the string into JSON, or whether the string is already JSON encoded. In these ambiguous cases, yada assumes the string is JSON encoded already.

Therefore this code would produce an error when the user agent attempts to decode the JSON string.

{:produces "application/json"
 :methods
 {:get
  {:response (fn [_] "This is not JSON")}}}

If you are faced with this situation, you should choose one of the following options:

  • Return a map with the JSON string embedded. For example: {:message "Plain old string"}.

  • Use a JSON encoder such as org.clojure/data.json or Cheshire and encode the string yourself.

9.6. Consuming content

(coming soon)

10. Responses

In resource interactions, a request is processed by a method, usually resulting in a response. The response contains a status code, headers and a body.

yada attempts to set the correct status code and headers according to the semantics of both the method and mime-type. It also coerces the result of a method into the response body if necessary.

Sometimes, however, the response returned by yada is not what you want. There are times that you want more fine-grained control, want to provide custom bodies for certain status codes (such as 404 errors), or want deviate from the HTTP standards entirely.

10.1. Explicit responses

When you need complete control over the response you should return the request-context's response, modified if necessary. In which case yada will see that you want to be explicit and get out of your way.

(fn [ctx]
 (let [response (:response ctx)]
  ;; return a response, explicitly associating
  ;; (or updating) the status, headers or body.
  (update-in response [] )
 ))

10.2. Declared responses

yada declares the responses that may normally be produced by a method. It adds these declarations to the resource-model when creating a resource prior to processing.

However, with explicit responses you may generate response codes that are unexpected by yada and which could be declared through the resource-model.

In these cases, you should declare the response codes in the :responses entry of the resource’s resource-model.

{:responses
 {418 {:description "I'm a teapot"}}}

The keys in the :responses map can be integers, sets of integers, or the wildcard: *. Only sets are supported, so if you need to produce a range of status codes, create a set programmatically:

{(set (concat (range 400 404) (range 405 500))}
 {:description "All errors besides 404"
  
  }}

Individual status codes take precedence over sets of status codes, which take precedence over wildcards. After that, the order in which keys are checked is undefined. If you are using sets, avoid declaring the same status code in multiple keys.

10.3. Status responses

Usually yada will return the responses produced by methods, and create ones for errors that occur along the way.

Often it is useful to be able to control the response body, or even the complete response, for responses with certain status codes.

We can specify these controlled responses by specifying a response entry in the value corresponding to the status code.

A common example is providing a custom 404 page when a resource cannot be found, which may provide the user with details of why the resource couldn’t be found and perhaps what to do next.

Let’s see how this is done:

(require '[yada.yada :as yada])

{:properties {:exists? false}

 :responses
 {404 {:description "Not found"
       :produces #{"text/html" "text/plain;q=0.9"}
       :response (let [msg "Oh dear I couldn't find that"]
                   (fn [ctx]
                     (case (yada/content-type ctx)
                       "text/html" (html [:h2 msg])
                       (str msg \newline))))}}}

Note that the response definition can include a declaration of the representations that the response can produce.

If the response is caused by an error, the actual error is available in the context under :error.

11. Security

Built-in to the library, yada offers a complete standards-based set of security features for today’s secure applications and content delivery.

11.1. Security is part of the resource, not the route

In yada, resources are self-contained and are individually protected from unauthorized access. We agree with the HTTP standards authors when we consider security to be integral to the definition of the resource itself, and not an extra to be bolted on afterwards. Nor should it be complected with routing. The process of identifying up a resource from its URI is independent of how that resource should behave, and shouldn’t be coupled to it.

Building security into each resource yields other benefits, such as making it easier to test the resource as a unit in isolation of other resources and the router.

As in all other areas, yada aims for 100% compliance with core HTTP standards when it comes to security, notably RFC 7235. Also, since HTTP APIs are nowadays used to facilitate transactional integration between systems via the user’s browser, it is critically important that yada fully supports systems that offer APIs to other applications, across origins, as standardised by CORS.

11.2. The :access-control entry

All security aspects for a resource are specified in its model’s :access-control entry.

11.3. Authentication

Let’s look at authentication first. Authentication is the process of establishing and verifying the identity and credentials of the user, with reasonable confidence that the user is not an impostor.

In yada, authentication happens after any request parameters have been processed, so if necessary they can be used to establish the identity of the user. However, it is important to remember that authentication happens before the resource’s properties have been loaded, since credentials do not have to do with the actual resource. Thus, if the user is not genuine, we might well save a wasted trip to the resource’s data-store.

In HTTP, resources can exist inside a protection space determined by one or more realms. Each resource declares the realm (or realms) it is protected by, as part of the :access-control entry of its resource-model.

11.3.1. Authentication schemes

Each realm declares one or more authentication schemes governing how requests are authenticated.

yada supports numerous authentication schemes, including custom ones you can provide yourself.

Each scheme has a verifier. Depending on the scheme this is usually a function. The verifier is used to extract or otherwise establish the credentials in the request, ensuring they are authentic and true, in which case it returns these credentials as a value to be stored in the request context. These credentials may contain information such as the user’s identity, roles and privileges. If no credentials are found, the verifier should return nil.

If no credentials are found by any of the schemes, a 401 response is returned containing a WWW-Authenticate header.

11.3.2. Basic authentication

Here is an example of a resource which uses Basic authentication described in RFC 2617

{:access-control
  {:realm "accounts"
   :scheme "Basic"
   :verify (fn [[user password]] )}}

There are 3 entries here. The first specifies the realm, which is defaults to default in Basic Authentication, but if specified is contained in the dialog the browser presents to the user.

The second declares we are using Basic authentication.

The last entry is the verify function. In Basic Authentication, the verify function takes a single argument which is a vector of two entries: the username and password.

If the user/password pair correctly identifies an authentic user, your function should return credentials.

(fn [[user password]]
  
  {:email "bob@acme.com"
   :roles #{:admin}})

If the password is wrong, you may choose to return either an empty map or nil. If you return an empty map (a truthy value) and the resource requires credentials that aren’t in the map, a 403 Forbidden response will be returned. However, if you return nil, this will be treated as no credentials being sent and a 401 Unauthorized response will be returned.

From a UX perspective there is a difference. If the user-agent is a browser, returning nil will mean that the password dialog will reappear for every failed login attempt. If you return truthy, it will show the 403 Forbidden response.

You may choose to limit the number of times a failed login attempt is tolerated by setting a cookie or other means.

11.3.3. Digest authentication

(coming soon)

We can also use cookies to present authentication credentials. The advantage of cookies is that they can be set by the server based on custom authentication interaction with the user, such as the submission of a login-form.

To protect a site with cookies:

{:access-control
  {:scheme :cookie
   :cookie "session"
   :verify (fn [cookie] }}

11.3.5. JWT authentication

(coming soon)

11.3.6. Form-based logins

Basic Authentication has a number of weaknesses, such as the difficulty of logging out and the lack of control that a website has over the fields presented to a human. Therefore, the vast majority of websites prefer to use a custom login form generated in HTML.

You can think of a login form as a resource that lets the user present one set of credentials in order to acquire additional ones. The credentials the user presents, via a form, are verified and if they are true, a cookie is generated that certifies this. This cookie provides the certification to subsequent requests in which it is sent.

Let’s start by building this login resource that will provide a login form page to browsers and verify the form data when that form is submitted.

Here’s a simplistic but viable resource model for the two methods involved:

(require
 '[buddy.sign.jwt :as jwt]
 '[schema.core :as s]
 '[hiccup.core :refer [html])

{:methods
 {:post
  {:consumes "application/x-www-form-urlencoded"
   :parameters {:form
                {:user s/Str :password s/Str}}

   :response
   (fn [ctx]
     (let [{:keys [user password]} (get-in ctx [:parameters :form])]
       (if (valid-user user password)
         (assoc (:response ctx)
                :cookies {"session"
                          {:value
                           (jwt/sign {:user user} "lp0fTc2JMtx8")}})
         "Try again!")))}
  :get
  {:produces "text/html"
   :response (html
              [:form {:method :post}
               [:input {:name "user" :type :text}]
               [:input {:name "password" :type :password}]
               [:input {:type :submit}]])}}}

The POST method method consumes incoming URL-encoded data (the classic way a browser sends form data). It de-structures the two parameters (user and password) from the form parameters.

We then determine if the user and password are valid (we don’t explain here how this is done, but assume a valid-user function exists that can tell us). If the user is valid we associate a new cookie called "session" with the response. By starting with the :response value of the request context, we ensure yada interprets our return value as a Ring response rather than some other value.

We use Buddy’s sign function to sign and encoded the cookie’s value as a JSON string. We only specify the credentials as {:user user} in this case, but we could put much more into that map. The sign function requires us to provide a secret symmetric key that we can use for both signing and verification, but the library does allow us asymmetric key options too.

The other method, GET, simply produces a form for user-agents that can render HTML (browsers, typically) to post back. For reasons of cohesion, it’s a good idea to provide these two methods in the same resource to encapsulate and dedupe the fields which are relevant to both the GET and the POST.

11.3.7. Protecting resources

(coming soon)

11.3.8. Logout

The recommended way of logging out is to remove the session.

11.3.9. Bearer authentication (OAuth2)

(coming soon)

11.3.10. Multifactor authentication

(coming soon)

11.4. Authorization

Authorization is the process of allowing a user access to a resource. This may require knowledge about the user only (for example, in Role-based access control). Authorization may also depend on properties of the resource identified by the HTTP request’s URI (as part of an Attribute-based access control authorization scheme).

In either case, we assume that the user has already been authenticated, and we are confident that their credentials are genuine.

In yada, authorization happens after the resource’s properties have been loaded, because it may be necessary to check some aspect of the resource itself as part of the authorization process.

By default, yada will use a declarative role-based authorization scheme.

11.4.1. Default authorization scheme

Any method can be protected by declaring a role or set of roles in its model.

{:access-control
 {:authorization
  {:methods
   {:post :accounts/create-transaction}}}}

If multiple roles are involved, they can be composed inside vectors using simple predicate logic.

{:access-control
 {:authorization
  {:methods
   {:post [:or [:and :accounts/user
                     :accounts/create-transaction]
               :superuser}}}}

Only the simple boolean operators of :and, :or and :not are allowed in this authorization scheme. This keeps the role definitions declarative and easy to extract and process by other tooling.

Of course, authentication information is available in the request context when a method is invoked, so any method may apply its own custom authorization logic as necessary. However, yada encourages developers to adopt a declarative approach to resources wherever possible, to maximise the integration opportunities with other libraries and tools.

11.4.2. Custom authorization scheme

A custom authorization scheme can be declared that will completely replace the default authorization scheme already discussed.

First, decide on a keyword that will be used to dispatch your authorization function. In this example, we’ve chosen :my/custom-authorization.

Now declare the authorization function that will be called by yada during request processing. This is a defmethod, as follows:

(defmethod yada.authorization/validate
  :my/custom-authorization
  [ctx credentials authorization]

)

The credentials argument contains all the verified credentials sent in the request.

Now add an :authorization map to the :access-control part of your resource model. The map must contain a :scheme value specific to your resource model, along with any extra parameters you want to be passed as the authorization argument to your authorization function. In this example, we want to pass the :my/ensure parameter set to [:same-account]. You can specify anything you like to be passed as parameters (there are no schema restrictions here).

{:access-control
 {:authorization
  {:scheme :my/custom-authorization
   :my/ensure [:same-account]}}}

11.5. Realms

yada supports multiple realms. By default, there is a single realm in operation called "default". However, you can group authentication schemes and authorization models in separate realms. Each realm can contain multiple authentication schemes (it might be that a realm offers a choice of how to authenticate).

{:access-control
  {:realms {"Gondor" {:authentication-schemes []
                      :authorization {}}
            "Mordor" {:authentication-schemes {}
                      :authorization {}}}}}

11.6. Cross-Origin Resource Sharing (CORS)

yada fully supports Cross-Origin Resource Sharing (CORS) allowing you to provide APIs that are accessible from other origins.

For example, you may be creating an API that you wish other websites to make use of, by allowing browsers visiting those websites access to your API.

CORS is specified in the :access-control section of the resource-model.

{:access-control
 {:allow-origin "*"
  :allow-credentials false
  :expose-headers #{"X-Custom"}
  :allow-methods #{:get :post}
  :allow-headers ["Api-Key"]
 }}

With the exception of :allow-credentials (which must be a boolean), any of the values can be declared as single-arity functions, which are called with the request-context as an argument to determine the value for the corresponding response header.

11.7. HTTP Strict Transport Security (HSTS)

clojure {:strict-transport-security {:max-age 12000}}

Defaults to a maximum age of 31536000.

The HSTS header is only set if the scheme is HTTPS or the service is behind a proxy (determined by the presence of the X-Forwarded-For request header).

11.8. Content Security Policy

{:content-security-policy "url-src"}

Defaults to default-src https: data: 'unsafe-inline' 'unsafe-eval'.

11.9. Clickjacking prevention

A browser’s iframe can be used for click-jacking. By default yada tells browsers not to allow this. The default value is SAMEORIGIN, unless you override it in the resource-model.

{:x-frame-options "NONE"}

11.10. Cross-site Scripting (XSS) protection

yada also sets the X-Xss-Protection response header to 1; mode=block. This can be overridden in the resource model.

{:x-content-type-options "0"}

11.11. Media-type sniffing protection

By default, yada sets the X-Content-Type-Options response header to nosniff. This tells browsers not to try to attempt to determine the content-type of the response body.

Since yada sets the Content-Type header according to HTTP standards, there should never be a need for a browser to sniff the response body for this information, preventing an attack that might exploit some vulnerability in this process.

12. Routing

Since the yada function returns a Ring-compatible handler, it is compatible with any Clojure URI router.

However, yada is designed to work especially well with its sister library bidi, and unless you have a strong reason to use an alternative routing library, you should stay with the default.

While yada is concerned with semantics of how a resource responds to requests, bidi is concerned with the identification of these resources. In the web, identification of resources is a first-class concept. Each resource on the web is uniquely identified with a Uniform Resource Identifier (URI).

No resource is an island, and it is common that resource representations need to embed references to other resources. This is true both for ad-hoc web applications and hypermedia APIs, where the client traverses the application via a series of hyperlinks.

These days, hyperlinks are so critical to the reliable operation of systems that it is no longer satisfactory to rely on ad-hoc means of constructing these URIs, they must be generated from the same tree of data that defines the API route structure.

12.1. Declaring your website or API as a bidi/yada tree

A bidi routing model is a hierarchical pattern-matching tree. The tree is made up of pairs, which tie a pattern to a (resource) handler (or a further group of pairs, recursively).

Both bidi’s route models and and yada's resource models are recursive data structures and can be composed together.

The end result might be a large and deeply nested tree, but one that can be manipulated, stored, serialized, distributed and otherwise processed as a single dataset.

(require
  [yada.yada :refer [resource]]
  [hiccup.core :refer [html]]
  [clojure.java.io :refer [file])

;; Our store's API
["/store/"
 [ ; Vector containing our store's routes (bidi)
  ["index.html"
   {:summary "A list of the products we sell"
    :methods
    {:get
     {:response (file "index.html")
      :produces "text/html"}}}]
  ["cart"
   {:summary "Our visitor's shopping cart"
    :methods
    {:get
     {:response (fn [ctx] )
      :produces #{"text/html" "application/json"}}
     :post
     {:response (fn [ctx] )
      :produces #{"text/html" "application/json"}}}}]
  
 ]]

A yada handler (created by yada’s yada function) and a yada resource (created by yada’s resource constructor function) that extends bidi’s Matched protocol are both able to participate in the pattern matching of an incoming request’s URI.

For a more thorough introduction to bidi, see https://github.com/juxt/bidi/README.md.

12.2. Declaring policies across multiple resources

Many web frameworks allow you to set a particular behavioral policy (such as security) across a set of resources by specifying it within the routing mechanism.

In our view, this is wrong, for many reasons. A URI is purely a identifier for a resource. A resource’s identifier might change, but such a change should not cause the resource to bahave differently.

In the phraseology offered by Rich Hickey in his famous Simple Made Easy talk, we should not complect a resource’s identification with its operation. Neither should we complect a protection space with a URI space. Doing so adds an unnecessary constraint to the already difficult problem of naming things (URIs) while adding an unnecessary constraint to the ring-facing of resources into protection spaces. For this reason, yada and bidi are kept apart as separate libraries.

Some web frameworks can be excused for offering a pragmatic way of reducing duplication in specification, but this really ought not to be necessary for Clojure programmers who have powerful alternatives.

What are these alternatives? How can we avoid typing the same declarations over and over in every resource?

One option is to create a function that can augment a set of base resource models with policies. That function can then be mapped over a number of resources.

A variation of this option is to use Clojure’s built-in tree-walking functions such as clojure.walk/postwalk. If you specify your entire API as a single bidi/yada tree, it is easy to specify each policy as a transformation from one version of the tree to another. What’s more, you will be able to check, debug and automate testing on the end result prior to handling actual requests.

13. Example 2: Phonebook

We have covered a lot of ground so far. Let’s consolidate our knowledge by building a simple application, using all the concepts we’ve learned so far.

We’ll build a simple phonebook application. Here is a the brief:

13.1. Phonebook requirements

Create a simple HTTP service to represent a phone book.

Acceptance criteria. - List all entries in the phone book. - Create a new entry to the phone book. - Remove an existing entry in the phone book. - Update an existing entry in the phone book. - Search for entries in the phone book by surname.

A phone book entry must contain the following details: - Surname - Firstname - Phone number - Address (optional)

13.2. The database

Create a new namespace called phonebook.db.

(ns phonebook.db)

We’ll create a database constructor, and some functions to access its contents.

This constructor creates a map with two entries, both refs. We could use an atom, but refs offer more flexibility.

(defn create-db [entries]
  {:phonebook (ref entries)
   :next-entry (ref (inc (apply max (keys entries))))})

Now let’s add some code to add an entry.

(defn add-entry [db entry]
  (dosync
   (let [nextval @(:next-entry db)]
     (alter (:phonebook db) conj [nextval entry])
     (alter (:next-entry db) inc)
     nextval)))

13.3. Creating new phonebook entries

For this requirement, we are going to support the POST method.

Let’s add the following entry to the static properties of the IndexResource.

{:parameters {:post {:form {:surname String
                           :firstname String
                           :phone [String]}}}

}

This declaration tells yada what parameters we are expecting in the POST method. In return, yada will do the following:

  1. Validate the request, ensuring that all the mandatory parameters have been provided

  2. Coerce the parameters to the desired types (if possible).

  3. Return a 400 response (if not).

  4. Parse the request body.

  5. Help prevent XSS scripting attacks, by ensuring that no unexpected parameters are allowed to pass through. We should still be careful of String parameters though.

14. Swagger

All yada resources are built on data which can be published in a variety of formats. A popular format is Swagger, which allows APIs to be quickly documented. This is particularly useful when multiple teams of developers need to share their service documentation with others during development.

Swagger involves the creation of JSON-formatted specifications which, in the absence of libraries like yada, are hand-authored. There exist a variety of code generation libraries that can take hand-authored specifications and generate code in various programming languages. However, a key disadvantage with code-generation approaches like this is they do not support round-trip engineering, that is, the steady iterative co-evolution of the specification with the code.

Since Swagger covers both URI routing as well as resources, we must involve routing information in the set of resources we wish to publish. Currently, bidi is the only supported router, but it should be possible to support other data-driven routers such as Silk in future.

The first task is to create and publish the swagger specification for a set of resources.

14.1. Creating the specification: the easy way

The easiest way of creating a Swagger spec is by following these steps:

14.1.1. Step 1:

Creating a bidi route structure containing your yada handlers (remember a yada handler is a Ring handler). Remember, bidi is infinitely recursive, so you can group your resources however you like. Just use the vector-of-vectors syntax in place of a usual handler.

["/greetings"
  [
    ["/hello" (yada "Hello World!\n")]
    ["/goodbye" (yada "Goodbye!\n")]
  ]
]

14.1.2. Step 2: Wrap the route structure in swaggered

The swaggered function can be used to further wrap the route structure.

This function takes 2 arguments. The first argument is a base template map which contains all the static data that should appear in the spec. such as the Swagger service meta-data. The second argument is simply the bidi routing tree containing your yada resources. The data contained in both the routing tree and the resources themselves is used to construct the specification.

For example, let’s take the bidi routing tree below:

(require '[yada.swagger :refer [swaggered]]')

["/api"
 (swaggered
  ["/greetings"
   [
    ["/hello" (yada "Hello World!\n")]
    ["/goodbye" (yada "Goodbye!\n")]
   ]
  ]
  {:info {:title "Hello World!"
          :version "1.0"
          :description "A greetings service"}
   :basePath "/api"}
  )]

This declares a route that (partially) matches on the URI path /api. The second element of the route pair is a custom record created with swaggered which satisfies bidi’s Matched protocol. This acts as a sort of junction which matches on the routes given in the second argument. Critically, however, it also adds an additional sub-route, /swagger.json, which exposes the Swagger specification of the given base template, routes and resources.

Therefore, to access the JSON swagger specification in the example above, you would navigate to /api/swagger.json. Also, use /api/greetings/hello and /api/greetings/goodbye to access the services.

14.1.3. Creating the specification: the simple way

The Swagger spec is merely a map that can be created with yada.swagger/swagger-spec-resource. Once you have this map, publish it with yada (you should know how to do this already. Hint: (yada m)).

There is a function yada.swagger/swagger-spec-resource that creates a yada resource for you, and can optionally take a content-type to publish the spec in HTML and EDN too.

This approach gives you more flexibility, since you aren’t tied to publishing your swagger spec in the same route structure as your API (you might want to publish it on another server perhaps).

14.2. The Swagger UI

Once you have published the Swagger specification you should use the Swagger UI to access it.

If you use the swaggered convenience function, a Swagger UI will automatically be hosted under the route. Use a single / to redirect to the UI for the Swagger specification of your routing tree. In the example above, navigating to /api/ will bring up the Swagger UI allowing you to browse and play with the API.

Greetings API in Swagger

It is also possible to host your own Swagger UI and link it to your published Swagger specifications. Just pass the url query parameter to the Swagger UI to indiciate the location of the yada-produced Swagger specification you want to browse.

Advanced users: For an example of custom Swagger UI configuration see dev/resources/swagger/phonebook-swagger.html for an example.

14.3. Swagger and REST

In some sense Swagger definitions compete with REST as an architecture. Where REST encourages self-describing APIs, Swagger tends towards well-documented APIs. REST APIs are particularly well suited to fluid public APIs which can support gradual evolution and a diverse and potentially unknown set of clients. In contrast, Swagger APIs and suited to fixed private APIs inside a single organisation of between multiple collaborating parties.

14.4. Data descriptions of APIs

Although the REST approach avoids publishing full API specifications up-front, preferring discovery over documentation, there are still many situations where it is useful to derive a data representation of an API.

One example is for API deployment to Amazon Web Services, where an API on the cloud can be created programmatically.

14.5. References

See IBM’s Watson Developer Cloud for a sophistated Swagger example.

Advanced topics

15. Async

Under normal circumstances, with Clojure running on a JVM, each request can be processed by a separate thread.

However, sometimes the production of the response body involves making requests to data-sources and other activities which may be I/O-bound. This means the thread handling the request has to block, waiting on the data to arrive from the I/O system.

For heavily loaded or high-throughput web APIs, this is an inefficient use of resources. Today, this problem is addressed by asynchronous I/O programming models. The request thread is able to make a request for data via I/O, and then is free to carry out further work (such as processing another web request). When the data requested arrives on the I/O channel, a potentially different thread carries on processing the original request.

As a developer, yada gives you fine-grained control over when to use a synchronous programming model and when to use an asynchronous one.

15.1. Deferred values

A deferred value is simply a value that may not yet be known. Examples include Clojure’s futures, delays and promises. Deferred values are built into yada — for further details, see Zach Tellman’s manifold library.

In almost all cases, it is possible to return a deferred value from any of the functions that make up our resource record or handler options.

For example, suppose our resource retrieves its state from another internal web API. This would be a common pattern with µ-services. Let’s assume you are using an HTTP client library that is asynchronous, and requires you provide a callback function that will be called when the HTTP response is ready.

On sending the request, we could create a promise which is given to the callback function and returned to yada as the return value of our function.

Some time later the callback function will be called, and its implementation will deliver the promise with the response value. That will cause the continuation of processing of the original request. Note that at no time is any thread blocked by I/O.

Async

Actually, if we use Aleph or http-kit as our HTTP client the code is even simpler, since both libraries return promises from their request functions.

(require '[yada.yada :refer [resource]]
         'aleph.http :refer [get])

(resource
  {:methods
    {:get (fn [ctx] (get "http://users.example.org"))}})

In a real-world application, the ability to use an asynchronous model is very useful for techniques to improve scalability. For example, in a heavily loaded server, I/O operations can be queued and batched together. Performance may be slightly worse for each individual request, but the overall throughput of the web server can be significantly improved.

Normally, exploiting asynchronous operations in handling web requests is difficult and requires advanced knowledge of asynchronous programming techniques. In yada, however, it’s easy, thanks to manifold.

For a fuller explanation as to why asynchronous programming models are beneficial, see the Ratpack documentation. (Note that yada provides all the features of Ratpack and more).

16. Example 3: Search engine

(coming soon)

17. Server Sent Events

17.1. Introduction

Server Sent Events (SSE) is a part of the HTML5 generation of specifications that describes a capability for delivering events, asynchronously, from a server to a browser or other user-agent, over a long-lived connection.

SSE conceptually similar to web sockets. However, a key difference is that SSE is layered upon HTTP and thus inherits the protocol’s support for proxying, authorization, cookies and is integrated with Cross-Origin Resource Sharing (CORS).

In contrast, web sockets are raw TCP sockets that share nothing with HTTP except for the ability for a user agent to use the HTTP protocol to initiate a web socket connection. After that, everything is up to agreements between the client and server.

Since yada is designed to support HTTP, it does not provide anything extra to support web sockets beyond that which is provided by the web server.

17.2. SSE with yada

To create Server Sent Event streams with yada, return a stream of data from a response.

For example, a stream of data could be a core.async channel. It is important that you set the representation to be text/event-stream, so that a client recognises this as a Server Sent Event stream and keeps the connection open.

(require '[clojure.core.async :refer [chan]])

{:methods {:get {:produces "text/event-stream"
                 :response (chan)}}}

It is, however, highly unusual to want to provide a channel of data to a single client. Typically, what is required is that each client gets a copy of every message in the channel. This can be achieved easily by multiplexing the channel with clojure.core.async/mult, which yada will recognise and tap on your behalf.

(require '[clojure.core.async :refer [mult]])

(let [mlt (mult channel)]
  {:methods {:get {:produces "text/event-stream"
                   :response mlt}}})

Of course, you can tap the mult yourself in your own logic and provide the tapping channel directly to yada, which will do the right thing depending on what you provide.

18. Example 4: Chat server

(coming soon)

19. Handling request bodies

(coming soon)

20. Example 5: Selfie uploader

(coming soon)

21. Handlers

(coming soon)

22. The Request Context

When given the HTTP request, the handler first creates a request-context and populates it with various values, such as the request and the resource-model that corresponds to the request’s URI.

The handler then threads the request-context through a chain of functions, called the interceptor-chain. This chain is just a list of functions specified in the resource-model that has been carefully crafted to generate a response that complies fully with HTTP standards. However, as with anything in the resource-model, you can choose to modify it to your exact requirements if necessary.

The functions making up the interceptor-chain are not necessarily executed in a single thread but rather an asynchronous event-driven implementation enabled by a third-party library called manifold.

22.1. path-for

Rather than hardcode URLs, you can create the correctly URL with yada’s path-for function.

(require '[yada.yada :refer [path-for]])

(path-for ctx :resources/user-profile)

23. Interceptors

The interceptor chain, established on the creation of a resource. A resource’s interceptor chain can be modified from the defaults.

23.1. Core interceptors

(coming soon)

23.1.1. available?

(coming soon)

23.1.2. known-method?

(coming soon)

23.1.3. uri-too-long?

(coming soon)

23.1.4. TRACE

(coming soon)

23.1.5. method-allowed?

(coming soon)

23.1.6. parse-parameters

(coming soon)

23.1.7. authenticate

(coming soon)

23.1.8. get-properties

(coming soon)

23.1.9. authorize

(coming soon)

23.1.10. process-request-body

(coming soon)

23.1.11. check-modification-time

(coming soon)

23.1.12. select-representation

(coming soon)

23.1.13. if-match

(coming soon)

23.1.14. if-none-match

(coming soon)

23.1.15. invoke-method

(coming soon)

23.1.16. get-new-properties

(coming soon)

23.1.17. compute-etag

(coming soon)

23.1.18. access-control-headers

(coming soon)

23.1.19. create-response

(coming soon)

23.1.20. logging

(coming soon)

23.1.21. return

(coming soon)

23.2. Modifying interceptor chains

Say you want to modify the interceptor chain for a given resource.

You might want to put your interceptor(s) at the front.

(yada.resource/prepend-interceptor
  resource
  my-interceptor-a
  my-interceptor-b)

Alternatively, you might want to replace some existing core interceptors:

(update resource
  :interceptor-chain
  (partial replace {yada.interceptors/logging my-logging
                    yada.security/authorize my-authorize}))

Or you may want to insert some of your own before a given interceptor:

(yada.resource/insert-interceptor
  resource yada.security/authorize my-pre-authorize)

You can also append interceptors after a given interceptor:

(yada.resource/append-interceptor
  resource yada.security/authorize my-post-authorize)

24. Sub-resources

Usually, it is better to declare as much as possible about a resource prior to directing requests to it. If you do this, your resources will expose more information and there will be more opportunities to utilize this information in various ways (perhaps serendipitous ones). But sometimes it just isn’t possible to know everything about a resource, up front, prior to the request.

A classic example is serving a changing directory of files. Each file might be a separate resource, identified by a unique URI and have different possible representations. Unless the directory’s contents are immutable, you should only determine the number and nature of files contained therein upon the arrival of a request. For this reason, yada has sub-resources. Sub-resources are resources that are created just-in-time when a request arrives.

24.1. Declaring sub-resources

Any resource can declare that it manages sub-resources by declaring a :sub-resource_ entry in its resource-model. The value of a sub-resource is a single-arity function, taking the request-context, that returns a new resource, from which a temporary handler is constructed to serve just the incoming request.

Sub-resources are recursive. A resource that is returned from a sub-resource function can itself declare that it provides its own sub-resources, ad-infinitum.

24.2. Path info

When routing, it is common for resources that provide sub-resources to match a set of URIs all starting with a common prefix, and extract the rest of the path from the request’s :path-info entry. This is achieved by declaring a :path-info? entry in the resource-model set to true.

(resource
  {:path-info? true
   :sub-resource
   (fn [ctx]
     (let [path-info (get-in ctx [:request :path-info])]
       (resource {})))})

(For a good example of sub-resources, readers are encouraged to examine the code for yada.resources.file-resource to see how yada serves the contents of directories.)

25. Example 6: File server

(coming soon)

26. Testing

Using yada.response-for, you can create a response from a bidi route structure containing yada resources for testing purposes:

(response-for ["/foo" (yada "hello")] :get "/foo")

Reference

27. Glossary

Resource

A resource

Handler

A handler

Appendix A: Reference

A.1. Resource model schema

This is defined in yada.schema.

A.2. Handler schema

(coming soon)

A.3. Request context schema

(coming soon)

A.4. Protocols

yada defines a number of protocols. Existing Clojure types and records can be extended with these protocols to adapt them to use with yada.

A.4.1. yada.protocols.ResourceCoercion

(coming soon)

A.4.2. yada.methods.Method

Every HTTP method is implemented as a type which extends the yada.methods.Method protocols. This way, new HTTP methods can be added. Each type must implement the correct semantics for the method, although yada comes with a number of built-in methods for each of the most common HTTP methods.

Many method types define their own protocols so that resources can also help determine the behaviour. For example, the built-in GetMethod type uses a Get protocol to communicate with resources. The exact semantics of this additional protocol depend on the HTTP method semantics being implemented. In the Get example, the resource type is asked to return its state, from which the representation for the response is produced.

A.4.3. yada.body.MessageBody

Message bodies are formed from data provided by the resource, according to the representation being requested (or having been negotiated). This removes a lot of the formatting responsibility from the resources, and this facility can be extended via this protocol for new message body types.

A.5. Built-in types

There are numerous types already built into yada, but you can also add your own. You can also add your own custom methods.

A.5.1. Files

The yada.resources.file-resource.FileResource exposes a single file in the file-system.

The record has a number of fields.

Field

Type

Required?

Description

file

java.io.File

yes

The file in the file-system

reader

Map

no

A file reader function that takes the file and selected representation, returning the body

representations

A collection of maps

no

The available representation combinations

The purpose of specifying the reader field is to apply a transformation to a file’s content prior to forming the message payload.

For instance, you might decide to transform a file of markdown text content into HTML. The reader function takes two arguments: the file and the selected representation containing the media-type.

(fn [file rep]
  (if (= (-> rep :media-type :name) "text/html")
    (-> file slurp markdown->html)
    ;; Return unprocessed
    file))

The reader function can return anything that can be normally returned in the body payload, including strings, files, maps and sequences.

The representation field indicates the types of representation the file can support. Unless you are specifying a custom reader function, there will usually only be one such representation. If this field isn’t specified, the file suffix is used to guess the available representation metadata. For example, a file with a .png suffix will be assumed to have a media-type of image/png.

A.5.2. Directories

The yada.resources.file-resource.DirectoryResource record exposes a directory in a filesystem as a collection of read-only web resources.

The record has a number of fields.

Field

Type

Required?

Description

dir

java.io.File

yes

The directory to serve

custom-suffices

Map

no

A map relating file suffices to field values of the corresponding FileResource

index-files

Vector of Strings

no

A vector of strings considered to be suitable to represent the index

A directory resource not only represents the directory on the file-system, but each file resource underneath it.

The custom-suffices field allows you to specify fields for the FileResource records serving files in the directory, on the basis of the file suffix.

For example, files ending in .md may be served with a FileResource with a reader that can convert the file content to another format, such as text/html.

(yada.resources.file-resource/map->DirectoryResource
  {:dir (clojure.java.io "talks")
   :custom-suffices {"md" {:representations [{:media-type "text/html"}]
                           :reader markdown-reader}}})

28. Bibliography

  • [RFC2616] HTTP 1.1

  • [RFC7231] HTTP 1.1 - Semantics and Content

29. Colophon

This book is authored in AsciiDoc, which is a plain-text format like Markdown. AsciiDoc has the benefit of being based on the mature DocBook standard.

Sources for this book can be found in the yada repository under the doc/ directory. The file book.adoc describes the structure of the book.

29.1. HTML

Asciidoctorj is used to generate HTML.

29.2. PDF

The main font is Noto Sans, licensed under the Open Font License.

Monospace font is Droid Sans Mono, licensed under the Apache License.

The yada font is based on Rochester licensed under the Apache License, Version 2.0.

Asciidoctor is used to generate DocBook 5 XML which is output to LaTeX using dblatex.


1. https://www.pandastrike.com/posts/20151019-create-more-web