Introducing yada

A new approach to creating API services on the web

by Malcolm Sparks

Published 2016-06-06

Today, more and more companies are building APIs (Application Programming Interfaces) between their internal systems, and many are also exposing public APIs to the web.

The catalyst to this trend has been the increasing dominance of the HTTP protocol. The emergence of micro-services is only accelerating the adoption of APIs within organisations.

But developing the software behind solid reliable web APIs is terribly difficult. There are many things that matter: time-to-market, reliability, security, performance, sustainability, longevity and more.

Current approaches to API development require too much care and attention from software developers, a precious and scarce resource.

This is the first in a series of blog articles that will introduce yada, a major new library from JUXT that we have designed from scratch after a major re-think of how we were approaching the development of APIs for our clients.

Why Clojure?

First off, let us just come out and say it: Clojure is an almost perfect language for writing web APIs, probably the best there is. It's designed from the ground up to immunize against the nasty side-effects that plague systems built with 'object-oriented' languages like Java, C#, Ruby and Python. The result is safe, solid, reliable services you can trust.

And as Clojure is hosted on Java's JVM, it gets great performance, mature support for internationalization, unicode, character sets, and access to thousands of libraries, in particular those that allow it reach into every conceivable corner of an organisation.

Furthermore, Clojure has powerful functions for data crunching and munging. Serializing and deserializing data, often implemented in Java with agonizing effort, is a breeze in Clojure thanks to its built-in support for nested maps, lists and trees. If you've ever had to serialize Clojure data to JSON you'll know what we mean.

But the promise of Clojure for web API has not yet been realized. This is partially because we have too readily borrowed approaches from other languages that don't fit Clojure's radical approach to programming.

It's time to change that.

The Shadow of the Past

Whether it's CGI, WSGI, Rack, Servlets or Ring, it's surprising that our approach to developing web APIs hasn't changed since the birth of HTTP, 20 years ago. Almost every web library, framework and tool works in much the same way: some data representing the HTTP request is assembled, presented to the programmer, who is then expected to re-implement the relevant parts of the HTTP specification to form a response. It's no wonder most web APIs implement HTTP poorly and inaccurately.

Our new web library, yada, offers a different approach, one that was initiated by Erlang's WebMachine and continued by Liberator.

Resources as Data

Liberator (like WebMachine that inspired it) presents a flow-chart oriented model for HTTP where the user can provide predicates to provide decisions at various points in the HTTP request flow. There are elements to Liberator that are declarative but it is mostly about predicates. yada, however, promotes the declarative aspects of Liberator to a maximum. The result strikes a harmonious balance between data and code.

Let's illustrate this balance with an example, exercising some of the yada's features. See if you can work out what it does, and in particularly try to figure out the username and password that protects it. Once you have, give it a try!

{:properties
 (fn [ctx]
   (let [thing (-> ctx :parameters :query :thing)]
    ;; For ETag generation
    {:version thing}))

 :methods
 {:get
  {:parameters {:query {:thing s/Str}}
   :produces [{:media-type #{"text/html" "text/plain"}
               :language #{"en" "fr" "de"}}]
   :response
   (fn [ctx]
     (let [thing (-> ctx :parameters :query :thing)
           message (case (yada/language ctx)
                     "en" (format "Hello %s!" thing)
                     "fr" (format "Bonjour, %s!" thing)
                     "de" (format "Guten Tag! %s!" thing))]
       (case (yada/content-type ctx)
         "text/html" (str "<h1>" message "</h1>")
         "text/plain" message)))}}

 :access-control
 {:scheme "Basic"
  :verify (fn [[user password]]
            (when (= [user password] ["guest" "guest"])
              {:user "guest"
               :roles #{"user"}}))
  :authorization {:methods {:get "user"}}}}

This doesn't look like any web handler you'd create by wrapping Ring middleware together. It's as much a declarative description of the handler as an implementation of one.

With a declarative description we can know a lot about the service without running it, or knowing what the functions do. We know what methods it has (GET). We know what parameters are needed in the query string. We know what media types and languages are available. We even know what authentication scheme is in place, and what roles a user needs to access the resource.

A huge advantage of this declarative approach is that handlers can be produced with data, and data can be generated on an industrial scale. This allows us to transition, finally, the development burden of writing HTTP services from a craft to an industry. We shall elaborate this point in a future post, but first, let's focus on how yada works.

One Ring (handler) to rule them all

Essentially, yada is a Clojure function that takes a declarative description (like the one above) and returns a Ring handler function. That's it!

Due to yada's design, there is now just one Ring handler to forge (and we've already done the forging). Every developer using yada exploits the benefits of a single, expertly designed, Ring handler. What's more, as this handler improves these improvements are inherited by existing applications, often without requiring any developer intervention.

So what features are contained within this handler?

  • Parameter validation
  • Content negotiation (media-types, charsets, encodings and languages)
  • Conditional requests
  • Method semantics
  • Status codes
  • Response body coercion
  • Pluggable methods and media-types
  • Authentication (basic, session, OAuth2, custom)
  • Authorization (role-based, custom)
  • Security headers
  • Swagger
  • Optional async
  • Server sent events
  • bidi integration
  • Multipart request bodies
  • Asynchronous uploads
  • Schema validation (with coercion) of resource models
  • Dynamic sub-resources
  • Modifiable interceptor chains
  • Test helpers

Some of these are introduced below and will be expanded upon further in future blog posts.

Input validation of parameters

On our OnTheMarket project, fellow JUXTer Martin Trojer built the defences to validate every web request was genuine and valid. As the ideas for yada began to form, Martin insisted that there were no compromises on strict validation throughout. Along the way, Martin also introduced us to the power of coercions.

Poor validation is a common cause of security holes that can be exploited by attackers. And while it's time-consuming to add robust validation to every individual web endpoint you write, that doesn't make it any less critical.

Like fnhouse, yada uses schema to declare query parameters, route (path) parameters, custom headers, form fields and request bodies.

Content Negotiation

Sadly, there are few Clojure web libraries that provide built-in support for HTTP Content Negotiation (Liberator being a notable example). This is surprising because Content Negotiation is a key ingredient to producing web APIs that stand the test of time.

'Proactive' negotiation in yada is fully supported, with 'reactive' negotiation soon to follow. Resources can declare rich and precise combinations of media-type, charset, encoding and language.

Conditional Requests

A conditional request is one where the user-agent (browser or other device) already has some versions of the content and wants to ensure that version is the most recent.

Imagine you are polling a resource for a list of your emails every second. By using a conditional request and an indication of which version of the list you have, the server can reply with a 304 Not Modified without rebuilding the list of recent emails each time.

Conditional requests are a key ingredient to building scaleable HTTP services and now has its own RFC (RFC 7232).

Methods

In HTTP, every request to a URI is combined with a verb, called a method. HTTP itself defines the semantics for some methods, while others have been defined by other RFCs. The semantics of each method differ depending on what the method is for. These semantics define what the method should do, and the status codes that should be returned.

In yada the semantics for the core methods are baked into the library, so your HTTP service will 'do the right thing' without you having to do anything. However, there are escape hatches for you to override these semantics when necessary. Plus, you can add your own methods and code their semantics in the same way.

Security

Today, security is too important to be left to extensions. Security is built into yada. Today, it supports 'Basic' authentication, session (cookie-based), JWT tokens, OAuth2 and you can add your custom schemes. There is also role-based authorization built in, but again, you can define your own authorization scheme.

Staying true to the HTTP specifications, yada supports multiple concurrent authentication schemes across multiple realms, if you need that level of granularity.

As you might expect, CORS is built in, allowing you fine-grained control over which websites can access your services. Plus various security headers are built in to give your APIs the best possible resiliance to a range of attacks.

Asynchronous support

yada has complete support for utilizing asynchronous programming techniques to maximise efficiency, concurrency and avoid blocking threads.

With yada every callback function can elect to return a value, or something that will eventually provide one (called a deferred value), such as a future, a promise or core.async channel. This makes it easy to replace blocking calls with asynchronous ones, gradually if necessary. Even when you need to make a series of non-blocking calls, these can be composed into a single deferred value. Again, we'll discuss more about how this is done in a future post but some readers may know already that Zach Tellman's manifold library is heavily involved.

Downsides

With every choice there are trade-offs, and there are some significant downsides with choosing to use yada that you should be aware of.

Firstly, there's no getting around it that yada isn't as flexible as Ring and comes with many constraints (although these constraints are broadly those that are imposed by the HTTP specification).

Secondly, the library is new, and its more advanced features (like fully asynchronous multipart uploads) are still being battle-tested. That said, we serve our entire website with yada, even the article you're reading now.

Another potential downside for some is that yada only works at the level of individual handlers, not routes. In yada, routes (resource identifiers) are treated separately from resources themselves. We'll address this point in another future post.

Finally, right now the library only supports Netty 4.1 (via Aleph). Netty isn't a bad choice, it's used by Apple, Google and others for their most demanding workloads. But in the due course we hope yada will support more web servers, such as Undertow and Jetty.

Conclusion

In the Clojure tradition, yada is extensible and composable. It isn't an application framework, indeed, it omits many of the things most web 'frameworks' include, such as templating, database access and even URI routing (we have another popular library for that). However, it is a large library conguent with HTTP's large surface area.

If you're a purist and like to deploy services that comply with standards, you'll love yada.

If you're more of a pragmatist and just want to get the job done (and many of us are, especially in the Clojure community), you might still find that once you gain familiarity, yada is quicker and easier than other approaches. Plus you might also start building up your own library of functions that produce resources that fit your organisation's particular needs and policies.

If you can't wait until the end of this series and want to jump into yada now, visit the GitHub project page. We also have a documentation website at juxt.pro/yada with a user manual and examples. Plus, there's a low-volume discussion list you can join for announcements and discussion, and many yadarians hang out on the Slack channel. There is a Gitter channel too.

Next time we'll compare yada with more conventional approaches to writing HTTP services in Clojure. In the meantime, please feel free to ask any questions in comments section below.

submit to reddit