Rewriting the MailOnline

Keeping the world’s busiest news website rolling

by Jon Pither

Published 2017-07-31

In 2013, The MailOnline, wanted to overhaul their news website, servicing 11.34 million readers worldwide each day. The revised website needed to be faster and more extensible, simpler and easier to maintain.

The MailOnline needed a flexible platform that could cater for multiple views across different devices and to handle more views of personalised content. A previous attempt to rewrite their site using a Java EE stack had been cut short. It was vital that the next attempt succeeded.

Against a backdrop of organisational transformation - in particular Programmer Anarchy introduced by Fred George - Jon Pither, of JUXT, led a team to rewrite the newspaper website in Clojure.

Node.js and Ruby were also considered, but after two weeks, the Clojure team produced a sample skeleton application that mirrored the current homepage almost in entirety. Unanimous agreement followed that the Clojure project should be given the resources and personnel to succeed.

website

To ensure a smooth and efficient process, JUXT ran a series of training courses to help developers make the transition to Clojure. At its peak, there were 12 developers working on the project and the team was able to incorporate a number of satellite services into the redevelopment of the website.

The MailOnline’s brief was to rebuild from scratch an identical website; to run the new and old website side by side and to gradually change over the load-balancer routes, gracefully fading out the old system whilst testing and monitoring the abilities of the new stack.

To reassure stakeholders that the rewrite was under control, the team built a bespoke regression testing system to compare the old and new websites, ably demonstrating that the new system was semantically identical.

website

Powered by ElasticSearch for speed and document retrieval via query, the resulting Clojure system was an order of magnitude less in code size than the previous Java system.

In 2016 JUXT returned for six months to review the current state of the code-base and to perform some recommended refactorings to keep the code-base modern and modular.

submit to reddit