Or: how to generate AWS HTTP request signatures with Clojure.
I recently wrote a simple service that is essentially an HTTP proxy for S3-based storage.
The existing server that hosts this is based on Netty, which is framework for high-performance asynchronous network services, and it already uses Netty’s HTTP I/O support in several places. So it seemed obvious that the new service would use Netty’s HTTP support to receive HTTP requests from a client and talk to S3. However, I hadn’t banked on Amazon’s authentication mechanism.
At first glance Amazon’s AWS4 header-based authentication signatures looked alarmingly complex, and I was sorely tempted to run into the arms of Amazon’s Java AWS client library. However, the fact that Amazon’s library would live entirely outside of the Netty I/O framework we’re using gave me pause.1
Why DSL's can be better than the real thing.
It’s always nice to see well-written, considered, and non-inflammatory writing on the static vs dynamic typing debate, and this post by Elben Shira titled “The End Of Dynamic Languages” is definitely in that category.
I’m not actually going to talk much about static vs dynamic typing here, except to say this: as someone who came from many years of working with static languages (C, C++, Java) and switched allegiance to a dynamic language,1 I don’t think there’s a yes/no answer to which is “best”. One delivers benefits by catching logic errors early, while restricting programs to those that the developer can prove to be correct within its type system. The other enables free expression and experimentation, at the cost of allowing type errors to manifest at runtime, and putting more onus on the developer to keep their logic maintainable.
Put it this way: I’d use one for controlling a nuclear reactor, and the other for developing a web application — you can probably guess which.
Writing clearer code without map and filter.
I love programming by slicing and dicing maps and lists in Clojure. The data-first approach of Clojure — and functional programming languages in general — feels right after years of hiding data behind object-oriented walls.
But one thing has been a thorn in my side since starting with Clojure four years ago: sometimes I still feel like a clumsy beginner when it comes to processing nested trees of maps and lists.
Here’s what I mean. Say I have a list of sites owned by a restauranteur. Each one of the sites has one or more stations that provide service to customers, and I want to generate a list of the ID’s of every station on all sites.
Java (and Clojure) developers sending Apple push notifications often need to do this, and it’s surprisingly under-documented.
If you search for “apns certificate java keystore” online you’ll find precious little information, even though surely there must be many, many Java-based servers talking to iOS devices via Apple’s Push Notification Service (APNs). Is everyone using some library I’m missing?
So I’m going to donate my experience in the hope that it saves at least one poor sod from dealing with the tedious, error-prone, and frustrating process of getting a certificate from Apple into a format you can use with Java and Clojure.
The process described here will only work verbatim on a Mac, which is unlikely to be controversial if you’re an iOS development shop. This system was last tested on Mac OS X 10.9/Java 7, so if you have that environment, you’re good to go.
A device driver 'DSL' using core.async
.
One of the killer features of Livespaces has always been its ability to automate control of many disparate hardware devices in our meeting rooms and labs. For example, a Livespace meta app can set up a room for a meeting by setting the level of the room’s lighting, reseting audio mixer levels, powering up the front displays and setting their default inputs, and then loading up a presentation on one display, all the while playing a pleasing background humming noise just to let you know how happy it is to do all this for you.
Livespaces acts like an operating system for a roomful of hardware and software, and, like an operating system, it has a plugin device driver architecture. Livespaces currently has drivers for audio amplifiers, echo cancellers, audio/video switching matrix’s, video cameras, LCD displays, electrical control buses (lights, etc), projectors, fibre switching matrixes, servo actuators, and more.
Which is all very froody, except that these drivers have been written by a variety of developers on an ad hoc basis over nearly a decade now, and I think it’s fair to say the word ‘ghetto’ begins to describe the quality of the code as it stands today.
A few of the interesting mistakes I made in Basis. Part 1: Groundhog Day.
A few days after Shopi’s launch I was checking the server logs, scanning down to get an idea of how things were going. And out of the blue I was in Groundhog Day: the same transactions were being repeated over and over again as I scrolled down. I started paging quickly to see where they ended … and they didn’t. Eventually one cycle ended, then some normal interactions, followed by another seemingly endless cycle. It was at this point I thought to check the log size: half a gigabyte and climbing rapidly.
Now, tales of product launches going horribly awry are old hat these days when everyone’s dog seems to run an online service. I enjoy these war stories as much as the next nerd, and I had thought I knew what it would be like. But let me tell you there’s a special sort of exquisite gut-wrenching dread that you can only experience when something you thought was going great, something you put your heart into, something you had tested literally for over a year, dammit, starts going wrong before your eyes and you have no idea why. All you know is that the poor sods on the other end of it are probably starting to think that maybe they should try something else, you know, something that actually works.
Now, this may not seem related, but bear with me: did you know that floating point numbers are the devil’s work? Floating point numbers kill. They cause accidents costing billions of dollars. And they lie: we can’t even ask them to represent honest numbers like 0.1 without them deceiving us.
Whoops, I built a data replication engine (3/3).
In Part One and Part Two of this epic series, I covered the reasons why I decided to build my own sync service for Shopi, and the git-inspired approach I ended up taking. In this final part I’ll talk about how Basis handles some other tricky bits, and wrap up with some lessons learned.
The scenario: while on the train home from work, Bob opens the shopping list app on his phone to add a few items. His phone gets a terrible signal while on the train, and so it’s not connected to the sync server while he adds the items.
Meanwhile that morning, Bob’s partner Alice has already added several other items. When Bob’s phone finally gets a signal, and the shopping list app checks in with the sync server, the server sees a situation like this:
Whoops, I built a data replication engine (2/3).
In the first installment of this post, I talked about how I came to the depressing realisation that I might need to go out on the bleeding edge to build a sync service for a simple shopping list app. In this part I’ll go into what I did about it.
The core of the problem is that, in an asynchronous, delta-based sync protocol, both the client and server need to track the other’s state in order to be able to compute the smallest delta needed to bring each other up to date.
Framing the problem this way put me in mind of a distributed data replication system that I was already using every day: git. Maybe I could just build on that?
Whoops, I built a data replication engine (1/3).
Hello, my name is Matt and I made a distributed data replication system. I did this in full possession of the knowledge that doing this is something that no sane programmer should ever contemplate, being akin to writing your own crypto in the lexicon of programmer sins. This is my confession.
The system is called Basis, and it exists to provide the sync service behind Shopi, my company’s iPhone shopping list app. This post is the first of an epic series of three that relate the story of how it came to be, why I thought it was a good idea to meddle with things I have no ken of, and how I approached the challenges it presented.
Just over three years ago, around the start of 2011, I decided to wade into the already over-crowded category of iPhone shopping list apps. Entering an app category that was already so extremely commoditised — even as of 2011 — wouldn’t on the face of it seem like a smart plan. But I had personally experienced the mediocrity of the available shopping list apps while trying to come up with a good system for both me and my wife to manage shopping trips. All of the apps I tried lacked what seemed to me to be crucial features.
Building a blogging system is like making your own Jedi lightsaber.
It’s not a new thing for programmers to write their own blogging platform, but it certainly appears appear to have become de rigueur of late, especially those that generate static sites.
In fact, it may have become the programmer-nerd equivalent of a Jedi knight completing his apprenticeship by fashioning his own light saber.
Not to be left out, I’ve gone and done the same thing. My reasons are probably the same as everyone else’s, but here they are anyway: