I recently wrote a simple service that is essentially an HTTP proxy for S3-based storage.

The existing server that hosts this is based on Netty, which is framework for high-performance asynchronous network services, and it already uses Netty’s HTTP I/O support in several places. So it seemed obvious that the new service would use Netty’s HTTP support to receive HTTP requests from a client and talk to S3. However, I hadn’t banked on Amazon’s authentication mechanism.

At first glance Amazon’s AWS4 header-based authentication signatures looked alarmingly complex, and I was sorely tempted to run into the arms of Amazon’s Java AWS client library. However, the fact that Amazon’s library would live entirely outside of the Netty I/O framework we’re using gave me pause.1

So, I embarked on implementing AWS4 HTTP header-based authentication signatures in Clojure. Luckily Amazon’s documentation is superb, going into great detail with HTTP request examples, code examples and even a suite of test data. It’s one the perks of using AWS.

So here, without further ado is the code. Treat this as licensed like Creative Commons By Attribution, i.e. if you use it please add a comment that you got it from here. There is also no warranty, etc, etc.

You can download the code here, and I’ve added some commentary below.

The first two functions generate HTTP GET and PUT requests for an item in an S3 bucket. These are Netty-specific, and generate a Netty HttpRequest or HttpResponse respectively.

(defn s3-bucket-get-request [url bucket region access-key-id secret-key]
  (let [uri (str "/" url)
        headers {"Content-Length" "0"
                 "Host" bucket
                 "x-amz-content-sha256" EMPTY_SHA256
                 "x-amz-date" (.format iso8601-date-format (Date.))}
        request (DefaultFullHttpRequest. HttpVersion/HTTP_1_1 HttpMethod/GET uri)
        request-headers (.headers request)]
    (doseq [[k v] headers]
      (.set request-headers ^String k ^String v))
    (.set request-headers "Authorization"
      (aws4-authorisation "GET" uri headers region "s3" access-key-id secret-key))
    request))

(defn s3-bucket-put-request [url content-sha256 content-length mime-type bucket
                             region access-key-id secret-key]
  (let [uri (str "/" url)
        headers {"Host" bucket
                 "Content-Length" (str content-length)
                 "Content-Type" mime-type
                 "x-amz-content-sha256" content-sha256
                 "x-amz-date" (.format iso8601-date-format (Date.))}
        request (DefaultHttpRequest. HttpVersion/HTTP_1_1 HttpMethod/PUT uri)
        request-headers (.headers request)]
    (doseq [[k v] headers]
      (.set request-headers ^String k ^String v))
    (.set request-headers "Authorization"
      (aws4-authorisation "PUT" uri headers region "s3" access-key-id secret-key))
    request))

In both of these, the signature needed to coax S3 into talking to us is generated by aws4-authorisation. This function is generic, taking the HTTP request information (method, URI, and headers), the AWS region and service, and AWS access credentials, and generating the corresponding cryptographic signature.

(defn aws4-authorisation [method uri headers region service access-key-id secret-key]
  (let [canonical-headers (aws4-auth-canonical-headers headers)
        timestamp (get canonical-headers "x-amz-date")
        short-timestamp (.substring ^String timestamp 0 8)
        string-to-sign (str
                         "AWS4-HMAC-SHA256\n"
                         timestamp "\n"
                         short-timestamp "/" region "/" service "/aws4_request" "\n"
                         (sha-256 (to-utf8 (aws4-auth-canonical-request method uri
                                             canonical-headers))))
        signing-key (-> (hmac-256 (to-utf8 (str "AWS4" secret-key)) short-timestamp)
                        (hmac-256 region)
                        (hmac-256 service)
                        (hmac-256 "aws4_request"))
        signature (hmac-256 signing-key string-to-sign)]
    (str
      "AWS4-HMAC-SHA256 "
      "Credential=" access-key-id "/" short-timestamp "/" region "/" service
      "/aws4_request, "
      "SignedHeaders=" (str/join ";" (keys canonical-headers)) ", "
      "Signature=" (as-hex-str signature))))

(defn aws4-auth-canonical-request [method uri canonical-headers]
  (str
    method \newline
    uri    \newline
    \newline                          ; query string
    (stringify-headers canonical-headers)   \newline
    (str/join ";" (keys canonical-headers)) \newline
    (get canonical-headers "x-amz-content-sha256" EMPTY_SHA256)))

(defn aws4-auth-canonical-headers [headers]
  (into (sorted-map)
    (map (fn [[k v]] [(str/lower-case k) (str/trim v)]) headers)))

(defn stringify-headers [headers]
  (let [s (StringBuilder.)]
    (doseq [[k v] headers]
      (doto s
        (.append k) (.append ":") (.append v) (.append "\n")))
    (.toString s)))

(defn ^bytes to-utf8 [s]
  (.getBytes (str s) "utf-8"))

(defn ^String sha-256 [bs]
  (let [sha (MessageDigest/getInstance "SHA-256")]
    (.update sha ^bytes bs)
    (as-hex-str (.digest sha))))

(defn hmac-256 [secret-key s]
  (let [mac (Mac/getInstance "HmacSHA256")]
    (.init mac (SecretKeySpec. secret-key "HmacSHA256"))
    (.doFinal mac (to-utf8 s))))

One thing to note is that I did not write logic needed if the URL has parameters (e.g. the bits after ‘?’ in http://example.com/thing?this=that&foo=bar). These need to be included in the signature, but since I don’t use parameters I didn’t take the time to implement that.

Enjoy!

1. And indeed, later on I was proven somewhat prescient when I needed to add back-pressure between client and S3 connections, which was easy between two Netty channels. Also, the entire Clojure implementation ended up at ~500 lines of code, which is miniscule compared to pulling Amazon’s Java library. ↩︎