Clojure + Ring + API Gateway + Lambda

Serverless is all the rage right now, so I’ve been playing with AWS Lambda, and just by the name itself what could possibly be a better fit for lambda than using Clojure on it? Since I’m mostly interested in webapps I decied to give API Gateway a spin with lambda, sadly it does not immedialty work with the awesome Ring library for building web applications in clojure. Luckily it is reasonably easy to map between the two, and inspired by ring-aws-lambda-adapter some time later I now have a working version of Ring + API Gateway. Together with AWS SAM and AWS SAM CLI this makes for quite a nice, local or cloud, development and hacking experience. And all it took is about 90 lines of Clojure

(ns playground.core
  (:require [ :as json]
            [ :as io]
            [clojure.string :as s]
            [ring.util.codec :as codec]
            [ring.util.response :as r])
   :name playground.core.Handler
   :implements [])
  (:import [ RequestStreamHandler]
           [ InputStream File]
           [clojure.lang ISeq]))

(defn sample-handler
  (r/response "Hello Lambda, from ring!"))

(defn stream->event
  (json/read (io/reader in) :key-fn keyword))

(defn str->int
    (Integer/parseInt s)
    (catch NumberFormatException _)))

(defn maybe-remote-address
  "Take a guess at the remote client ip based on the X-FORWARDED-FOR header.
  This is not perfect as the X-FORWARDED-FOR can easily be spoofed. "
  (some-> x-forwarded-for
          (s/split #"," 2)

(defn maybe-http-version
  "Take http version frim VIA header if present"
  (some-> via
          (s/split #" " 2)

(defn api-gw-event->ring-request
  ([ev] (api-gw-event->ring-request ev nil))
  ([ev ctx]
   (let [headers (get ev :headers {})]
     {:server-port (str->int (:X-Forwarded-Port headers))
      :body (:body ev)
      :server-name (:Host headers)
      :remote-addr (maybe-remote-address (:X-Forwarded-For headers))
      :uri (:path ev)
      :query-string (codec/form-encode (get ev :queryStringParameters {}))
      :scheme (maybe-http-version (:Via headers))
      :request-method (:httpMethod ev)
      :protocol (:X-Forwarded-Proto headers)
      :headers headers
      :event ev
      :context ctx})))

(defmulti wrap-body class)
(defmethod wrap-body String [body] body)
(defmethod wrap-body ISeq [body] (s/join body))
(defmethod wrap-body File [body] (slurp body))
(defmethod wrap-body InputStream [body] (slurp body))

(defn ring-response->api-gw-response
  {:statuCode (:status response)
   :headers (:headers response)
   :body (wrap-body (:body response))})

(defn handle-request
  "Handle an api-gateway request from IN via a RING-HANDLER writing
  response to OUT."
  [ring-handler in out ctx]
  (let [event (stream->event in)
        request (api-gw-event->ring-request event ctx)
        response (ring-handler request)
        event-response (ring-response->api-gw-response response)]
    (with-open [w (io/writer out)]
      (json/write event-response w))))

(defn -handleRequest
  "Implementation for RequestStreamHandler handleRequest using IN
  instream OUT outstream and CTX context object, delegating to
  handle-request to handle request using ring"
  [_ in out ctx]
(handle-request sample-handler in out ctx))

All I want for XMas … is a Brainfuck interpreter in Clojure?

Over the holidays I felt it was time to play with a little Clojure, and since I recently became interested in compilers and interpreters I felt like it was a good idea to start extremly simple, it is the holidays after all. So I built the simplest possible interpreter, a Brainfuck interpreter. It is quite a need excercise and most certainly made me more interesting in building more complex ones.

So here it goes clojure-brainfuck for anyone interested.

Projectile and TRAMP

Projectile requires a lot of file system interaction to stay up to date, this does not work well when accessing remote files and causes massive slowdowns when used over a less than ideal SSH connection. My solution based on a comment regarding this issue has been to deactivate projectile on remote files, sadly its not just files but also dired buffers which suffer so a small addition is needed to the solution in the comment adding dired buffers to the list of things dired should not be enabled for.

(defadvice projectile-on (around exlude-tramp activate)
  "This should disable projectile when visiting a remote file"
  (unless  (--any? (and it (file-remote-p it))

furthermore to avoid unneeded calculation of the current project the projectile modeline needs to be set static.

(setq projectile-mode-line "Projectile")

with this in place projectile works nicely locally and remote file interactions are as quick as they can be.

First class functions over channels


Inspired by reading go-cache and reading about core.async I was interested how well sending functions over a channel can work, here is the quick hacky result.


Clojure with its access to a very concise way to construct functions lends itself very nicely to the idea of using CSP with first class functions as messages passed over a channel.

(let [c (chan)]
  (>!! c #(zero? %)))

Given that a channel is only consumed by one thread, this provides an interesting way to sync on a data structure without the need for explicit locking as the channel with by itself serialize the operations. This could be quite performant depending on the consumer as multiple operations could be pipelined for example by polling the channel for a batch of operations to be dispatched to different workers.

Why not?

This is mainly an idea taken from playing with go-lang and implemented in clojure, for access to a basic data structure like a map, even so it can now be a transient map, the built in transactions will provide much better performance.

Installing golang-1.7 on ubuntu 16.10

Ubuntu, and in fact any debian as far as I’m aware has the the

$ update-alternatives

command to manage different versions of installed software. Currently when installing golang ubuntu will default to 1.6, but the repository contains 1.7 as well as package golang-1.7. To allow switching between the versions with the builtin system it can be setup via

$ update-alternatives --install /usr/bin/go go /usr/lib/go-1.7/bin/go 1

Where 1 is the priority, when using auto setup.

$ update-alternatives --config go

Now allows switching between the favorite go version.

GoGuru scope in emacs

GoGuru is an amazing tool and can be really useful when exploring a project, setting up the scope can be a bit of a pain so as the default function doesn’t even support autocompleting. After some quick hacking here is the first attempt to make at least allow for some easier setup, taking the current file and assuming it is the main package, this little function will setup the scope relative to this file.

Building hack-lang on Ubuntu 16.10

Having hacked with PHP every now and then, and running my blog on wordpress I wanted to give hack-lang a spin. Obviously wanting to be bleeding edge I decided to build hhvm from source on my Ubuntu 16.10 machine. It almost worked straight out the box but some poking was needed as I first encountered an error on the build following the instructions on the page. Here is what got me past the initial errors.

Install the packages documented on the page, but using gcc-4.9

$ sudo apt install autoconf automake binutils-dev build-essential cmake g++-4.9 gcc-4.9 gawk git 
  libboost-dev libboost-filesystem-dev libboost-program-options-dev libboost-regex-dev 
  libboost-system-dev libboost-thread-dev libboost-context-dev libbz2-dev libc-client-dev libldap2-dev 
  libc-client2007e-dev libcap-dev libcurl4-openssl-dev libdwarf-dev libelf-dev 
  libexpat-dev libgd2-xpm-dev libgoogle-glog-dev libgoogle-perftools-dev libicu-dev 
  libjemalloc-dev libmcrypt-dev libmemcached-dev libmysqlclient-dev libncurses-dev 
  libonig-dev libpcre3-dev libreadline-dev libtbb-dev libtool libxml2-dev zlib1g-dev 
  libevent-dev libmagickwand-dev libinotifytools0-dev libiconv-hook-dev libedit-dev 
  libiberty-dev libxslt1-dev ocaml-native-compilers libsqlite3-dev libyaml-dev libgmp3-dev 
  gperf libkrb5-dev libnotify-dev libpq-dev

Checkout the hhvm source from github

$ git clone git:// --depth=1
$ cd hhvm
$ git submodule update --init --recursive

I updated the submodules and they are now at the following versions for me

$ git submodule status --recursive
 af252b08f715a0e348175d0754d533949f2e390c third-party (af252b0)
 98ed7a23a83d64133b0a36a884e489bffb0eb864 third-party/brotli/src (v0.3.0)
 8bb673f4410819df06920fdcfd24e18d235d84f7 third-party/brotli/src/terryfy (third-15-g8bb673f)
 f1b955eee1cb94c51074878264aecdd6ee1350dd third-party/double-conversion (v1.1-86-gf1b955e)
 f1217348a868bdb9ee0730244475aee05ab329c5 third-party/fastlz/src (heads/master)
 e21df038e0b58855ef9344d1a3a4389e03ba2cc7 third-party/fatal (0.2-573-ge21df03)
 1d2d4f326acc0825690c151c38ac92d146b78146 third-party/folly/src (v2016.11.07.00-21-g1d2d4f3)
 eb021a100e761616b35ad62c910d6c336e0711d2 third-party/libafdt/src (heads/master)
 1d8b1ac4d20b8ef8d3f5d496dabebaa0ff9019ff third-party/libzip/src (rel-1-1-3-3-g1d8b1ac)
 d86dc916771c126afb797637dda9f6421c0cb998 third-party/lz4/src (r131)
 dbf8878c0136e5f94e4bb0aef6f931d5aecfa74d third-party/mcrouter/src (v0.34.0-2-gdbf8878)
 931a3231375201ee58fb495f4fad0d8fdfc36251 third-party/proxygen/src (v0.32.0-410-g931a323)
 718df09610fee584c9038d8d519697e507e09c9b third-party/re2/src (2016-07-01-6-g718df09)
 8468f07fcc8d96b4af69f1c8b75d404458f8e6b4 third-party/squangle/src (8468f07)
 5e5773be567eeb53a75565a911ece7de2c89ee2b third-party/thrift/src (v2016.11.07.00-8-g5e5773b)
 34a04f01668ae88f0dd7ca18aedf50e41689e8f9 third-party/wangle/src (v2016.11.21.00)
 a9e580b5a0baa768210ef10544c8fab52003ec0b third-party/webscalesqlclient/mysql-5.6 (a9e580b)
 dcc898b0215cee3b1baa88149c1f39e37e9bfd09 third-party/webscalesqlclient/mysql-5.6/rocksdb (rocksdb-3.13-431-gdcc898b)

with those versions building with gcc-4.9 seems to work fine

$ cmake -DCMAKE_C_COMPILER=/usr/bin/gcc-4.9 
        -DMYSQL_UNIX_SOCK_ADDR=/var/run/mysqld/mysqld.sock .
$ make -j <number of cores for me 4>

Happy hack-ing!

Zeus console in Emacs

Zeus is quite a useful tool when working with larger rails applications, or other larger ruby applications for that matter. Especially as it reduces the time to open a interactive environment for the project and makes the whole development process much more fluid. Combining it with Emacs power to send pieces of code directly from a buffer to be executed in a ruby process this is quite a nice way to explore and work. To quickly get access to a Zeus console simply drop the following in your Emacs configuration.

(add-to-list 'inf-ruby-implementations
             `("zeus console" . "zeus console"))

Maybe my suggestion to add this to inf-ruby will be added, but even if not it’s simple enough to get a console with this via C-u M-x inf-ruby.

Hacking on rails using emacs

Rails contributor?

Apparently I committed to rails before, I’m pretty sure it was a doc update, but none the less this is something I actually like to do more. So first step get a working setup for running rails tests locally and getting a reasonably productive environment setup. Since I’m mostly using emacs these days it would be nice if everything would actually work out of this editor/operating system.

rake test

The easiest way to run the tests after cloning down rails and bundling is to use rake.

$ bundle exec rake test

in the root rails folder will run all the available tests, this is great but when working on something also not what I want most of the time. But running it in a subfolder like activesupport will actually do the right thing and run only this subset of tests

$ cd activesupport 
$ bundle exec rake test

To make this easier I’m using textmate.el to execute most commands while inside a project from the determined project-root. Meaning if I switch to a file I set the working directory to the current root automatically.

(defun coder/default-project-dir ()
  (if (textmate-project-root)
      (setq default-directory (textmate-project-root))))

(add-hook 'find-file-hook 'coder/default-project-dir)

Adding .emacs-project files to all the subdirectories will now make sure that when I run a rake command it will be run in the correct directory. Combining this with rake.el allows to run test via a simple M-x rake and selecting test, M-x rake-rerun will now rerun the given tests with a simple command.

This works great while working in quick running test suites but running the whole suite on every change can become inconvenient quickly, this is solved by 2 more enhancements to the flow

running a single test file

Rake allows for passing a file name to limit the run to a single test file

$ bundle exec rake test TEST="test/abstract/callbacks_test.rb"

for example will run only this file of the actionpack testsuite for example. Make sure to run with bundle exec, rake.el does not detect this when in a subfolder, but C-u M-x rake will let you modify the command as needed.

running a single test

Should a file have a lot of tests it can be a good idea to only run a single one, this is possible by passing an option to the minitest testrunner.

$ bundle exec rake test TEST="test/abstract/callbacks_test.rb" TESTOPTS="--name=test_around_action_works"

TESTOPTS will be passed through to minitest which will only run this one test.

integration in emacs

All this works on the command line, as well as using rake.el, personally I use rake.el and have rake rerun bound to C-c r to allow me to quickly run the tests I’m currently working on, this executes in compilation-mode, which allows emacs to parse out file urls and makes them clickable in case of errors.

Next up to make it even better would be to figure out the test name I’m currently in, this works quite simply for files which use the

class TestClass < Minitest::Testcase
  def test_foo
    assert_true doing_the_foo

syntax, but is much harder with files

class TestClass < Minitest::Testcase
  test "doing the foo" do
    assert_true doing_the_foo

So for now I stick with doing it by hand.

Some F# fun

I’m making my way through the The book of F# and been playing with implementing the RPN calculator

personally I found it quite interesting to see my choices during this small exercise, especially the use of Option types to handle error cases comes quite naturally. Another interesting thing I found was that, similar to how I would write the same thing in Lisp, it is very obvious to define functions inside other functions to hide implementation details I would like to give a name to but don’t want to share with the rest of the program.