Code from the sideline

Setting up Couchbase views with Rake

If you are like me, you like your code and at best everything else about your app to be versioned. This makes setting up a new development environment quick, and even more important reproducible. So one thing which always bothered me was setting up external code outside of the application to generate Couchbase views, since those views are build using JavaScript, and by default live in the web interface, far away from my application code. If you want to know more about them watch the Couchbase 103 - Views and Map-Reduce by the way.

Since those JavaScript functions are code which is closely related to my app, I like to keep it inside as well, and provide a task to actually set them up for the given environments. The following quick gist provides a rake task to do this, adding all views in config/couchbase_views to Couchbase.

This works great together with dotenv to provide the connection details via environment variables, but can also depend on any other way to set up the database connection easily.

Installing Couchbase sync_gateway via Homebrew

Homebrew is my favorite package manager on the Mac, sadly there is no package to setup and install the Couchbase sync_gateway which is used for interaction between a Couchbase Server and a Couchbase Lite mobile database. Since this is easily fixed, I created a formula for sync_gateway. Currently the Pull-Request is still open so if you like to have this in homebrew make sure to let the homebrew maintainers know. For the time being you can install it by referencing my fork of homebrew directly in the brew install.

brew install https://raw.github.com/sideshowcoder/homebrew/sync-gateway-add-formula/Library/Formula/sync_gateway.rb

Safari Readinglist to Readability

Ok so I got a Kindle Fire and since there is no way to sync my Safari Readinglist over I decided it is time to move to something “portable”. Luckily it is quite easy to get a list of all the current Safari Bookmarks, which includes everything on the Safari Readinglist, the basics are in a Gist by andphe

So all there is left to do is use the readit gem to get everything into Readability, using oauth to authenticate first. Overall this went pretty well and quickly, so for anybody interested you can find the scripts on github.

Enjoy

Exploring the Erlang NIF with Murmur3: Part 1

Yesterday I read about calling C libraries in Haskell, especially how the Haskell foreign function interface works. Never had to do this in Erlang I decided it would be a nice exercise, so here we go wrapping Peter Scotts Murmur3. The end result is available on github for anybody interested. Also I am not an Erlang so take everything with a grain of salt.

Looking for FFI in Erlang

There are actually multiple ways to call out from Erlang, you can use Ports and actually communicate with an external process via passing messages which is nice and save, or you can implement wrappers in C and use them as NIFs. NIF stands for native implemented function, so the library is actually dynamically linked to Erlang which can be dangerous since an error in the library can crash the Erlang VM, but it is also the fastest way to call out to C land. Since I was interested in the closest thing to FFI really I decided to create a NIF wrapper.

Exploring Murmur3

The C library for the Murmur3 hash function actually consists of 3 functions, 2 optimized for x86 and one for x64, but since they all work on any architecture it made sense to wrap all of them. All functions look really similar.

void MurmurHash3_x86_32 (const void *key, int len, uint32_t seed, void *out);
void MurmurHash3_x86_128 (const void *key, int len, uint32_t seed, void *out);
void MurmurHash3_x64_128 (const void *key, int len, uint32_t seed, void *out);

The first argument is the thing to be hashed, being of length len. The hash is also seeded so passing in the seed value is mandatory. After it is done the result will be placed in out, which is either 32 Bit or 128 Bit in size.

Setting up the project

Using rebar has proved to be the easiest way to setup a project.

$ mkdir murmerl
$ cd murmerl
$ wget http://cloud.github.com/downloads/basho/rebar/rebar && chmod u+x rebar
$ mkdir src
$ touch src/murmerl.app.src
$ touch src/murmerl.erl

murmerl.app.src is really just a really basic app file containing only the following.

Now all we need is a place for storing the C code, and put the Sources in.

$ mkdir c_src
$ cp <Download>/murmur3.c c_src/
$ cp <Download>/murmur3.h c_src/
$ touch c_src/murmur3_nif.c

After telling rebar about the C sources the project can be compiled and run.

$ rebar compile

Which will create priv/murmerl_drv.so as well as the beam and app files in ebin.

NIF in the Erlang world

NIFs behave like any other function from a callers perspective, so we need to define a module for them which is in charge of loading the library as well as provide some fallback if loading fails. Providing fallbacks is done via providing alternative function implementations.

To make it now work we need to load the library via the on_load for the Erlang module and make sure all the functions are exported as well.

Thats it for the Erlang side of things, now for C.

NIF in the C world

Everything needed to bind everything together is present in erl_nif.h which needs to be imported. Each of our functions looks really similar, they return an Erlang Term, and take the current environment, argument count and arguments array. Pulling the arguments apart needs to be done inside the function so this is the first thing.

Erlang provides a lot of functions here we expect something string like as the first argument so we pull the first argument inside the in struct and extract the second as an int. The actual return value will be a binary so we need to allocate one of the right size, 4 to get 4 Bytes = 32 Bit. Since this is just a piece of memory, it is best to zero it out before using. Now all there is to do is let MurmurHash3_x86_32 write to the allocated chunk and return as a Erlang Binary for everybody to use.

To make the functions available to Erlang we need to tell the VM about them, this is done here

Which exports the nif_funcs array containing a structure with

{ name_in_erlang, arity, c_function }

for each of the exported functions. And thats it the functions can now be called from Erlang like any normal Erlang Function in the module. For the full code see the github repository.

Where to go from here?

Currently all the module does is forward the calls as is to the C library, which does not feel very Erlang like, so building a nicer API is up next.

Resources

More examples can be found in davisp github repository, also the Erlang NIF page can be of great help but you need to know what to look for.

Patching Vim via Homebrew to fix YouCompleteMe error messages ‘Back to original’

One of the plugins for vim I rely on heavily is YouCompleteMe. It really is amazing but there is one small problem as it generates completion error messages all the time while typing.

There is a fix for it which sadly has not landed in Vim yet, but as detailed in the issue on github it is easy to patch it yourself. Since I installed vim via Homebrew the way to go was to add the patch to the Homebrew formula and go from there. So this is what I did. First I don’t like to work in the Homebrew master so I created a new branch

$ cd `brew --prefix`
$ git checkout -b add-ycm-message-silence-patch
$ vim Library/Formula/vim.rb

Afterwards it’s time to put the new patch in the formula, at the bottom of the file after __END__. The method

def patches; DATA; end unless build.head?

will apply all patches preset in the DATA section, which is the part of the file after __END__. I created a version in my fork of homebrew.

And low and behold after a reinstall

$ brew uninstall vim
$ brew install vim --override-system-vi \
  --with-client-server --with-lua \
  --with-mzscheme --with-perl --with-tcl

The annoying errors are gone.

Enjoy.

New version of Todo.txt for Vim

It’s time to make stuff better, today I got around to ease the use of Todo.txt in Vim.

  • made the it respect the fact that todotxt fileformat does not like linebreaks
  • made it aware of :TodoDone as well as :TodoArchive commands
  • removed some weired functions around activating and deactivating which were really just a fix for my own setup which was screwed up

Enjoy and download at Github see the Readme for further information

README Driven Development: Testing libraries from a user perspective

What really is outside in testing when writing a library? I have seen different approaches to this problem like cucumber, rspec or bacon, but one step that seems to be missing in a lot of cases is testing what the user sees first, the README. On Github the README file has really become the first thing any developer coming in touch with a library looks at, so it should be the getting started guide, and something that can make or break the experience a developer has. But many developers including myself seem to face a problem so keeping the documentation working and updated. This got me thinking what I really want in there are code samples, and those code samples should work copy and paste.

To achieve this the README has to be part of my build process, so I started experimenting with including a runner for code inside as a Rake task

The Idea really is nothing new, but I have been struggling to find a well integrated solution. This is probably the simplest solution I could come up with. I’m currently driving the development of a library idea for my talk at takeoffconf, and it allows me to really focus on the concept of the code from a user perspective writing the README first and building the working code from there on.

Currently the solution is not robust so any help is greatly appreciated, even though for now it works.