Testing shell scripts

Shell scripts serve an import role in a lot of cases, sadly mostly they are also the mother of all legacy pre. Don’t get me wrong, those one-off scripts are important and great, but after they serve there purpose there are really 2 options,ensure that they are well maintained, or get rid of them. Just leaving them around will just make them fight you later on.

This to me means that shell scripts need to be tested like any other software, and even though it does not seem obvious, testing shell scripts is quite easy. My personal tool of choice is assert.sh, which is simple enough and integrates nicely into a testing workflow.

Just a quick example on how this works, the script to test is example.sh
https://gist.github.com/sideshowcoder/7172327.js?file=example.sh

the tests go in example.test.sh
https://gist.github.com/sideshowcoder/7172327.js?file=example.test.sh

and finally running everything is as easy as
https://gist.github.com/sideshowcoder/7172327.js?file=run

And that’s it, a simple test harness around shell scripts.

Setting up RiakCS from source

In preparation for my talk at the
Munich Rubyshift about Riak I decided that one
topic I will talk about should be
RiakCS, since running S3 locally is just
so useful, be it for development purposes or production moving away from S3
towards a self hosted service. Setup can be really simple, just clone
the vagrant setup and run

    $ vagrant up

and be done with it.

But I still think to really explore something, I need to run it from source. Ok
so here we go, I did this on my Macbook with Erlang R15B01 installed via
kerl.

Get the source, and build it

Ok this is easy, just clone the repos needed

    $ git clone https://github.com/basho/riak.git
    $ git clone https://github.com/basho/riak_cs.git
    $ git clone https://github.com/basho/stanchion.git

Ok now to build it, since I am running erlang via kerl, I first need to make
sure the right version is activated

    $ source /path/to/erlang/activate

Now build all three projects

    $ cd riak && make rel
    $ cd riak_cs && make rel
    $ cd stanchion && make rel

This might take some time, since all dependencies need to be downloaded and
compiled as well but rebar (the Erlang build tool) will take care of this.

Configuration

For this I closely followed
this guide
the only thing to do really is to ajust the paths in the config files as needed,
so for /path/to/riak/rel/riak/etc/riak/app.config I followed the guide in step
3 but adjusted the paths to look like this

    {add_paths, ["/path/to/riak_cs/rel/riak-cs/lib/riak_cs-1.4.0/ebin"]},
    {storage_backend, riak_cs_kv_multi_backend},
    {multi_backend_prefix_list, [{>, be_blocks}]},
    {multi_backend_default, be_default},
    {multi_backend, [
        {be_default, riak_kv_eleveldb_backend, [
            {max_open_files, 50},
            {data_root, "/path/to/riak/rel/riak/data/leveldb"}
        ]},
        {be_blocks, riak_kv_bitcask_backend, [
            {data_root, "/path/to/riak/rel/riak/data/bitcask"}
        ]}
    ]},

For the IPs to listen to I decided to use 0.0.0.0 which binds it to every interface,
just to make sure I don’t run into trouble there. The guide suggest setting it to
the external interface IP which is probably the wiser choice.

Otherwise all settings apply here as given by the guide.

Startup

Ok with this little change everything is ready, and we can start each service

    $ /path/to/riak/rel/riak/bin/riak start
    $ /path/to/stanchion/rel/stanchion/bin/stanchion start
    $ /path/to/riak_cs/rel/riak-cs/bin/riak-cs start

order is important so.

Next Steps

Now it’s back to
this guide
and create a user and have fun with your own local S3.

Interested in more things Riak, and located in Munich?
Checkout Riak München

Setting up Riak with Vagrant and Puppet

Setting up Riak is pretty easy and straight forward,
but depending on my needs there is the need to change configuration
depending on the current use case. This means that having one instance of it on
my development machine just doesn’t cut it. Luckily there is
Vagrant which provides a quick way to setup VMs as
needed for a given project, along with Puppet this allows me to bundle all the
configuration for a given project.

Basho provides a nice module to setup Riak with puppet, but is pretty complex
to handle all the different use cases. Since I just need it for development I
decided to make it easier for me and other devs by providing an easy to setup and
a quickly understandable module for my needs. It just installs the most recent
version of Riak from the debian-pkg provided for Ubuntu 12.04 LTS 64Bits. So here we go

Vagrant configuration is short and simple, puppet folder setup as well.

https://gist.github.com/sideshowcoder/6439548.js

while for riak_precise64 I just add a submodule via git

$ git submodule add
https://github.com/sideshowcoder/riak_precise64.git
./puppet/modules/riak_precise64

and here we go a simple

$ vagrant up

will give us a nice VM with riak ready to use, and any config nicely contained in app.config.

Rails Quick Tip: Set a secret token for Heroku deploy

Rails apps use a secret token to keep user information save, so the more entropy the better. This means using the same secret token on heroku and locally is not ideal, and quite easy to fix.

First make the secret_token.rb set the token from an ENV variable

MyApp::Application.config.secret_key_base = ENV['SECRET_KEY_BASE'] || "sometoken"

And now create the token in the heroku config

heroku config:set SECRET_KEY_BASE=`rake secret`

That’s it, enjoy!

New Version of canned, with custom return code support

Canned is now 0.2.0, this has been the start of a major internal refactor, so it is no longer just one big ball of JavaScript but starting to have some internal structure.
A major new feature is the support for specifying the return code inside the file, so special codes like 201 for create etc. can now simply be added via //! statusCode: 201.
For anybody who likes to help the tests got some love as well and are now in more friendly jasmine and hopefully easier to understand than before.

Enjoy!

Vagrant by example: Setting up Phantomjs

Vagrant is an amazing tool, especially switching between projects it comes in
handy to keep the dependencies separated. This works even better with puppet to
provision the machines in a reproducible fashion.

Installing vagrant

First things first, we need to install vagrant,
which has changed some time ago from being a gem to provide
installer packages. Installation now is really
simple, just download and install.

Setting up vagrant

To get of the ground lets setup an example box.

$ mkdir myproject && cd myproject
$ vagrant init precise32 http://files.vagrantup.com/precise32.box

This will download, and setup a basic box with ubuntu precise32.

Configure puppet

Inside the Vagrantfile we now configure vagrant to use
puppet, to provision the box for us. This will make
it easy to spin up a similar box whenever we need a new one.

http://gist-it.appspot.com/github/sideshowcoder/puppet-vagrant-by-example/blob/master/Vagrantfile?slice=55:76

Personally I prefer to modify the default vagrant behavior a bit and keep
everything puppet under puppet/* inside the project.

Setting up phantomjs

Now to the meat of this example, setting up phantomjs from the tarball download

Netinstall

Luckily the good guys from example42 provide a nice
puppet module to handle just these installs, it comes packaged with
puppi. Right now puppi seems a little much
such we just extract the netinstall code inside a module of its own and set it
up
inside the repo
Netinstall is a quite simple module, the only dependency we need to worry about
is the uri_parser which is implemented as a ruby extension and located in lib.
I also decided to rename netinstall.pp to init.pp since it is autoloaded
this way, and removed the puppi prefix.

http://gist-it.appspot.com/github/sideshowcoder/puppet-vagrant-by-example/blob/master/puppet/modules/netinstall/manifests/init.pp

Setting up the init.pp

Under manifests we can now create the custom init.pp in which we install
phantomjs and libfontconfig1 on which it depends.

http://gist-it.appspot.com/github/sideshowcoder/puppet-vagrant-by-example/blob/master/puppet/manifests/init.pp

Vagrant up

Now a simple

$ vagrant up

Will provision the machine with phantomjs installed, which we can test via after
provisioning is done.

$ vagrant ssh
$ phantomjs

Missed something?

Installing new packages is now easy as adding them to the init.pp file and
running

$ vagrant provision

Bundle all the dependencies!

Make sure your rails migrations work, thank you.

Rails migrations are the ugly stepchild of most projects, even though
actually making them work correctly saves so much time. Even though they seem
like one-off scripts, and also are most of the time when you do need to run them
more than once you are already in a bad place, so be sure that the migration you
wrote, when there was time is not making it worse.

At least doing a quick testing of the rollback is worth your time, in all cases!
So just run

$ rake db:migrate && rake db:rollback && rake db:migrate

When creating a new migration, thank you, you are awesome :).

Setting up ProXPN with Viscosity

ProXPN, is nice OpenVPN service which is free for basic
use, but also worth paying for in case you need the features. When setting it up
on my Macbook yesterday I first thought I would have to use their client, which
would have been a deal-breaker for me. I get why services want you to use a
branded client which they control, but when you already have a client, which
should work and need to install another it’s kind of weird.

So here we go I have Viscosity installed
and want to use it with ProXPN. A quick look at the source
download
will show the needed certs and
configuration. So just download
proXPN / Tunnelblick Source,
extract and check the config.

ProXPN config

The config is almost ready to be imported into Viscosity, the only thing to do
is to uncomment the remote server, just open the config in a text editor and fix
this.

Uncomment Remote Server

Now it’s time to import the file in Viscosity and it’s done. The import will
copy all needed certs and keys to the right folder to be used.

Import from File

Ready to use ProXPN now, enjoy the privacy ;)