Wednesday, April 30, 2014

Restoring Ruby 2.0 on Ubuntu 14.04

For some reason, Ubuntu 14.04 sets up MRI 1.9.1 as the default Ruby (/usr/bin/ruby and friends) version, even if Ruby was installed using the ruby2.0 package. This post outlines a quick and dirty method for restoring Ruby 2.0 as the default Ruby version.


Issue the following commands in a shell.

sudo rm /usr/bin/ruby /usr/bin/gem /usr/bin/irb /usr/bin/rdoc /usr/bin/erb
sudo ln -s /usr/bin/ruby2.0 /usr/bin/ruby
sudo ln -s /usr/bin/gem2.0 /usr/bin/gem
sudo ln -s /usr/bin/irb2.0 /usr/bin/irb
sudo ln -s /usr/bin/rdoc2.0 /usr/bin/rdoc

sudo ln -s /usr/bin/erb2.0 /usr/bin/erb
sudo gem update --system
sudo gem pristine --all


The underlying issue is that the ruby2.0 package in Ubuntu 14.04 depends on the ruby package, which sets up Ruby 1.9.1. The ruby package file list shows that it contains the /usr/bin symlinks used to run ruby and manual pages. The fact that the files are architecture-independent is a pretty good hint that they're not binaries, and merely point to other files. The ruby2.0 package file list (for amd64) shows the binaries that the symlinks need to point to for Ruby 2.0 to become the default version on the system.

The commands above restore the symlinks in the ruby package and rebuild all the gems, to ensure that their /usr/bin stubs are set up correctly as well. Regenerating the gems is a good idea after a system upgrade anyway, because this causes extensions to be re-compiled against the updated libraries. 


The ruby packaging in Ubuntu 14.04 has a nasty surprise for upgraders who use ruby2.0 as the default Ruby interpreter. This post provides a quick workaround for the problem. I hope you found this useful, and look forward to your comments and feedback!

Sunday, March 31, 2013

Setting Up munin on Ubuntu Server 13.04 with nginx

This post explains how to configure munin on a single Ubuntu 13.04 server that uses nginx.

Monitoring is supposed to save time by debugging and predicting / avoiding catastrophes. However, setting up munin on Ubuntu was a time-consuming trial-and-error process for me. The official instructions and various blog posts that cover this topic skip important steps, such as having monit's FastCGI processes start automatically at boot time. I have documented the setup that worked for me, hoping that others can reuse my work.

Ubuntu packages

Run the following command to install the packages needed for munin.
sudo apt-get install munin munin-node spawn-fcgi libcgi-fast-perl

The following sections configure the munin packages.

munin configuration

Write the munin configuration below to /etc/munin/munin-conf.d/90-fcgi

graph_strategy cgi
html_strategy cgi
cgiurl_graph /munin/munin-cgi-graph

nginx configuration

Write the nginx configuration below to /etc/nginx/sites-enabled/munin.conf

server {
  listen 443 ssl;
  listen 80;
  charset utf-8;

  location ~ ^/munin/munin-cgi-graph/ {
    fastcgi_split_path_info ^(/munin/munin-cgi-graph)(.*);
    fastcgi_param PATH_INFO $fastcgi_path_info;
    fastcgi_pass unix:/var/run/munin/fastcgi-graph.sock;
    include fastcgi_params;

  location /munin/static/ {
    alias /etc/munin/static/;
    expires modified +1w;

  location /munin/ {
    fastcgi_split_path_info ^(/munin)(.*);
    fastcgi_param PATH_INFO $fastcgi_path_info;
    fastcgi_pass unix:/var/run/munin/fastcgi-html.sock;
    include fastcgi_params;

  location / {
    rewrite ^/$ munin/ redirect; break;

This configuration assumes that you have a DNS entry set aside for reaching the monit pages. I have separate DNS entries for all my applications, and they're all CNAMEs for the (same) machine that they're running on.

Once you're done tweaking the script, reload nginx.
sudo /etc/init.d/nginx reload

FastCGI daemons

Write the script below to /etc/init.d/munin-fcgi


# Provides:          munin-fcgi
# Required-Start:    $remote_fs $syslog $network
# Required-Stop:     $remote_fs $syslog $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start munin FCGI processes at boot time
# Description:       Start the FCGI processes behind http://munin.*/

# Ubuntu 12.10: /usr/lib/cgi-bin/munin-cgi-graph
# Ubuntu 12.10: /usr/lib/cgi-bin/munin-cgi-html


. /lib/lsb/init-functions

start() {
  echo -n "Starting munin graph FastCGI: "
  start_daemon -p ${graph_pidfile} /usr/bin/spawn-fcgi -u munin -g munin \
      -s /var/run/munin/fastcgi-graph.sock -U www-data ${graph_cgi}
  echo -n "Starting munin html FastCGI: "
  start_daemon -p ${html_pidfile} /usr/bin/spawn-fcgi -u munin -g munin \
      -s /var/run/munin/fastcgi-html.sock -U www-data ${html_cgi}
stop() {
  echo -n "Stopping munin graph FastCGI: "
  killproc -p ${graph_pidfile} ${graph_cgi} -QUIT
  echo -n "Stopping munin html FastCGI: "
  killproc -p ${html_pidfile} ${html_cgi} -QUIT

case "$1" in
    echo "Usage: munin-fcgi {start|stop|restart}"
    exit 1
exit $retval

Make the script executable, and fix some permissions while you're at it.

sudo chmod +x /etc/init.d/munin-fcgi
sudo chown munin /var/log/munin/munin-cgi-*

Now that the init.d script is in place, start it and have it run on every boot.

sudo /etc/init.d/munin-fcgi start
sudo update-rc.d munin-fcgi defaults


You should be able to point your browser to and see many graphs. If that doesn't work out, the logs below should give you a clue as to what went wrong.

  • /var/log/nginx/error.log
  • /var/log/munin/munin-cgi-graph.log
  • /var/log/munin/munin-cgi-html.log


I hope that you have found my post useful, and I hope that it helped you get your munin monitoring setup up and running in a manner of minutes. I look forward to your comments and feedback!

Sunday, December 16, 2012

Free SSL for Your Web Application

This post documents my experience adding SSL support to my Web applications. Most of the steps here are applicable even if you choose a different SSL certificate provider and/or if you host your Web application on Heroku or Google App Engine.

Sign up with StartSSL

StartSSL offers free personal SSL certificates. These are not 30-day trials. The certificates will show your name, not a company name, but do give you all the technical benefits of SSL, such as encryption, the ability to run a SPDY server, and the ability to host background pages in Chrome.

The Web UI leaves a lot to be desired, but their pricing can't be beat. You need to perform the following steps.

  • Create an account
  • Verify your e-mail address
  • Get a client certificate
  • Export your client certificate somewhere safe
  • Verify your domain
  • Generate a CSR (details below)
  • Obtain a SSL certificate for your server
  • Build a SSL certificate chain (details below)

Generate a CSR

Most online tutorials have you generate a password-protected RSA key, which cannot be used automatically. Most deployment tools, as well as Heroku and Google Apps, require un-encrypted RSA keys behind your certificates. Use the command below to generate a CSR with an un-encrypted key.

openssl req -new -newkey rsa:2048 -sha256 -nodes -keyout server_name.pem -out server_name.csr -batch

Replace server_name in the command with your server's name. If you want to generate CSRs for other providers, leave out the -batch at the end of the command and carefully answer the openssl prompts.

At the time of this writing, the StartSSL Web UI expects the CSR in a textarea input field, so open server_name.csr and copy-paste its content into the provided textarea.

After the CSR is provided to StartSSL, the .csr file can be deleted. However, hang on to the .pem file storing your server's private key!

Build a certificate chain

StartSSL's Web UI currently provides the server certificate in a textarea on a page that also points to the relevant CA certificates. However, most server software expects all the certificates to be bundled in a single file. Follow these steps to put together the certificate bundle.

First, open cert.cer in a text editor and copy-paste the StartSSL certificate text from the textarea.

gedit server_name.cer

Download StartSSL's root CA and intermediate CA certificates.

curl > ca.pem
curl > ca2.pem

Last, put together the certificate bundle.

cat server_name.cer ca2.pem ca.pem > server_name.crt
rm server_name.cer ca.pem ca2.pem

For consistency, replace server_name in the command above with the same name that you used for the .pem file.

Set up your server for SSL

If you use nginx as your front-end server, you're in luck. Merge the bits below into your configuration file, and your application should support SSL without any further changes.

http {
  listen 443;
  ssl on;
  ssl_certificate /path/to/server_name.crt;
  ssl_certificate_key /path/to/server_name.pem;

  location / {
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Host $host;

Tuesday, October 18, 2011

Getting MRI 1.9.2 on Ubuntu 11.10

The release of Ubuntu 11.10 (Oneiric) brings much-needed updates to the Ruby packaging. We can now get a sane Ruby environment in Ubuntu with a few commands.

sudo apt-get install ruby1.9.1-full
sudo update-alternatives --set ruby /usr/bin/ruby1.9.1

sudo env REALLY_GEM_UPDATE_SYSTEM=1 gem update --system

This sets up a fully working MRI 1.9.2 environment, with an up-to-date version of Rubygems.


Tuesday, February 8, 2011

Quick Fix for Ugly Font Rendering in Windows Browsers

This post presents a quick fix for the broken font rendering problem on Windows browsers. The quick fix disables @font-face CSS directives on Windows, using JavaScript. The next sections present the code, and describe the motivation behind this solution.


All the code is standard JavaScript, except for $(window).load, which is jQuery's way of hooking into the onload DOM event. Replace that with your framework's specific method, if you're not using jQuery.

I got bug reports stating that some fonts I chose look horrible on Windows. I looked into the problem, and saw that most downloaded fonts look much worse on Windows than on the other OSes. While playing with font choices, I also noticed that default fonts look decent, and I concluded that I shouldn't be downloading fonts via @font-face on Windows.

I didn't want to degrade the experience for all my other users, so taking out the @font-face directives from CSS was out of the question. Because of the way that Rails handles CSS optimizations, I couldn't produce a different CSS stylesheet just for Windows clients, so I decided to remove the @font-face directives in JavaScript.

The JavaScript above can be dropped into any Web application that uses @font-face, regardless of server-side language or framework, and it will make life better for Windows users. However, because it is JavaScript, Windows users may notice a viewport re-paint while JavaScript changes the page's stylesheet. This will only happen on the first visit to the site, as long as your Web application serves correct caching headers for the JavaScript and CSS.

The code is also very bad programming practice. It is entirely possible that Windows 2020 will have better font rendering, or that browser providers will take matters into their own hands. If that happens, you will have to do feature detection on the font rendering engine.

Thank you for reading this far! I hope that you have found the post useful, and I'm looking forward to your questions and suggestions for improvements!

Sunday, August 15, 2010

Helper Testing in Rails 3

This post shows how to test both simple and complicated testers in Rails 3. The method outlined here should work in Rails 2.3 as well, but it might not be the best method.

The skeleton helper testcase class generated by Rails 3 inherits from ActionView::TestCase. Simple helpers can be tested by comparing their return value with a golden string. For more complex helpers, use render :text to fill the view state with the helper return value, then use assert_select. assert_tag doesn't work.

Simple Helpers
Helpers that produce a single tag, or a string, are probably best tested simply by comparing their output with an expected value. Helpers can be called directly from the test class, as inheriting from ActionView::TestCase will set up an appropriate test environment, which includes a mock controller if necessary.

The code listing below demonstrates this.

Complex Helpers
To me, canonical example of a complex helper is one that includes a call to form_tag or form_for. These methods produce a div inside the form tag, and the contents of the div has been very brittle during Rails 3 development, mostly due to the snowman debate. A test shouldn't depend on the div contents, if all that matters is that there's a form in the helper's output.

This post suggests that at some point, Rails had an assert_tag_in method, which works similarly to assert_select, but takes a string input instead of testing against the current view output. Unfortunately, the method isn't available in Rails 3.

However, ActionView::TestCase provides a render method, which can be used to create fake view output. assert_select checks for the fake view output, but assert_tag doesn't, so your tests will have to stick to the former method.

The code listing below demonstrates this approach.

I wrote this post because I wanted to properly test a helper with a form tag. I spent a few hours sifting through blog posts, RDocs, LightHouse tickets and GitHub commits. I hope this post helps other developers avoid duplicating my effort.

Rails 3 makes testing helpers easy, as long as you know what methods to call. I hope you found my post useful, and I'm looking forward to your comments and suggestions!

Monday, August 9, 2010

Replicating GitHub's Repository Serving Setup

This post describes my approach to replicating GitHub's awesome setup for hosting SSH repositories. GitHub's engineers already outlined the main ideas in their blog, so this post fills in the details by going over all the little roadblocks that I've stumbled on, and explaining the solutions I adopted for each of them.

GitHub allows you to create a git repository via a Web interface, then interact with the repository using a few protocols. I focused on git+ssh, because it's the only protocol that allows pushing to repositories.

I assume that the Git server is a Linux box, and I tested my work on Ubuntu. The setup will most likely require changes for MacOS. I use the already-available infrastructure, such as git and ssh. On Debian, you should have the git-core and openssh-server packages installed.

I assume you're not willing to change the OpenSSH server configuration, as you want to stick to the secure defaults on your production infrastructure.

My Web application is written in Ruby on Rails, and it uses the grit gem, also written by the GitHub engineers. While the code is specific to this technology, most of the article is relevant outside the Rails world.

To avoid edge cases in putting together shell commands, I assume repositories and Web users have very sane names (only letters, digits, underscores).

The code for this article is available here (GitHub). All the code links in this article reference a particular tree, so that the code would match the writing. Future revisions are available in the same repository, but they may be more optimized and harder to follow.

Big Picture
A GitHub blog post describes their SSH serving architecture (section Tracing a SSH Request). Scaling aside, these are the main components:
  • the server has a git user; git pushes and pulls get processed under that user account
  • authorized_keys (the sshd configuration) for the git user contains all the public SSH keys in the Web application (GitHub modified openssh to query a database, since their authorized_keys would have been huge)
  • authorized_keys also sets up sshd to run a restricted shell instead of the user's normal login shell (git provides git-shell for this purpose), so users can't use the git account to get shell access on the server
  • sshd is not pointed to git-shell directly; instead, a proprietary wrapper checks that the SSH key's owner is allowed to access the repository and, if the operation succeeds, flushes or updates any Web application caches associated with the modified repository
The Git User
I use this script to set up the git user. The script accepts any user name, but to keep it simple, I'll use git in this article. The script's effects are undone by this script, which I won't cover here.

I don't expect that the Web application will run under the git account, so instead I set the git user's primary group to be the same as a group on the Web application's account. I assume that the Web application will run under its own user, and nothing else will use that user and group. It's also possible to create a git group, and add the Web application to it. What matters is that the Web application can write to the git user's home directory.

The git repositories are stored in the repos directory, under git's homedir. The repos directory must be writable by the Web application, so its permission bits are 770 (rwxrwx---).

Recent versions of sshd are very strict with the permissions on authorized_keys. I appease them by setting the git home directory bits to 750 (rwxr-x---), setting .ssh's bits to 700 (rwx------), and setting .ssh/authorized_keys bits to 600 (rw-------). So the Web application will not be able to change authorized_keys directly, which is a problem, since I'd like to manage authorized SSH keys in the Web application.

To compensate for the issue above, I create an install_keys setuid executable in git's homedir that overwrites authorized_keys with the contents of repos/.ssh_keys. Due to the setup above, only the Web application should be able to write to this file. Furthermore, install_keys's bits are 750 (rwxr-x---), so it can only be run by the Web application.

I encountered two more minor but annoying issues. install_keys cannot be a script, because the setuid flag doesn't work for scripts with shebang lines. My setup script writes out a C program and runs gcc to obtain a binary. The setuid bit is lost when a file is chowned, so chmod must be called after chown.

SSH Configuration and Integration
authorized_key has one line for each key. The code that generates the lines is here. I used all the options I could find to lock down git's shell. The command line points to a shell script in my Rails application, and passes the key's ID as an argument, so I can identify the Web application user.

My keys are standard Rails models, and I use the after_save and after_destroy ActiveRecord callbacks to keep authorized_keys in sync with the database. More specifically, for each key addition or modification, I re-generate the contents of authorized_keys, write it to /home/git/repos/.ssh_keys and then run install_keys to get the actual file modified. If response time becomes an issue, I can move this process to a background work queue.

git's restricted shell script (stub here, real code here, test here) is stored in my Rails application, for my development convenience. Once the implementation is crystallized, it could be moved in git's home directory, so the git user doesn't need read access to the Web application's directory. The script performs the following steps:
  • parses the command line to extract the git command to be executed, the repository path, and the key ID
  • checks that the git command line matches the list of allowed commands, and determines whether the command does a pull or a push
  • issues a HTTP request against the Web application to check whether the SSH key's owner has permission to pull or push to the repository
  • runs the git command, prefixing the repository path with repos/ 
  • if the command succeeds, pings the Web application so it can update its caches
  • relays the git command's exit code

Web Application and Testing Considerations
An application user is modeled by a Profile, and a git repo is represented by a Repository. Both models use ActiveRecord callbacks to synchronize on-disk state: profiles correspond to directories under repos/ (sync code here) and repositories correspond to bare Git repositories created by grit (sync code here). The callbacks are slightly more complicated than for SSH keys, because I need the old name when a profile or repository's name changes, in order to rename the corresponding directory.

Unit tests stub out the methods that compute the path to repos/ (code here and here), so they can run without a full setup. This is desirable so I can get CI running later. I also have a giant integration test (code) that runs all the code described in this article. Since it creates a new user, it requires sudo access, which makes it unlikely that I'll ever be able to set up CI for it.

A particularly challenging for the integration test was pointing git to a SSH key to be used when pushing and pulling. Git doesn't take a command-line argument, and the easiest solution I found was to override the ssh command used by git with a custom wrapper (wrapper-generating code) that contains the options I need to point to a custom key. Overriding is achieved by setting the GIT_SSH environment variable (code). Also, the permission bits on the key file must be set to 600 (rw-------), otherwise ssh will ignore the key.

I find GitHub's setup to be awesome, and I always wanted to have my own server implementation to play with. Since I read the team's blog post on how to do it, I wanted to give it a try. When I finally found the time, my experience was rewarding, but also frustrating. POSIX permission bits are limiting, and working around them is non-trivial, at least for me.

I hope you found this post useful, or at least interesting. I look forward to your comments and suggestions for simplifying the setup, or improving its security!