Hosting Ruby on Rails on scripts.mit.edu

This post describes a method for developing a Rails application and deploying it on scripts.mit.edu, a free hosting service for MIT students and staff. The information here is primarily intended for students in MIT's Web Programming Competition (6.470), but might be useful for other MIT students and staff, as well as for Rails developers deploying in constrained environments.

Summary
The first part of my post shows how to create a Rails application on scripts.mit.edu. By default, the application will use the Rails version that is installed on scripts, which will usually be behind the edge version. The second part of the post copies the application to the developer's computer, and modifies it to use the latest version of Rails. The third part of the post describes the steps needed to deploy a new version of the application back on the scripts.mit.edu servers.

Creating the Application
In the steps below, replace costan with your Athena username, and substitute sixprep with the name of your application.

1. SSH into Athena: ssh costan@linux.mit.edu
2. Sign up for scripts.mit.edu and MySQL: add scripts; signup-web; signup-sql
3. Generate the Rails application: add scripts; scripts-rails app_name
   
1. When prompted to enter 1 or 2, enter 1 (you're a user, not a group).
2. Add the application name to the Desired address: field. For example: http://costan.scripts.mit.edu/app_name
3. When asked to press Enter to continue, check your application's directory. It should be something along the lines of ~/web_scripts/app_name
   
4. Verify that the application works by pointing Firefox to http://costan.scripts.mit.edu/app_name/

Move to the Latest Rails Release
The scripts-rails tool uses the rails command on the server,

1. Open a command shell in your Eclipse workspace: cd ~/workspace
   
2. Copy the application: scp -r costan@linux.mit.edu:~/web_scripts/app_name .
   
3. Go into the application: cd app_name
4. Hack around a configuration bug: ln -s . public/app_name
   
5. Install the latest version of Rails: sudo gem install rails
6. See what the latest version of Rails is (this example assumes 2.3.5): rails -v
7. Edit config/environment.rb to say RAILS_GEM_VERSION='2.3.5'
8. Update the configuration files: rake rails:update
   
9. Add a requirement (the line below) for the Rack gem to your app's config/environment.rb:
     config.gem 'rack', :version => '1.0.1'
   
10. Vendor Rails: rake rails:freeze:gems VERSION=2.3.5
    
11. Vendor any other gems that your application uses: rake gems:unpack:dependencies
12. Follow the steps below to deploy the application back to scripts.mit.edu, and verify that it still works.
    

Deploy Back on Scripts
The command sequence below will update your application in an almost safe way.

1. Go into the application directory: cd ~/workspace/app_name
2. Copy the application to scripts: scp -r . costan@linux.mit.edu:~/web_scripts/app_name
   NOTE: I'm using scp for simplicity. You'll probably want to put sixprep under version control, then remove web_scripts/app_name and check it out from the repository.
   
3. SSH into a scripts server:
   ssh costan@linux.mit.edu
   ssh -k scripts
   
4. Go into the application directory: cd app_name
5. Build any new native gems you might have added: rake gems:build
   
6. Apply database migrations: RAILS_ENV=production rake db:migrate

NOTE: earlier versions of this post included an extra step for killing FastCGI workers that are caching stale code. The scripts.mit.edu software improved, and this step is not necessary anymore.

The upgrade process is almost safe. The following things can go wrong:

• a FastCGI worker is spawned with new code, works on the old database schema, and trashes the data (window of opportunity: HTTP requests between steps 2 and 6, assuming the request makes FastCGI spawn a new worker process)
  
• a FastCGI worker caching the old code works on the new database schema, and trashes the data (window of opportunity: HTTP request between steps 6 and 7, assuming the requests hits a live FastCGI worker)
  

It shouldn't be too hard to figure out a solution to these problems, but I decided the windows of opportunity are small enough that I don't care. If you do find a nice and easy solution, please let me know in the comments.

Discussion
This is my method for hosting Rails applications on scripts.mit.edu. It is simple, and it allows me to use the latest version of Rails. There are other strategies that work well.

For example, you could use the version of Rails on scripts.mit.edu (rails -v to find the version on scripts, then gem install rails --version 2.3.2 to install a specific version of Rails). The scripts.mit.edu team back-ported the fixes for security vulnerabilities since 2.3.2 came out, and intends to do so in the future. This might be a better solution, if you don't plan to be proactive about testing and updating your application whenever a new Rails release comes out.

One more thing - in case you're wondering, you will not be able to use scripts.mit.edu with its current software with Rails 3. The reason is that the scripts server use Ruby 1.8.6, and Rails 3 requires Ruby 1.8.7 or above. Unfortunately, Ruby cannot be vendored :)

I hope you found this post useful, and I'm looking forward to your questions and suggestions in the comments.

Acknowledgements
Alice Li and Christopher Luna helped me iron out some bugs in this post by following earlier versions of it.

Automated Backup for Picasa Web Albums

This post describes a script for downloading all the photos in a Picasa Web account. My script is mainly intended for Linux users, because Windows and Mac users can use the Picasa desktop software to download Web albums easily.

Usage
The script is written in Ruby, and uses RubyGems. You will need to install the appropriate packages for your distribution. For example, in Debian and Ubuntu:

sudo apt-get install ruby rubygems libopenssl-ruby

The script uses Google's gdata RubyGem, so you need to install it:
sudo gem install gdata

Download and unpack the script from my GitHub gist. Create an account.yml file in the same folder, using the following template:

username: costan

password: "my secret password"

Run the script:

ruby picasa_downloader.rb

Find your photos, nicely grouped by album, in the picasa_username folder.

For Developers
I used Google's gdata RubyGem, and the Picasa Web Albums protocol guide and protocol reference.

The code is pretty straight-forward. There is one subtlety: for photos bigger than 1600x1200, the Web Albums API returns the URL for 1600x1200 thumbnails by default. To get the original uploaded file, I use the imgmax=d parameter in album feed URIs.

Motivation and Conclusion
I wrote this script because the Picasa desktop software does not seem to be able to download from Picasa Web Albums on Linux.

I hope you found my post useful. I welcome your comments, as well as your pull requests on GitHub.

Ruby on Rails Development Setup

This post documents my development setup for Ruby on Rails on Linux, MacOS X, and Windows. The Linux-specific instructions are geared towards Ubuntu, but should be easy to translate to other distributions.

Platform-Dependent Steps
The first part of the setup is installing the Ruby programming language, the Rubygems package management library, and the MySQL and SQLite database software. Due to the binary nature of the packages, the instructions are different for each OS.

Keep in mind that Windows and MacOS X differ in various ways from Linux, which will likely be your production environment. For best results, I strongly recommend installing Ubuntu as a dual-boot, or using Virtual Box to get an Ubuntu VM.

Ubuntu Linux
The steps below have been tested on a fresh installation of Ubuntu 9.10 Desktop.

1. Update your package repository: sudo apt-get update
2. Install Ruby: sudo apt-get install build-essential ruby irb rubygems libopenssl-ruby
3. Install MySQL: sudo apt-get install mysql-client mysql-server libmysqlclient-dev
4. When prompted, leave the MySQL root password blank. This will make it easy to share a database.yml file with the rest of your team.
5. Install SQLite:  sudo apt-get install sqlite3 libsqlite3-dev
6. Find out the system-wide RubyGems binary path (pick the one that is not in your home directory): gem env gempath
7. Add the path to the system-wide binary path: sudo echo "PATH=/var/lib/gems/1.8/bin:$PATH" > /etc/profile.d/rubygems1.8.sh
8. Reboot the system so the path change goes into effect.

MacOS X
The steps below have been tested on a fresh installation of MacOS X 10.6: Snow Leopard.

1. Install Xcode from Apple's Developer site, or from the MacOS installation DVD.
2. Go to the MySQL download site, and download the Mac OS X (package format) for your OS (Leopard: 10.5 x86, Snow Leopard: 10.5 x86_64). 
1. You don't have to register, there is a No thanks, just take me to the downloads! link on the mirror page.
2. Install the mysql- pkg file.
3. Install MySQLStartupItem.pkg.
   
4. Install (by double-clicking) MySQL.prefPane, and choose to Install for all users. Start the MySQL server, and check the Automatically Start MySQL Server on Startup box.
   
3. In the Platform-Independent Steps, substitute gem install mysql with the following command, taken from the Riding Rails blog: sudo env ARCHFLAGS="-arch x86_64" gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config

Windows
The steps below have been tested on a fresh Windows XP SP3 32-bit installation.

1. Install Firefox: www.getfirefox.com
2. Install the latest stable release of 7-Zip: http://www.7-zip.org/
3. Launch the 7-Zip File Manager (in Start > All Programs > 7-Zip). From the menu, select Tools > Options, and check .7z under Associate 7-Zip with. Click Ok to dismiss the dialog, and exit the file manager.
   
4. Go to the OneClick Ruby download site.
1. From the latest RubyInstaller, download the rubyinstaller-1.8.6 executable. Install in the default path c:\Ruby and "add Ruby executable to your path".
2. From the latest Development Kit, download the devkit- archive. Extract its contents in c:\Ruby
   
5. Go to the MySQL download site, and download the Windows MSI Installer (x86). 
1. You don't have to register, there is a No thanks, just take me to the downloads! link on the mirror page.
2. After downloading, do a Complete Install.
3. At the end of installation, check Configure the MySQL server now, and uncheck Register the MySQL server now.
4. When configuring the server, select Detailed Configuration, then Developer Machine, then Multifunctional Database. Accept the defaults, until the Install as Windows Service option shows up, then select both Install as Windows Service and Include Bin Directory in Windows PATH. For ease of development, don't change the root password (so it stays blank).
   
6. Go to the SQLite download site, under the Precompiled Binaries For Windows section. Download the sqlite- and the sqlitedll- zip files, and extract them into c:\windows\system32.
7. Restart your computer so the PATH changes go into effect.
   
8. You can go into a Ruby-enabled command prompt via Start > All Programs > Ruby 1.8.6 > Start Command Prompt with Ruby.
9. In the Platform-Independent Steps section, substitute gem install mysql with the following command. gem install mysql --source http://gems.rubyinstaller.org
   

Platform-Independent Steps
The second part of the development environment setup is identical in all OSes, with one small exception. The gem command should be executed with superuser privileges on Unix systems (Linux, MacOS X). To achieve that, all gem commands should be prefixed by sudo. For example, gem update would become sudo gem update.

Issue the following commands, and you're done setting up!

gem update --system

gem update

gem install rdoc 

gem install rails

gem install mysql

gem install sqlite3-ruby

gem install json mongrel ruby-debug-ide

Motivation
My post is intended to help the students in MIT's Web Programming Competition (6.470) who choose Rails as their back-end platform.

I hope you have found this post to be useful. Please leave a comment if you have trouble using the instructions above, or if you have a suggestion for improving these steps.

Advice for 6.470 Teams Using Ruby on Rails

This post contains a suggested strategy for students competing in 6.470 (MIT's yearly Web programming competition) who are planning to use Ruby on Rails.

Motivation
Ruby on Rails will be covered in 6.470 on Friday. The competing back-end platform, PHP, will be covered on Tuesday. This post addresses students or teams who are considering using Ruby on Rails, but are concerned by the disadvantage of not being exposed to the platform earlier in the course.

Advice
My advice is based on my experience as a 6.470 contestant in 2008. My team won the 2nd place. The points below are the parts of our strategy that went well, and what I wish we would have done better.

1. Don't agonize over your idea. Like in a startup, you will end up revising your idea as you learn what you can and can't accomplish in a month. Start prototyping, and start learning about the data that you can obtain.
   
2. Research integration points (other sites' APIs) that you will use early. For some APIs, your developer account needs to be approved manually, and that can take a few days. It's a good idea to start as early as possible.
3. Put your infrastructure in place. Set up version control (I recommend git, but I know svn is easy to set up on Athena). Set up story tracking (see Pivotal Tracker) so you don't forget about bugs. Set up regular meeting and coding times.
   
4. Finalize your team staffing as quickly as possible. Either compete solo, or get a team of the maximum size allowed. Solo developers might impress the judges. That aside, nobody cares about your team size, and you don't want to miss the big prize because your work isn't polished enough.
5. Set ground rules. 6.470 brings together competition, pride, and money, which are big potentials for drama. That can be mitigated
   
6. Spend most of your time until Friday on the front-end. Make sure you learn HTML, CSS, JavaScript and Photoshop.
   
7. Start mocking the UI using HTML, CSS and JavaScript. Get used to Firebug. Both the skills and the digital assets you'll develop are easily transferable into a Rails application.
   
8. Pay particular attention to jQuery. Rails recommends Prototype for JavaScript, but jQuery is better at keeping your JavaScript out of your HTML. At the very least, you'll have an easier time learning Prototype if you know the principles of jQuery.
9. Don't worry too much about SQL, but pay attention to database design.
10. Think about whether you'll have testers. Our testers gave us great feedback and helped us make the site more usable. On the other hand, one of the early adopters showed us a security vulnerability by crashing our production database right before the deadline. So choose your testers wisely.
    

Relax
Rails is the most productive well-known Web development framework. There's a very good reason why the Rails book has a hammock on its cover. You can totally catch up with your competitors' back-ends over the first weekend.

Good luck!

Fat iPhone Static Libraries: One File = Device + Simulator Code

Abstract
This post explains how to build a static iPhone library that contains ARM code (for iPhones and iPod Touches) and x86 code (for the simulator), in a single file.

This is useful for companies which do not wish to expose their source code when distributing iPhone code, e.g. AdMob. The currently accepted practice is to distribute separate static library files for devices and for the simulator. This means the that the library clients have to setup separate targets for the simulator and device. The separate targets duplicate most information, and only differ in the static libraries included. This violates the DRY principle,

Solution

Open a Terminal, cd into your library's Xcode project directory, and issue the following commands.

xcodebuild -sdk iphoneos3.0 "ARCHS=armv6 armv7" clean build

xcodebuild -sdk iphonesimulator3.0 "ARCHS=i386 x86_64" "VALID_ARCHS=i386 x86_64" clean build

lipo -output build/libClosedLib.a -create build/Release-iphoneos/libClosedLib.a build/Release-iphonesimulator/libClosedLib.a

The commands above assume your library's name is ClosedLib and you're targeting iPhone OS 3.0. It should be straightforward to tweak the commands to accomodate different library names and SDK versions.


Assumptions
I assume that the library is not updated too often, so a couple of manual steps are acceptable. I would like to make add an automatic build feature to my zerg-xcode tool, but there is no ETA for that. I also assume that your project's default target is the static library.

I do not assume that the device and simulator SDKs have similar headers. The beauty of my method is that the simulator code is built with the simulator SDK, and the device code is build with the device SDK.

Explanation
The format used in Mac OS X and iPhone OS libraries and executable files is Mach-O. The format supports "fat" binaries, which are pretty much concatenated Mach-O files for different architectures, with a thin header.

Xcode can build fat binaries. In fact, if you set Architectures to Optimized in your project's build options, you'll get a fat binary containing both ARMv6 (iPhone 2G + 3G, iPod Touch) and ARMv7 (iPhone 3GS, iPod Touch 2G + 3G) code.

Xcode's output is controlled by the options ARCH (Architectures) and VALID_ARCH (Valid architectures). The architectures that actually get built are the intersections of these two options. Due to the different ARM processors in iPhones, device builds have VALID_ARCH set to include ARMv6 and ARMv7. However, simulator builds only target the i386 platform. I want a fat binary for the simulator as well, so I change the VALID_ARCH option to include the AMD64 platform.

The last step in the process is the lipo tool that comes with the Xcode Tools, and manages fat binaries. I use it to create a fat binary containing the union of the architectures in the two static libraries. The device build contributes the ARM architectures, and the simulator build contributes the Intel architectures.

The build process can be tweaked to throw out the AMD64 code, but I wanted to avoid hard-coding processor constants. Most importantly, using a fat library does not translate to larger iPhone applications, because the GNU linker strips out unused architectures when building binaries.

Testing
I tested this method by creating a iPhone static library, and an iPhone Window Application. I built the library using the steps above, and I included the headers and static library in the application. Then I used the debugger to confirm that the library code works both on the simulator and on an iPod Touch 2G.

My solution was only tested with the latest version of Xcode at the time of this writing (Xcode 3.1.2), but it should work on any version of Xcode 3, assuming no bugs come up. I tested on Snow Leopard, but the Leopard version of the iPhone SDK should work as well.

References
Mac Dev Center: Mac OS X ABI Reference
Amit Singh: Mac OS X Internals

JCOP Smartcard Performance

Abstract
I use NXP JCOP smart-cards for prototyping in my research. I have recently benchmarked the cards using my research code, which contains computational and cryptographic workloads.

I found a couple of surprising results that I want to share, so fellow developers can make informed decisions when choosing their prototyping platforms.

Findings
There is a 1.5-2x speed difference between the different revisions of the same high-end chip, the NXP JCOP41 with 72kb of EEPROM. The V2.2 revision for smart-cards (no longer available, replaced by 2.2.1) has the best performance, and the V2.2.1 revision for the SIM (ID 000) form factor has the worst performance.

There is a significant speed difference between the same revision (V2.2.1) of the same chip (NXP JCOP41, 72kb of EEPROM), in different form factors. The chip for the smart-card form factor is almost as fast as the older V2.2 revision, while the chip for the SIM (ID 000) form factor is significantly slower.

There is a 2-4x speed difference between the same revisions (V2.2, smart-card form factor) of the NXP JCOP31 and the NXP JCOP41 chips.

The 3DES encryption/decryption engine has non-linear performance. The time it takes to decrypt 128 bytes is not very different from the time it takes to decrypt 24 bytes, on the JCOP41 chips. It seems that there is a huge setup cost for the DES engine, which outweighs the actual encryption cost.

There is a 4-8x speed difference between RSA and 3DES encryption on the JCOP41 chips, and a 3x speed difference on the JCOP31 chip. This goes against the conventional wisdom that symmetric encryption is 2 orders of magnitude faster then asymmetric encryption. This is probably due to the time it takes to setup the 3DES engine.

Conclusion
Secure processors in smart-cards have non-obvious performance characteristics. I hope my work saves you from the unpleasant surprises that I had.

Motivation
To the best of my knowledge, there are no easy to get benchmarks on smart-card processors. At least, I couldn't find anything when I searched.

Smart-card retailers disclose vital specifications, like EEPROM size and the cryptographic primitives that are implemented on the chip, but tend to be quiet about speed. The sites I used don't mention anything about the type of processor used, or the frequency of the processor.

For some applications (e.g. prototyping, where I want my unit tests to run quickly), speed is just as critical as the other specifications, and its more important than cost.

Data
The data that I used to reach my conclusions is available below. The benchmarks are described in section 5.1 (page 13) in my paper on a successor to the TPM.

decrypt_3des decrypts 24 bytes of data, while decrypt_3des_long and decrypt_rsa work on 128 bytes of data. 3DES is configured in EDE-CBC mode (112 bits of key material) and uses the ISO-9797 method 2 for padding. RSA decryption uses PCKS#1 padding.

The benchmarks can be reproduced by installing Rubygems, then installing the tem_ruby gem, and issuing the following commands

tem_upload_fw  # Uploads my JavaCard applet to the active smart-card.

tem_bench  # Runs the benchmarks.

NXP JCOP41 v2.2/72k (no longer available)
time_blank_bound_secpack_3des: 0.20757s
time_blank_bound_secpack_rsa: 0.86173s
time_blank_sec: 0.18017s
time_devchip_decrypt_3des: 0.05803s
time_devchip_decrypt_3des_long: 0.08042s
time_devchip_decrypt_rsa_long: 0.74047s
time_post_buffer: 0.08280s
time_simple_apdu: 0.00515s
time_vm_perf: 0.73887s
time_vm_perf_bound_3des: 0.78137s
time_vm_perf_bound_rsa: 1.43647s

NXP JCOP41 v2.2.1/72k (usasmartcard.com product link)
time_blank_bound_secpack_3des: 0.24155s
time_blank_bound_secpack_rsa: 0.89740s
time_blank_sec: 0.18937s
time_devchip_decrypt_3des: 0.08420s
time_devchip_decrypt_3des_long: 0.10800s
time_devchip_decrypt_rsa_long: 0.76480s
time_post_buffer: 0.08577s
time_simple_apdu: 0.00610s
time_vm_perf: 0.83257s
time_vm_perf_bound_3des: 0.90033s
time_vm_perf_bound_rsa: 1.55637s

NXP JCOP41 v2.2.1/72k USB token (usasmartcard.com product link, probably using this card)
time_blank_bound_secpack_3des: 0.41070s
time_blank_bound_secpack_rsa: 1.23089s
time_blank_sec: 0.34530s
time_devchip_decrypt_3des: 0.19010s
time_devchip_decrypt_3des_long: 0.21410s
time_devchip_decrypt_rsa_long: 1.05600s
time_post_buffer: 0.17213s
time_simple_apdu: 0.01000s
time_vm_perf: 1.11310s
time_vm_perf_bound_3des: 1.19703s
time_vm_perf_bound_rsa: 2.01420s


NXP JCOP31 v2.2 (usasmartcard.com product link)
time_blank_bound_secpack_3des: 0.84673s
time_blank_bound_secpack_rsa: 1.78957s
time_blank_sec: 0.78120s
time_devchip_decrypt_3des: 0.23553s
time_devchip_decrypt_3des_long: 0.50060s
time_devchip_decrypt_rsa_long: 1.54990s
time_post_buffer: 0.88864s
time_simple_apdu: 0.02813s
time_vm_perf: 1.84374s
time_vm_perf_bound_3des: 1.92594s
time_vm_perf_bound_rsa: 2.87900s

Quick Way to See Your Gems' Documentation

Summary
The fastest way to see the RDoc for your installed gems is to type the following command into a terminal, and point your Web browser to the address it returns (usually http://localhost:8808)
gem server

Details
The Web server created by the gem server command contains the RDocs for all the gems you have installed, unless you disabled RDoc generation when you installed your gems.


Motivation
I was used to either searching via Google, or going to a gem's RubyForge page to see the RDoc for the gem.

Initially, it seemed that the bit of effort is worth not having to learn some way to generate and bring up the RDocs myself. To my surprise, the procedure is very simple, and it's totally faster than browsing to some Web site that has the RDocs.

That aside, using a local server has the advantage that you'll see the RDocs corresponding to the exact versions of your gems. And, last but not least, RubyForge seems to be slowly falling into oblivion, and newer gems don't seem to bother publishing their RDocs.

I hope this post saves you some time.