Thursday, January 14, 2010

Hosting Ruby on Rails on scripts.mit.edu

This post describes a method for developing a Rails application and deploying it on scripts.mit.edu, a free hosting service for MIT students and staff. The information here is primarily intended for students in MIT's Web Programming Competition (6.470), but might be useful for other MIT students and staff, as well as for Rails developers deploying in constrained environments.

Summary
The first part of my post shows how to create a Rails application on scripts.mit.edu. By default, the application will use the Rails version that is installed on scripts, which will usually be behind the edge version. The second part of the post copies the application to the developer's computer, and modifies it to use the latest version of Rails. The third part of the post describes the steps needed to deploy a new version of the application back on the scripts.mit.edu servers.

Creating the Application
In the steps below, replace costan with your Athena username, and substitute sixprep with the name of your application.
  1. SSH into Athena: ssh costan@linux.mit.edu
  2. Sign up for scripts.mit.edu and MySQL: add scripts; signup-web; signup-sql
  3. Generate the Rails application: add scripts; scripts-rails app_name
    1. When prompted to enter 1 or 2, enter 1 (you're a user, not a group).
    2. Add the application name to the Desired address: field. For example: http://costan.scripts.mit.edu/app_name
    3. When asked to press Enter to continue, check your application's directory. It should be something along the lines of ~/web_scripts/app_name
  4. Verify that the application works by pointing Firefox to http://costan.scripts.mit.edu/app_name/
Move to the Latest Rails Release
The scripts-rails tool uses the rails command on the server,
  1. Open a command shell in your Eclipse workspace: cd ~/workspace
  2. Copy the application: scp -r costan@linux.mit.edu:~/web_scripts/app_name .
  3. Go into the application: cd app_name
  4. Hack around a configuration bug: ln -s . public/app_name
  5. Install the latest version of Rails: sudo gem install rails
  6. See what the latest version of Rails is (this example assumes 2.3.5): rails -v
  7. Edit config/environment.rb to say RAILS_GEM_VERSION='2.3.5'
  8. Update the configuration files: rake rails:update
  9. Add a requirement (the line below) for the Rack gem to your app's config/environment.rb:
      config.gem 'rack', :version => '1.0.1'
  10. Vendor Rails: rake rails:freeze:gems VERSION=2.3.5
  11. Vendor any other gems that your application uses: rake gems:unpack:dependencies
  12. Follow the steps below to deploy the application back to scripts.mit.edu, and verify that it still works.
Deploy Back on Scripts
The command sequence below will update your application in an almost safe way.
  1. Go into the application directory: cd ~/workspace/app_name
  2. Copy the application to scripts: scp -r . costan@linux.mit.edu:~/web_scripts/app_name
    NOTE: I'm using scp for simplicity. You'll probably want to put sixprep under version control, then remove web_scripts/app_name and check it out from the repository.
  3. SSH into a scripts server:
    ssh costan@linux.mit.edu
    ssh -k scripts

  4. Go into the application directory: cd app_name
  5. Build any new native gems you might have added: rake gems:build
  6. Apply database migrations: RAILS_ENV=production rake db:migrate
NOTE: earlier versions of this post included an extra step for killing FastCGI workers that are caching stale code. The scripts.mit.edu software improved, and this step is not necessary anymore.

The upgrade process is almost safe. The following things can go wrong:
  • a FastCGI worker is spawned with new code, works on the old database schema, and trashes the data (window of opportunity: HTTP requests between steps 2 and 6, assuming the request makes FastCGI spawn a new worker process)
  • a FastCGI worker caching the old code works on the new database schema, and trashes the data (window of opportunity: HTTP request between steps 6 and 7, assuming the requests hits a live FastCGI worker)
It shouldn't be too hard to figure out a solution to these problems, but I decided the windows of opportunity are small enough that I don't care. If you do find a nice and easy solution, please let me know in the comments.

Discussion
This is my method for hosting Rails applications on scripts.mit.edu. It is simple, and it allows me to use the latest version of Rails. There are other strategies that work well.

For example, you could use the version of Rails on scripts.mit.edu (rails -v to find the version on scripts, then gem install rails --version 2.3.2 to install a specific version of Rails). The scripts.mit.edu team back-ported the fixes for security vulnerabilities since 2.3.2 came out, and intends to do so in the future. This might be a better solution, if you don't plan to be proactive about testing and updating your application whenever a new Rails release comes out.

One more thing - in case you're wondering, you will not be able to use scripts.mit.edu with its current software with Rails 3. The reason is that the scripts server use Ruby 1.8.6, and Rails 3 requires Ruby 1.8.7 or above. Unfortunately, Ruby cannot be vendored :)

I hope you found this post useful, and I'm looking forward to your questions and suggestions in the comments.

Acknowledgements
Alice Li and Christopher Luna helped me iron out some bugs in this post by following earlier versions of it.

Sunday, January 10, 2010

Automated Backup for Picasa Web Albums

This post describes a script for downloading all the photos in a Picasa Web account. My script is mainly intended for Linux users, because Windows and Mac users can use the Picasa desktop software to download Web albums easily.

Usage
The script is written in Ruby, and uses RubyGems. You will need to install the appropriate packages for your distribution. For example, in Debian and Ubuntu:
sudo apt-get install ruby rubygems libopenssl-ruby

The script uses Google's gdata RubyGem, so you need to install it:
sudo gem install gdata

Download and unpack the script from my GitHub gist. Create an account.yml file in the same folder, using the following template:
username: costan
password: "my secret password"

Run the script:
ruby picasa_downloader.rb

Find your photos, nicely grouped by album, in the picasa_username folder.

For Developers
I used Google's gdata RubyGem, and the Picasa Web Albums protocol guide and protocol reference.

The code is pretty straight-forward. There is one subtlety: for photos bigger than 1600x1200, the Web Albums API returns the URL for 1600x1200 thumbnails by default. To get the original uploaded file, I use the imgmax=d parameter in album feed URIs.

Motivation and Conclusion
I wrote this script because the Picasa desktop software does not seem to be able to download from Picasa Web Albums on Linux.

I hope you found my post useful. I welcome your comments, as well as your pull requests on GitHub.

Saturday, January 9, 2010

Ruby on Rails Development Setup

This post documents my development setup for Ruby on Rails on Linux, MacOS X, and Windows. The Linux-specific instructions are geared towards Ubuntu, but should be easy to translate to other distributions.

Platform-Dependent Steps
The first part of the setup is installing the Ruby programming language, the Rubygems package management library, and the MySQL and SQLite database software. Due to the binary nature of the packages, the instructions are different for each OS.

Keep in mind that Windows and MacOS X differ in various ways from Linux, which will likely be your production environment. For best results, I strongly recommend installing Ubuntu as a dual-boot, or using Virtual Box to get an Ubuntu VM.

Ubuntu Linux
The steps below have been tested on a fresh installation of Ubuntu 9.10 Desktop.
  1. Update your package repository: sudo apt-get update
  2. Install Ruby: sudo apt-get install build-essential ruby irb rubygems libopenssl-ruby
  3. Install MySQL: sudo apt-get install mysql-client mysql-server libmysqlclient-dev
  4. When prompted, leave the MySQL root password blank. This will make it easy to share a database.yml file with the rest of your team.
  5. Install SQLite:  sudo apt-get install sqlite3 libsqlite3-dev
  6. Find out the system-wide RubyGems binary path (pick the one that is not in your home directory): gem env gempath
  7. Add the path to the system-wide binary path: sudo echo "PATH=/var/lib/gems/1.8/bin:$PATH" > /etc/profile.d/rubygems1.8.sh
  8. Reboot the system so the path change goes into effect.

MacOS X
The steps below have been tested on a fresh installation of MacOS X 10.6: Snow Leopard.
  1. Install Xcode from Apple's Developer site, or from the MacOS installation DVD.
  2. Go to the MySQL download site, and download the Mac OS X (package format) for your OS (Leopard: 10.5 x86, Snow Leopard: 10.5 x86_64). 
    1. You don't have to register, there is a No thanks, just take me to the downloads! link on the mirror page.
    2. Install the mysql- pkg file.
    3. Install MySQLStartupItem.pkg.
    4. Install (by double-clicking) MySQL.prefPane, and choose to Install for all users. Start the MySQL server, and check the Automatically Start MySQL Server on Startup box.
  3. In the Platform-Independent Steps, substitute gem install mysql with the following command, taken from the Riding Rails blog: sudo env ARCHFLAGS="-arch x86_64" gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
Windows
The steps below have been tested on a fresh Windows XP SP3 32-bit installation.
  1. Install Firefox: www.getfirefox.com
  2. Install the latest stable release of 7-Zip: http://www.7-zip.org/
  3. Launch the 7-Zip File Manager (in Start > All Programs > 7-Zip). From the menu, select Tools > Options, and check .7z under Associate 7-Zip with. Click Ok to dismiss the dialog, and exit the file manager.
  4. Go to the OneClick Ruby download site.
    1. From the latest RubyInstaller, download the rubyinstaller-1.8.6 executable. Install in the default path c:\Ruby and "add Ruby executable to your path".
    2. From the latest Development Kit, download the devkit- archive. Extract its contents in c:\Ruby
  5. Go to the MySQL download site, and download the Windows MSI Installer (x86)
    1. You don't have to register, there is a No thanks, just take me to the downloads! link on the mirror page.
    2. After downloading, do a Complete Install.
    3. At the end of installation, check Configure the MySQL server now, and uncheck Register the MySQL server now.
    4. When configuring the server, select Detailed Configuration, then Developer Machine, then Multifunctional Database. Accept the defaults, until the Install as Windows Service option shows up, then select both Install as Windows Service and Include Bin Directory in Windows PATH. For ease of development, don't change the root password (so it stays blank).
  6. Go to the SQLite download site, under the Precompiled Binaries For Windows section. Download the sqlite- and the sqlitedll- zip files, and extract them into c:\windows\system32.
  7. Restart your computer so the PATH changes go into effect.
  8. You can go into a Ruby-enabled command prompt via Start > All Programs > Ruby 1.8.6 > Start Command Prompt with Ruby.
  9. In the Platform-Independent Steps section, substitute gem install mysql with the following command. gem install mysql --source http://gems.rubyinstaller.org
Platform-Independent Steps
The second part of the development environment setup is identical in all OSes, with one small exception. The gem command should be executed with superuser privileges on Unix systems (Linux, MacOS X). To achieve that, all gem commands should be prefixed by sudo. For example, gem update would become sudo gem update.

Issue the following commands, and you're done setting up!
gem update --system
gem update
gem install rdoc 
gem install rails
gem install mysql
gem install sqlite3-ruby
gem install json mongrel ruby-debug-ide

Motivation
My post is intended to help the students in MIT's Web Programming Competition (6.470) who choose Rails as their back-end platform.

I hope you have found this post to be useful. Please leave a comment if you have trouble using the instructions above, or if you have a suggestion for improving these steps.

Saturday, January 2, 2010

Advice for 6.470 Teams Using Ruby on Rails

This post contains a suggested strategy for students competing in 6.470 (MIT's yearly Web programming competition) who are planning to use Ruby on Rails.

Motivation
Ruby on Rails will be covered in 6.470 on Friday. The competing back-end platform, PHP, will be covered on Tuesday. This post addresses students or teams who are considering using Ruby on Rails, but are concerned by the disadvantage of not being exposed to the platform earlier in the course.

Advice
My advice is based on my experience as a 6.470 contestant in 2008. My team won the 2nd place. The points below are the parts of our strategy that went well, and what I wish we would have done better.

  1. Don't agonize over your idea. Like in a startup, you will end up revising your idea as you learn what you can and can't accomplish in a month. Start prototyping, and start learning about the data that you can obtain.
  2. Research integration points (other sites' APIs) that you will use early. For some APIs, your developer account needs to be approved manually, and that can take a few days. It's a good idea to start as early as possible.
  3. Put your infrastructure in place. Set up version control (I recommend git, but I know svn is easy to set up on Athena). Set up story tracking (see Pivotal Tracker) so you don't forget about bugs. Set up regular meeting and coding times.
  4. Finalize your team staffing as quickly as possible. Either compete solo, or get a team of the maximum size allowed. Solo developers might impress the judges. That aside, nobody cares about your team size, and you don't want to miss the big prize because your work isn't polished enough.
  5. Set ground rules. 6.470 brings together competition, pride, and money, which are big potentials for drama. That can be mitigated
  6. Spend most of your time until Friday on the front-end. Make sure you learn HTML, CSS, JavaScript and Photoshop.
  7. Start mocking the UI using HTML, CSS and JavaScript. Get used to Firebug. Both the skills and the digital assets you'll develop are easily transferable into a Rails application.
  8. Pay particular attention to jQuery. Rails recommends Prototype for JavaScript, but jQuery is better at keeping your JavaScript out of your HTML. At the very least, you'll have an easier time learning Prototype if you know the principles of jQuery.
  9. Don't worry too much about SQL, but pay attention to database design.
  10. Think about whether you'll have testers. Our testers gave us great feedback and helped us make the site more usable. On the other hand, one of the early adopters showed us a security vulnerability by crashing our production database right before the deadline. So choose your testers wisely.
Relax
Rails is the most productive well-known Web development framework. There's a very good reason why the Rails book has a hammock on its cover. You can totally catch up with your competitors' back-ends over the first weekend.

Good luck!