We recently underwent a pretty big change here at TransFS.com… all entirely “under the hood”. We decided to migrate from our old hosting setup at Joyent, to our new home at EngineYard Cloud. We’re now running on Amazon EC2 instances, all transparently and easily managed by the excellent admin tools provided by Engine Yard.

Why the switch?

I had several reasons for deciding that Joyent was no longer working for us. First, the Solaris architecture that Joyent uses provided no obvious benefits, and a surprising number of headaches. Solaris has most of the same tools as linux… but many of them take different command-line arguments or are annoyingly lacking in features. For example: “grep -R” for recursive search through files, doesn’t exist on Solaris. Why? I have no idea. But these types of things can become a real pain over time, and while they seem small, they really do matter.

Secondly, Joyent provides no automatic backup solution for your data. I was very surprised when we first switched to Joyent that I had to roll my own backup solution with mysqldump, rsync, and a script that uploaded to Amazon S3. I never felt good about this, because backups are so important, and because it isn’t an area that I’m very confident in… and I was never able to properly test that my backups would save us in the case of massive data loss. When selecting a hosting provider, I really want someone to hand me a pre-built backup solution that I can trust uses industry best-practices and has been carefully tested. EngineYard provided me with much more confidence on this front, and automated daily or hourly backups are dead-simple to set up (no hacking of my own scripts!)

Finally, we have been having some real problems with the memory footprint of our app. It has ballooned to over 150MB per rails instance, sometimes as high as 250MB. Between our Passenger children, background workers, mysql, etc… we kept bumping up against the RAM limit on our Joyent slice. This was obviously our fault, but the process for dealing with it with Joyent made it a real pain to solve. Moving to a two-slice setup, where our mysql db lived on a separate server was an obvious solution… but provisioning and setting up a whole new server was something that I didn’t have time for, and all Joyent really offered was to clone my existing setup. I would have had to deal with everything else myself, and suffer through a lot of downtime while I figured it out. Even switching to a Joyent slice with more RAM was not an instantaneous process. I would have needed to create a support ticket, and then wait for someone to get back to me and clone us over to a new “accelerator” slice. I’m sure this would have worked… but it isn’t nearly as convenient as it should be, in the age of push-button admin interfaces like those provided by EngineYard Cloud. In fact, with EY Cloud, I was able to switch to running our db on a separate server with a single checkbox and re-deploy. Very cool. We’re now running safely under the RAM threshold of the smallest EC2 instance, and I can very easily scale up to more servers or larger server by selecting a different server type in an EngineYard dropdown.

Custom Chef Recipes

It actually took us a lot longer to switch from Joyent than I had hoped. I began the process of investigating EY Cloud a few months ago… and had delayed the process for a while because I had some real concerns about some parts of our setup. Gems, packages, etc are all super-simple to set up in EY Cloud… but certain things are not as easy as a “checkbox”. For instance, we run Starling & Workling for background processing at TransFS.com. Getting these set up in EY Cloud was non-trivial, because it required writing a custom chef recipe. Chef is a really amazing server configuration tool that uses a ruby DSL to define changes to your server config. EngineYard has some decent chef documentation, but it still took some time to learn how Chef works before I was comfortable enough with it to get our servers set up properly.

However, once you learn to be a master Chef… it gives you amazing power to set up your server exactly the way you want, and in a totally repeatable way. In fact, when I need to add a new application server to our cluster… it should be as simple as flipping a switch, and all of my custom configuration settings will be applied automatically to bring that new server into the proper state for TransFS.com. Very cool. Learning Chef is a necessity to getting up and running with EngineYard Cloud… but it is worth the effort.

There are a few other quirks to EY Cloud that took some work to figure out, such as migrating capistrano hooks… but overall I was very pleased with the process. I had us up-and-running on the new servers in about 30 hours, start to finish, and we only had about 30 minutes of real downtime while I rsync’d our data over.

Anyway, thanks EngineYard for a great experience so far. And thanks as well to Joyent for hosting us through our early days… we’ll miss you, but we’ve found a new home that is much better suited to our needs!

Coming up in the next post: some lessons learned about Nginx, Rails, and WordPress…