TransFS Blog / Archive RSS Feed
Archive for the ‘background processing’ Category
For the past year or so, we’ve been tracking our SEO progress with a tool that we developed internally. The idea was to keep an eye on the daily movement of our domain in the search results for various terms. We also wanted to collect some general statistics on these terms, like how much traffic they receive, the PageRank of the top ranking site, and advertising CPC estimates. All of this was rolled into a set of controllers, models, views, and background-processing code that I’ve now extracted out into a new rails engine plugin: SEOpener!
SEOpener combines data from the Google Ajax Search API, with other datasources such as Yahoo’s keyword estimation tool, and google toolbar pagerank queries, to try to give you a complete picture of how you are positioned on a given search term. It provides a simple admin interface for listing all of your terms, adding new ones, and viewing the top 64 search results for a given term.
SEOpener is designed to be a drop-in solution for existing apps, providing everything you might need to get started in tracking your sites search rankings. For the most part, it should be fairly easy to install, however you will need to come up with a background processing solution (you can use cron & workling/starling like we do, or something else) to run the data-scraping jobs at regular intervals.
SEOpener Admin Interface
We’ve been pretty happy with this tool, and have been running it in production for some time now…. but it would be great to get help from the community. Leave a comment if you have any ideas, or even better, fork us and hack away on it!
I recently completely overhauled our background task processing for TransFS.com. For the past 6 months, we’ve been running BackgroundRb… and I’ve been fairly satisfied with its performance. There are a lot of BackgroundRb haters out there, which I gather comes from some vicious problems with its usage of Drb in an earlier version. However, by the time I found and installed BackgroundRb… it had been rewritten from the ground up using Packet, and it was both stable and quite powerful.
I started to get anxious about the amount of RAM being taken up by my backgroundrb workers.
Nonetheless, as I’ve continued to create new background processing tasks… I started to get anxious about the amount of RAM being taken up by my backgroundrb workers. In addition, the BackgroundRb “style” has always seemed rather awkward to me. It offers some nice features, for sure, that are hard to find in a single background processing package elsewhere. But, at the end of the day, I started to wish that we were using something that was both lighter and easier to code against.
Finally, over the weekend… the final straw: I need to set up a new process on our Joyent server, and I didn’t have the RAM available. So, after reading a bunch of documentation and evaluating several solutions, it was “Goodbye BackgroundRb, Hello Workling & Starling”.
