The Eagle is Down

Yahoo! FireEagle is finally dead. One of those little loved Yahoo! products that was way ahead of it's time (Pipes being the obvious other example). I actually loved the way it integrated with several location based services like Dopplr and Brightkite. I even had a small widget on my blog powered by it. I used to track all my trips on Dopplr, which would in turn update FireEagle. Another service called EagleFeed consumed this location from FireEagle and exposed it as an RSS feed. I then had a simple widget that would consume the RSS feed and populate the "Currently At" widget on my blog. This widget would, of course, automatically update whenever I actually made a trip that I had recorded on Dopplr. The whole thing was surprisingly stable and worked flawlessly all these years without a single change. The RSS feed end point actually still appears to work with possibly a cached copy of the data. I guess it's finally time to kill the widget.

PS: Yahoo! "Paranoids" probably know why the domain was hosted off yahoo.net instead of yahoo.com :)

Real Time Voting App with Node.js, Socket.io and mongoDB

Tried my hand at building a real time voting app this weekend using Node.js, Socket.io and Mongoose. I had used Node.js and Socket.io before, so I was fairly familiar with them, but this was my first experience with Mongoose (and actually MongoDB as well) Some lessons learnt

  • Things move fast in Node.js land. And things break fast as well. There were several instances where some of the code I had written against older versions of Node.js/Express were just not valid anymore. Even the socket.io documentation has different steps for Express v3 and v2.
  • Redis makes things like this trivial to implement. Redis has pubsub baked into it, and you can simply publish events to it that your socket.io code can react to (and push to the browser) Your application code need not even be written in Node.js. You could have a Rails/Django/PHP website and just use Node.js to push out events to the browser, and use Redis pubsub as the intermediate queue. I had already tried Redis (and absolutely *love* it), and I wanted to stick with a Mongo only approach, since this was just for a hackday.
  • Mongo has a tailable cursor (the analogy they provide for this is the Unix tail -f command) that seems ideal for use cases like this. However, this works only for capped collections. Capped means that you need to specify the max. size up front, and more importantly, updates and retrieves are done in the order of insertion. I believe you can convert a non-capped collection to a capped collection, but I read about it after I deleted and recreated the one I had. (I didn't have any data, in any case) Not sure about the performance impacts of using a tailable cursor though. You could also skip or read the same data multiple times (cursor goes back to the start when it gets reset I believe).
  • Mongoose documentation makes almost no mention of tailable cursors. In fact, I couldn't find many examples of people using tailable cursors to implement a real time app like this. The one example I did find (mongolab/tractorpush-server ) used native Mongo drivers instead of Mongoose, but that was good enough to get me started (it also convinced me that something like this was possible)
  • Mongoose 2.7.x documentation mentions tailable cursors (for some reason the 3.5.x one does not) Fortunately, the API listed there still seemed to work. This is the code I used to react to insertions into my collection
    var Vote = mongoose.model('Vote');
    var stream = Vote.find().tailable().stream();
    stream.on('data', function(vote) {
        // do stuff like socket.emit
    });
    
    You can add filters to the find() query as well.
  • socket.io is simply awesome. On that note, Heroku doesn't actually support websockets. You need to configure socket.io to use AJAX long polling instead. Fortunately, this *is* documented here. We found this to be near real-time and a pretty acceptable option.
  • Node.js is great for API servers/exposing JSON etc. but for some reason, I had a tough time finding a templating engine that was easy to use. Jade just didn't work, while the other popular ones were all no-logic, no HTML type stuff. I just wanted to write some simple HTML and embed a few dynamic values, goddamnit! I finally went with Swig which was fairly similar to the stuff I'm used to (i.e, erb/django templates or even PHP for that matter) More importantly, it had clear instructions for use with express, and the instructions actually worked!

Overall, it was a good, learning experience and I eventually got it all to work. However, I don't think Node.js would be my default choice next time if I had to quickly mock up some web pages (it might be for a REST API server) The code is at rohitnair/sportshackday if anyone's insterested, although most of it might be useless.

Dear Logitech

Dear Logitech/Apple/makers of mice and keyboards,
As a software engineer, I've realized the importance of using good, comfortable input devices pretty early in my career. I've decided that I'm ready to spend whatever it takes to buy such devices before I'm forced to do so by RSI. Unfortunately, the current selection of keyboards and mice don't quite make the cut. (All wireless, of course. Who needs wires?) Here's my wishlist for the holiday season

  1. A full sized Bluetooth mouse. Please, for god's sake! Bluetooth does not imply portability, or at least it doesn't always need to. It does however mean one less USB receiver to worry about.
  2. A truly "unified" product range. Right now, the Performance MX and the G700 use different receivers, different charging cables and more importantly, different software.
  3. Better Macbook support. The Performance MX has software for the Mac but the G700 does not. At the same time, the G700 can work as a wired mouse while the Performance MX cannot. Why is this relevant to Macs? Because they have only two USB ports. I need to use both ports while charging the Performance MX, one for the charger and one for the receiver. WTF?
  4. Keyboards/mice that can pair with multiple devices at once. Most people have multiple devices, (including tablets) if not multiple computers. The Logiteck K810 is the step in the right direction (and has great reviews on Amazon so far), so how about a mouse that can do the same?
  5. (Update) Had one more item on the wishlist. Batteries instead of charging with a wire! (and better battery life to go with it) The Apple Wireless Keyboard has outstanding battery life (lasting several months!), and is a truly wireless experience. Compare that to a Logitech wireless keyboard/mouse that has to be charged every few days and ends up being permanently wired in the process :|

Adding Spree to an Existing Rails app (and not mess up authentication)

tl;dr - Read Custom authentication (#1512) on Github and follow the instructions mentioned there

The Problem - You have an existing rails app. You've implemented a user authentication layer for it (this could be using something like Devise) Now, you want your app to have some e-commerce capabilities and decide to use Spree, an excellent and extremely comprehensive open source solution. You follow their instructions, do a spree install etc. and things break (especially authentication). A lot of people have posted this problem on stackoverflow and the Spree mailing list, but no one actually provides a clear solution.

The current version of Spree expects developers to build their apps around it and not the other way around. It completely takes over authentication, and worse, renames your existing users table to spree_users and completely changes the schema as well. It also takes over your routing, your landing page now points to the Spree landing page, and if you had your own '/admin' page, (this could be via something like rails_admin) that now points to the spree admin page.

Fortunately, there's a branch in active development that aims to rip out the whole spree_auth component and fix the authentication problem. You can follow the entire thread on Github - Custom authentication (#1512) In particular, look at this comment and follow the instructions there. Essentially, these changes would make sure that your users table is untouched, and you can tell Spree to just use whatever authentication you already have in your existing rails app. The guide mentioned there lists out all the steps required, but there are a couple of other changes that I had to do. I've listed them here.

Installing PostgreSQL on OS X Lion (using homebrew)

So I decided to switch from MySQL to PostgreSQL for my rails app (primarily because Heroku supports PostgreSQL) OS X Lions ships with psql (the client) but you still need to install the server manually. The easiest way is to just install it via homebrew, but unfortunately that wasn't a smooth process. The fact that this I've never used PostgreSQL before didn't help matters either. Here's a list of commands and hacks that I had to do to finally make it work.

  1. Install homebrew if you haven't yet (worth installing even if you don't need PostgreSQL) Then do
     brew install postgres 
    
  2. Ideally, that should be it, but in my case, it wasn't. The brew install failed for me because some mirror (ftp.kaist.ac.kr to be specific) was down. I worked around this by adding an entry to my hosts file and pointing this to a working mirror. Add the following line to your /etc/hosts file if you face the same issue
     198.82.183.70 ftp.kaist.ac.kr 
    
  3. The brew install will output a set of instructions for setting up/initializing the DB. The first is to run initdb as follows
     initdb /usr/local/var/postgres 
    

    No prizes for guessing that even this step failed for me :D I got the following error
    creating template1 database in /usr/local/var/postgres/base/1 ... FATAL:  could not create shared memory segment: Cannot allocate memory
    DETAIL:  Failed system call was shmget(key=1, size=2138112, 03600).
    HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space, or exceeded your kernel's SHMALL parameter.  You can either reduce the request size or reconfigure the kernel with larger SHMALL.  To reduce the request size (currently 2138112 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
    	The PostgreSQL documentation contains more information about shared memory configuration.
    child process exited with exit code 1
    
    Thankfully, this seems to be a fairly common problem and there are solutions posted online. The following commands fixed the issue for me.
    sudo sysctl -w kern.sysv.shmall=65536
    sudo sysctl -w kern.sysv.shmmax=16777216
    
  4. You can now start the PostgreSQL server (again follow the instructions from the brew install output) Specifically, look for the following lines and execute the appropriate commands
    # Start/Stop PostgreSQL
      If this is your first install, automatically load on login with:
      mkdir -p ~/Library/LaunchAgents
      cp /usr/local/Cellar/postgresql/9.1.3/homebrew.mxcl.postgresql.plist ~/Library/LaunchAgents/
      launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
      If this is an upgrade and you already have the homebrew.mxcl.postgresql.plist loaded:
      launchctl unload -w ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
      cp /usr/local/Cellar/postgresql/9.1.3/homebrew.mxcl.postgresql.plist ~/Library/LaunchAgents/
      launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
     Or start manually with:
      pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start
     And stop with:
      pg_ctl -D /usr/local/var/postgres stop -s -m fast
    
  5. Now try connecting with the client (using the psql command). This will also fail, as you can expect by now. You might see an error like follows
    $ psql
    psql: could not connect to server: Permission denied
    	Is the server running locally and accepting
    	connections on Unix domain socket "/var/pgsql_socket/.s.PGSQL.5432"?
    
    This usually happens because psql still points to the pre-installed client, not the one that homebrew installed. To fix this, just edit /etc/paths and place the brew install path (usually /usr/local/bin) above /usr/bin . This change does not get picked up automatically (you could force it by using the source command, or by just opening a new terminal window). Once this is done, verify that it's using the right path with a "which psql" command, or you can avoid all of this and just use the absolute path
  6. psql should finally work now, but will complain about a non-existent database. So first create a DB with the following command
    createdb <db_name>
    
    psql by default creates a connection with the current terminal user as the username, and to a db with the same name as the username. You can however use the command line arguments -U and -d respectively to change them. Also, you can create a new user can with the following command
    createuser <username> -P
    
    The -P option is to set a password for that user.

If things still fail, take a look at the log file (/usr/local/var/postgres/server.log) for clues.

Now, if you plan to use PostgreSQL with Ruby on Rails, there are a couple more steps involved.

  1. Install the gem using the following command. If you installed it previously, uninstall it and install it again.
    env ARCHFLAGS="-arch x86_64"; gem install pg
    
    Also, add the 'pg' gem to your Gemfile, and you can remove your existing DB adapter from it as well (mysql2 for MySQL)
  2. Change the following line in the relevant sections of your config/database.yml
      adapter: mysql2
    
    to
      adapter: postgresql
    

Perform a rake:db migrate to create all the required tables for Rails to work. Seed the DB if you have to. Now restart your server and verify if everything works fine. In most cases, the switch should be fairly smooth, however there could be a few queries that might break.