Weathrman 3: The Weather Cloud

One of the big problems with Weathrman’s current implementation is that the whole implementation lives in your phone; a worst-case search can trawl through literally tens of thousands of search results, searching for an image relevant to your weather conditions and time of day.

Many of those searches are common to others; city-level searches are the same all over London, for example; local searches are the same for everyone sitting near me.  Much of this can be cached aggressively, massively reducing the amount of time it takes to get good results, and allowing me to do more searches, more often, at less cost and lower latency to end users.

It’s not ready yet - I’m nearly ready, probably another day away or so, and I’ll probably test it out for a week or so.  Come Google I/O, though, I’ll be ready to ship.

It’s notably faster, and pushing the image scaling to the server has resulted in massive improvements in image quality, while turning hundreds of RPC calls to flickr into a single call to the weatherman service.

Update on Sunday, May 16, 2010 at 2:22PM by Gregory Block

During this process, I’ve found some great bugs, and been able to do a few other really cool things.

I guess first and foremost is that it’s been much easier to test results globally, now that I can fire off image requests at will and a reload brings up a new image in a matter of seconds.

I’ve improved the quality of negative filters to get rid of more noise, improved the clock handling of flickr dates so we’re more likely to get images in the right time, and found a really awful bug that basically makes California look like it’s on fire whenever it’s cloudy.

This is going to be a great release.

Update on Sunday, May 16, 2010 at 8:21PM by Gregory Block

One of the difficulties in this move has been the responsiveness of Flickr’s search APIs.  I see two kinds of problems quite regularly:

• Flickr API searches sometimes return 0 results.
  As best I can tell, they just suck at the kinds of complex queries I’m throwing at them, and if they can’t answer in-time/without-errors, they just return no results.  All I can do, as a result, is try to simplify the queries.
• Flickr API searches sometimes take too bloody long.
  This is fairly lethal, and it’s forcing me to implement retries in the client in ways I’m not really happy with. When all this code lived in the client, it didn’t really matter how long things took; I didn’t have or need HTTP timeouts for individual requests.  On AppEngine, I’ve got a hard upper bound.

Retries on the client can deal with the 500s the service will (regularly) throw due to Flickr’s API performance, but it sucks to have to do it at all.  As for the no results… we’re going to occasionally cache the missing data, leading to results volatility.

I wish I had a good solution.  Hell, I wish I had a *better* solution.