Weathrman is no more...

Weathrman was built around a simple idea; search Flickr for a photograph taken near you, showing weather conditions and time of day roughly similar to your current location's weather conditions and time of day.

For the most part, it worked surprisingly well, given that I had limited API capabilities from Flickr, time-of-day is badly managed in images, most of Flickr's images weren't geocoded well, the API is horribly slow, and althoiugh we start searching hyperlocal and step upwards until we find something, each of those queries is run sequentially (due to both limited number of queries per day to Flickr and the cost of AppEngine to me).

View fullsize

It's been many years since I seriously looked at the codebase; the app still worked, but the quality of the images it pulled had steadily decreased, people aren't really using Flickr like they used to.

The net effect is that the app was old (written for Froyo, and last updated in 2011), had bad image choices due to limited API control... and for a while, was actually costing me money to run, as it had enough users to hit AppEngine's limit on free CPU time.

But no longer; it's fallen out of use, gets few requests. The app has bitrotted, people have moved on from being endlessly fascinated with live wallpapers, and there's no point leaving it up.

So down it comes, three years after my last update.

If it brought you joy, thank you; if it brought you tears, I apologize.

In Search Of Quality

Well, we’re getting back on track.  I’ve spent some time trying to return us to our previous levels of quality, and enable us to move forward on adding longer blacklists again.

I believe I’ve been able to return us to the previous state of affairs successfully, now - but the cost isn’t cheap.

The solution depends on the fact that we do have a known sort order for all returned results - interestingness.  If we assume that interestingness, as a score, is stable and not random, that should mean that our query can be split not only by positive term, but by negative terms, and taking the intersection of results for a particular positive query term; if a photo doesn’t appear in all of the split queries for a positive term, then it must contain one of the negative terms, and can be discarded from the result set.

This means, again, we’re doing a lot more queries. Specifically 6 locations * 2 years * # of positive search terms * (# of negative search terms / 9) ~= 1.5k Flickr API requests per search result as a worst case. (This used to be a lot higher - I’m having to bring down the search range in order to allow for more queries per positive term.)

Now, we do a lot of caching along the way whenever possible, because, again, this is a free service on freebie quota, but that’s a lot of outbound API calls per inbound user request.

Hopefully things look better again.

Weathrman 4.0: By Popular Demand

I’ve got a shiny new Xoom, and now you have a shiny new Weathrman client.

The biggest changes here are all around two user stories I’ve heard more than anything else:

• I installed the app, but it doesn’t do anything and I can’t find it.
• I need more control over how the application uses data on my device.

To address the former, we’ve got a launcher, dialogs on preview, and an embedded usage guide to help the user get the live wallpaper selected on their devices.

To address the latter, we’ve now got a pretty good story on data controls:

• We now obey background data settings. If you turn off background data support, you can still refresh the wallpaper manually, but all automated, non-interactive network activity is stopped.
• Support for disabling use of 2G/3G and WiMAX networks, for those who want wifi-only.
• Control over walllpaper refresh rate.

Now that we’re allowed a lot fewer search terms, we’re doing a lot more fan-out of requests to flickr; a single user can now result in about 5,000 flickr requests, fired off in under 30 seconds. The pressure is on to find better search terms and filter terms, and so our next steps will be to take advantage of the prediction API and start using machine learning to make better decisions.

Weathrman Search Quality: A Progress Report

I’m in the process of doing some server-side results joining, to reduce the complexity of our queries.  The upside of this is that I should be able to, over time, bring us back up to our previous quality of searches.  In fact, we’ll actually be slightly better off - performing a series of decomposed searches will ultimately result in more search results to process, as we’re doing more queries and getting more results.

This mean the load on the flickr API - and the load on my server - are going up.

I’m still in the awful position of not being able to charge for this, having made it free once - so I still want this to fall within freebie quota if at all humanly possible.

Last, I’ve got a shiny new Xoom, and I’ll be spending some time on the client shortly getting some basic changes into the UI to make it slightly prettier when used there.  I’ve already made some server-side tweaks to broaden the minimum image size so that Xoom resolutions get more Flickr results.  The biggest problem at the moment is that there just aren’t many large image sizes fetchable via the API, and the images that do come back are far below the 1920x1408 background image size that a xoom wants.

More soon.

Sick days, Weathrman, and you

So I’m home sick today, and going through the laundry list of things I haven’t gotten around to recently.

Weathrman is on that list.

I’ve pushed a new version of the server that performs searches on behalf of app users up.  This is designed to do two things:  reduce the cost of running the service, and improve the search quality.

In order to reduce the cost, I’m going to do the obvious thing:  I’m going to produce fewer searches.  Specifically, that means:

• Instead of asking for a 3-hour window, we’re asking for a 4-hour window around ‘now’.
• Increase the search window from 3 months to 4 months (2 in either direction)
• Fetch 4 pages of results instead of 5.
• Reduce the minimum number of results needed at any search level from 5 to 3.
• Remove street-level searching (level 16) - it too rarely has results, and just creates latency.

 This means our worst case goes from being 7 tiers of 5 parallel searches to being 6 tiers of 4 parallel searches.  It also means the average case, which cost 1 tier of 5 searches before we were likely to find any results, will now get faster, and I won’t be paying for that wasted time.

The downside is that I’m only fetching four pages of results; searches cover the entire four-month window, and it’s possible that none of the first four pages will have photos taken within the required time-of-day range; if that happens, we skip up a level.  Dropping one of the pages of search results makes that more likely, and it’s probably not completely mitigated by the widening of the 3-hour window to 4.  The net effect is that searches may feel… a little less local.

Hopefully, these changes mean I’ll be able to continue supporting this service for longer, and more cheaply.  Going back to the days when the client was responsible for performing all these searches just isn’t going to happen - it’s just too convenient to be able to run the searches from the server, and much more reliable.  It does mean that I’m bearing the cost of a free app - but as long as I can keep the costs down, I don’t mind.

Weathrman on the Galaxy Tab

v3.5.2 is out, with fixes for the Galaxy Tab.

Weathrman and Android Tablets

I thought I’d mention that I’ve tested on a Samsung Galaxy Tab, and the results are pretty damn good - the app works beautifully, adjusts to the new resolution by asking for larger images from Flickr automatically, and generally looks pretty damn hot.

I peed a little.

Weathrman 3.5.1: The Best Intentions

I’ve taken some time to do some internal cleanup; I’ve learned a great deal about Android’s programming model from building this and other projects, and it’s time to go back and re-apply some of that knowledge.

As such, I’ve restructured the bulk of execution in IntentService implementations, which has simplified a lot of the code; notification handling and update processing are now handled by dedicated services, rather than burying the whole processing chain in an AsyncTask.

We’ve lost a few things I’d like to have kept along the way, on the inside, but the code is cleaner and faster for it.

I’ve also done something I thought I’d never see myself do:  I’ve de-guiced the app.

Done well, I’m sure Guice is a good fit; but it’s a very small app, and moving to IntentService instances has basically meant that I’ve now got three independent application-parts that communicate by sending intents to one another.  What’s left isn’t worth Guicing.

What’s funny, and somewhat unfortunate, is that by coming to this with my server hat on, I missed the fact that Android applications don’t need to be built as the monolith binaries that Guice is designed to help break apart, and some of the tools at your disposal on android are designed to ensure that not only do you not need to allocate those objects, you can avoid most of the processing time you would have spent even thinking about those elements of the object graph - binding processing just isn’t free, and on mobile devices you might not see the cost, but you can feel it, almost imperceptibly.

So 3.5.1 is out; it’s got fixes for the cache growth problems, reaffirms my commitment to 2.1-based devices, and provides a much needed speed boost in both startup performance and, due to some more tweaking on my part, on the speed of image retrieval in many cases.

I’ve also tweaked the negative keywords associated with daytime searches on the server side; hopefully you’ll see improved search quality.

And for a short time, it’s free - I’m looking to get an idea of how active users correspond with server load, and giving away a free app is a cheap way of load testing.  :)