In Search Of Quality

Well, we’re getting back on track.  I’ve spent some time trying to return us to our previous levels of quality, and enable us to move forward on adding longer blacklists again.

I believe I’ve been able to return us to the previous state of affairs successfully, now - but the cost isn’t cheap.

The solution depends on the fact that we do have a known sort order for all returned results - interestingness.  If we assume that interestingness, as a score, is stable and not random, that should mean that our query can be split not only by positive term, but by negative terms, and taking the intersection of results for a particular positive query term; if a photo doesn’t appear in all of the split queries for a positive term, then it must contain one of the negative terms, and can be discarded from the result set.

This means, again, we’re doing a lot more queries. Specifically 6 locations * 2 years * # of positive search terms * (# of negative search terms / 9) ~= 1.5k Flickr API requests per search result as a worst case. (This used to be a lot higher - I’m having to bring down the search range in order to allow for more queries per positive term.)

Now, we do a lot of caching along the way whenever possible, because, again, this is a free service on freebie quota, but that’s a lot of outbound API calls per inbound user request.

Hopefully things look better again.

Update on Sunday, May 22, 2011 at 7:18PM by Gregory Block

I’m watching the request logs go by and looking at the results of some searches by hand to verify that quality is back to normal, and starting to try and pick up additional queries based on problems seen in Buffalo NY (USA), London (UK), and Brussels (Belgium).

If you see something awful, I’m listening.

Weathrman 4.0: By Popular Demand

I’ve got a shiny new Xoom, and now you have a shiny new Weathrman client.

The biggest changes here are all around two user stories I’ve heard more than anything else:

• I installed the app, but it doesn’t do anything and I can’t find it.
• I need more control over how the application uses data on my device.

To address the former, we’ve got a launcher, dialogs on preview, and an embedded usage guide to help the user get the live wallpaper selected on their devices.

To address the latter, we’ve now got a pretty good story on data controls:

• We now obey background data settings. If you turn off background data support, you can still refresh the wallpaper manually, but all automated, non-interactive network activity is stopped.
• Support for disabling use of 2G/3G and WiMAX networks, for those who want wifi-only.
• Control over walllpaper refresh rate.

Now that we’re allowed a lot fewer search terms, we’re doing a lot more fan-out of requests to flickr; a single user can now result in about 5,000 flickr requests, fired off in under 30 seconds. The pressure is on to find better search terms and filter terms, and so our next steps will be to take advantage of the prediction API and start using machine learning to make better decisions.

Weathrman Search Quality: A Progress Report

I’m in the process of doing some server-side results joining, to reduce the complexity of our queries.  The upside of this is that I should be able to, over time, bring us back up to our previous quality of searches.  In fact, we’ll actually be slightly better off - performing a series of decomposed searches will ultimately result in more search results to process, as we’re doing more queries and getting more results.

This mean the load on the flickr API - and the load on my server - are going up.

I’m still in the awful position of not being able to charge for this, having made it free once - so I still want this to fall within freebie quota if at all humanly possible.

Last, I’ve got a shiny new Xoom, and I’ll be spending some time on the client shortly getting some basic changes into the UI to make it slightly prettier when used there.  I’ve already made some server-side tweaks to broaden the minimum image size so that Xoom resolutions get more Flickr results.  The biggest problem at the moment is that there just aren’t many large image sizes fetchable via the API, and the images that do come back are far below the 1920x1408 background image size that a xoom wants.

More soon.

Sick days, Weathrman, and you

So I’m home sick today, and going through the laundry list of things I haven’t gotten around to recently.

Weathrman is on that list.

I’ve pushed a new version of the server that performs searches on behalf of app users up.  This is designed to do two things:  reduce the cost of running the service, and improve the search quality.

In order to reduce the cost, I’m going to do the obvious thing:  I’m going to produce fewer searches.  Specifically, that means:

• Instead of asking for a 3-hour window, we’re asking for a 4-hour window around ‘now’.
• Increase the search window from 3 months to 4 months (2 in either direction)
• Fetch 4 pages of results instead of 5.
• Reduce the minimum number of results needed at any search level from 5 to 3.
• Remove street-level searching (level 16) - it too rarely has results, and just creates latency.

 This means our worst case goes from being 7 tiers of 5 parallel searches to being 6 tiers of 4 parallel searches.  It also means the average case, which cost 1 tier of 5 searches before we were likely to find any results, will now get faster, and I won’t be paying for that wasted time.

The downside is that I’m only fetching four pages of results; searches cover the entire four-month window, and it’s possible that none of the first four pages will have photos taken within the required time-of-day range; if that happens, we skip up a level.  Dropping one of the pages of search results makes that more likely, and it’s probably not completely mitigated by the widening of the 3-hour window to 4.  The net effect is that searches may feel… a little less local.

Hopefully, these changes mean I’ll be able to continue supporting this service for longer, and more cheaply.  Going back to the days when the client was responsible for performing all these searches just isn’t going to happen - it’s just too convenient to be able to run the searches from the server, and much more reliable.  It does mean that I’m bearing the cost of a free app - but as long as I can keep the costs down, I don’t mind.

Weathrman on the Galaxy Tab

v3.5.2 is out, with fixes for the Galaxy Tab.