AdSense Dashboard 2.6

/

I’ve been pumping out dashboard changes for the last few days, based on feedback, bug reports, and general email discussions with users. Keep them coming - if you use the app, I want to hear from you.

Users on Eclair/Froyo got bit by a pretty nasty bug in the HTTP stack built into the phone; the common fix appears to be to not use that API on those platforms, so that’s what I’m doing.

New stuff:

  • Support for disabling auto-refresh.
  • Bugfixes for Eclair/Froyo. If it’s not fixed for you, I want to hear about it.
  • Device-specific compatibility bugfixes around currency and rounding.
  • Google TV support. This is a bit slapdash - I’ll improve it as I improve the tablet layout.
  • New widget (released as part of the 2.5 chain) by a contributor/coworker.

There’s a lot of activity going on this weekend and early next week; I intend to do a lot of ‘remastering’ of the numeric portion of the display to show more comparative data - today versus yesterday, today verus one week ago, this week verus last week, this month versus last month, etc., for a broader set of metrics than we currently include.

That will also be extracted into widget form - choose an arbitrary 1x1 fact.

Lifetime spend, custom channels, and url channels will also show up, in all probability.

Your feedback isn’t just welcome - it’s wanted. Ping me with your gripes, comments, suggestions, and requests.

AdSense Dashboard Released

/

I’ve released a new (free) app to the market for AdSense publishers to track their performance. AdSense Dashboard is exactly what it sounds like - a heads-up dashboard for Android devices that shows vital statistics about your AdSense account.

Over time, I’ll be adding features designed to fulfill a specific need: the need to know, at any moment, what’s happening to your revenue and traffic. We’ll be adding new facts, improving the quality and usefulness of the default report, pushing the data table onto more layouts, extending downwards to support smaller devices, and adding widgets so that you can get the most important information on the home screen, to be reviewed hundreds of times a day.

Check out Adsense Dashboard the Android Market.

In Search Of Quality

/

Well, we’re getting back on track.  I’ve spent some time trying to return us to our previous levels of quality, and enable us to move forward on adding longer blacklists again.

I believe I’ve been able to return us to the previous state of affairs successfully, now - but the cost isn’t cheap.

The solution depends on the fact that we do have a known sort order for all returned results - interestingness.  If we assume that interestingness, as a score, is stable and not random, that should mean that our query can be split not only by positive term, but by negative terms, and taking the intersection of results for a particular positive query term; if a photo doesn’t appear in all of the split queries for a positive term, then it must contain one of the negative terms, and can be discarded from the result set.

This means, again, we’re doing a lot more queries. Specifically 6 locations * 2 years * # of positive search terms * (# of negative search terms / 9) ~= 1.5k Flickr API requests per search result as a worst case. (This used to be a lot higher - I’m having to bring down the search range in order to allow for more queries per positive term.)

Now, we do a lot of caching along the way whenever possible, because, again, this is a free service on freebie quota, but that’s a lot of outbound API calls per inbound user request.

Hopefully things look better again.

Well, that was fun. Not.

/

Cataclysm.

  • A raft of exceptions started turning up with geolocation and weather. Turns out Yahoo’s weather service isn’t as smart as I thought it was, and outside of major city centers it can be difficult to tell at exactly which level weather is mapped in at, if anywhere. We now do weather feed pulls at levels 13, 11, 9, 6, and 3 in the hopes of finding something useful. If you live somewhere other than a major city, chances are your weather just got more reliable.
  • If we can’t find weather, we’ll return weather conditions for unknown. I think that means more people will see volcanoes. I think I’m okay with that.
  • We were previously directly calling the memcache API, which sounds nice on the surface until you realize that actually it’s quite unreliable as a cache (within the wider definition of ‘reliable’) and that you really ought to wrap it in a bit of cache management. I’ve done so, so that when problematic keys come up, they get discarded, and we silently ignore fails to update caches (as it’s not fatal, or at least, ought not to be.)
  • We’re doing so many more searches that the search results’s size basically exceed the storage limits on a memcache key. I’ve split that out into caching search results for each search, instead of the aggregate search set, to get better fan-out; it also means we’re doing an aggregate query for cached results across multiple keys simultaneously, which, due to magical google infrastructure, should improve performance for search caching.

Anyways, that’s what I get for getting too eager; I could easily have canaried it and paid a more watchful eye, but didn’t, and pushed it along with the new client binary all at once. While most people might only have seen a sluggishness in updates, I saw…. I think it peaked at about 10QPS of errors by midnight, due to a sea of broken shit that I didn’t test properly before I stuck it in production.

We’re back down to our more normal 1QPS of traffic, and our normal error rate (mostly due to Flickr taking too long to perform even these simpler searches - we’ve still got an upper bound on request fetch time of 10s), that client retries magic away.

More improvements, probably, soon.

Weathrman 4.0: By Popular Demand

/

I’ve got a shiny new Xoom, and now you have a shiny new Weathrman client.

The biggest changes here are all around two user stories I’ve heard more than anything else:

  • I installed the app, but it doesn’t do anything and I can’t find it.
  • I need more control over how the application uses data on my device.

To address the former, we’ve got a launcher, dialogs on preview, and an embedded usage guide to help the user get the live wallpaper selected on their devices.

To address the latter, we’ve now got a pretty good story on data controls:

  • We now obey background data settings. If you turn off background data support, you can still refresh the wallpaper manually, but all automated, non-interactive network activity is stopped.
  • Support for disabling use of 2G/3G and WiMAX networks, for those who want wifi-only.
  • Control over walllpaper refresh rate.

Now that we’re allowed a lot fewer search terms, we’re doing a lot more fan-out of requests to flickr; a single user can now result in about 5,000 flickr requests, fired off in under 30 seconds. The pressure is on to find better search terms and filter terms, and so our next steps will be to take advantage of the prediction API and start using machine learning to make better decisions.

Weathrman Search Quality: A Progress Report

/

I’m in the process of doing some server-side results joining, to reduce the complexity of our queries.  The upside of this is that I should be able to, over time, bring us back up to our previous quality of searches.  In fact, we’ll actually be slightly better off - performing a series of decomposed searches will ultimately result in more search results to process, as we’re doing more queries and getting more results.

This mean the load on the flickr API - and the load on my server - are going up.

I’m still in the awful position of not being able to charge for this, having made it free once - so I still want this to fall within freebie quota if at all humanly possible.

Last, I’ve got a shiny new Xoom, and I’ll be spending some time on the client shortly getting some basic changes into the UI to make it slightly prettier when used there.  I’ve already made some server-side tweaks to broaden the minimum image size so that Xoom resolutions get more Flickr results.  The biggest problem at the moment is that there just aren’t many large image sizes fetchable via the API, and the images that do come back are far below the 1920x1408 background image size that a xoom wants.

More soon.

24 Hours of Weathrman Search Failures

/

So, around 24 hours ago, searches began failing for the thousands of people running Weathrman.  I was notified by a user a few hours ago that he began seeing strange things happening.

Flickr has made a drastic change to its API: No more than 10 OR terms are allowed in searches.

This is pretty lethal to us; we were doing, in some cases, 15 or more OR queries to select photographs for inclusion, and another 30 or so negative queries to strip away porn, keg parties, musicians, concerts, indoor photographs, you name it; we relied heavily on the ability to provide complex queries to their engine for processing, and managed a comprehensive set of allow and block filters that we used to get the set of matching results from their service.

This leaves us in an odd quandary, and so I have to begin by apologising:  I’ve had to remove most of our keywords from our queries.  You’re going to see a lot more irrelevant crap, now, and there’s not much I can do about it…

Other than doing one of two things:

1) Stop asking flickr for photos, and move to someone who has a more comprehensive (and less comprehensively restrictive) API for performing queries.

2) Build a parallel database of flickr photos by exporting and indexing everything myself.

Option 1 might be possible, but is a long way off; and there aren’t databases of geocoded searchable photos that are easy to get to dotted all over the web.  There just aren’t many places to go.

Option 2 isn’t feasible without charging a lot more money than FREE.

We could be reaching the end of Weathrman’s useful life.  If I can’t resolve search quality issues, I may be forced to take it down; we’ll have to wait and see.