I love Google but its inferred demographics are mediocre at best

I love Google. Its logo is emblazoned on my second favourite coffee cup. I use its services a lot – probably as much, if not more than any other brand I interact with.

I also love shiny new things, and I’m a market researcher, so when Google Consumer Surveys launched on March 29, 2012, I was immediately drawn to it.

Faster Horses has used Google Surveys successfully for client projects, to gauge public opinion on topics like the same sex marriage plebiscite, and for our own edification.

Its use of inferred demographics – it guesses your age, sex and location based on a KFC-style recipe of 11 secret herbs and spices –saves us researchers from asking an additional 3 questions. A win-win for researchers and the poor respondent who is spared the pain of selecting the relevant box to yet more questions.

But how accurate are these inferred demographics?

Faster Horses ran a pilot study this week to find out.

The method is simple enough: set up a survey on Google Surveys platform (as it is now know) simply asking age, sex and location (state), enter your credit card details, and away you go.

The first thing we noticed was that this survey took longer to collect responses than studies we have run in the past. Why? Well that’s as mysterious as the algorithm Google uses to infer the demos – and customer support can shed no light on the issue. Suffice to say, it took longer than we expected, by some margin.

Anyway, let’s cut to the chase. How well do Google’s inferred demos match what people say?

It’s ok on gender …

For sex, the answer is ‘pretty well’. Of those Google flagged as female, 79% reported that they were in fact female. For males, Google did even better, correctly identifying 88%.

Not so good on age

Next, age. I’ve always wondered how Google goes about teasing apart the middle-aged decrepitude of a 54 year-old from the full-blown pessimism of a 55 year-old. Turns out, however it does it, it’s not very effective. Of those identified by Google as 45-54, just 44% said they were in fact in that age group. For those Google identified as 55-64, even worse…33% self-identified as being of that age.  And for those aged 65+, Google correctly identified an astonishingly poor 12% correctly – much worse than by chance alone. (Admittedly Google may not care about seniors, since advertisers all too frequently ignore this group despite its vast spending power.)

A tad better on location

Google fares a little better at inferring which state people live in – with accuracy ranging from 59% in Victoria to 80% in Queensland. I guess all those searches for XXXX are a clear giveaway.

Further reading on the subject concurs

To be fair to Google, its accuracy compares pretty well to third party data providers, who often struggle even with gender, as Dr Augustine Fou has pointed out in a recent article in Forbes magazine.

Plenty of interested parties have looked into the accuracy of Google’s inferred demographics, from market research industry publications , to academics, and data companies. The typical finding is that Google’s guess at your sex is pretty good, but that inferences on age and location are not so hot. And that is largely what we found in our pilot study.

Google’s own white paper on how Google Surveys works, and how inferred demographics are used can be found here

It reads rather more like an advertorial and ‘how to’ guide than a true white paper. It provides no estimate of accuracy of its inferred demos, but implies a level of precision that doesn’t really exist with a heading like “How we know age, gender and region”.

So what does this all mean?

Well of course the importance of this depends on why you need Google’s inferred demographics in the first place. 

For research purposes, if accurate demos are important to you, you can always include them in your Google Surveys questionnaire, although that leaves you just a handful of other questions that you can ask, due to Google’s punitive limit on number of questions.

If you are an advertiser spending large sums targeting on age and location, then you may want to think a little harder about a contextual buy (which it could be argued is pretty much the same thing, since Google makes its inferences partly on sites people visit), or go the ‘spray and pray’ option of a gen pop media buy. It will definitely be cheaper, and the money saved on targeting can buy you extra reach.

For more information and views on this topic, contact Peter Fairbrother on peter@fasterhorses.consulting or +61 402 958 615. We’d be very happy to share the results of our research on research further.