The Basement

Umbria Communications continues to listen. Ever since I pounced on Umbria for being a marketing bully a couple of months ago, folks from Umbria have been telling me that they appreciate my feedback and that they are taking it to heart. Lots of folks say that sort of thing to quiet a detractor, but Umbria is doing something remarkable... they're following through on the promise. A few weeks ago, I talked about Umbria VP Bill Tuohig and Umbria's new website.

Since then things have gotten even better on the Umbria site. They've got more case studies, white papers, and product details. But the most valuable new piece of info is a sample excerpts from a Buzz Report. You have to fill out a request form for the PDF report be mailed to you or else I'd link to it. While the sample report is genericized (Brand A, Brand B, etc.) it does an excellent job of illustration exactly the type of information a customer might expect. There is one graph to illustrate each major feature as well as some supporting analysis. Well done Umbria.

Of course, I'm a skeptic. There are two points in the sample report that I'm suspicious of: the analysis and the demographic profiling. I don't think either of these are show stoppers but they're the type of thing that I would want to scrutinize if I was gonna spend $100K with an analytics firm.

Is the Analysis Automatic and/or Scalable

The "analysis" I'm concerned with is a brief text description of the trends illustrated on a give graph. For example, under a graph that illustrated Umbria's Favorability Trend Analysis feature, the following appears:

While positive comments about Brand A were down, negative mentions of the brand also declined, as bloggers who had posted in previous weeks about the service outages they experienced did not mention this issue as frequently in their blogs after their service was restored (week ending 3/6).

Now at first blush, this analysis looks similar to what one might expect in a summary of traditional survey tracking study. This is a good thing. So what's my concern? Well, I don't think that this sentence was written by a machine. There's nothing wrong with that except that Umbria claims that they are offering higher value by trying to automate more - to go further with technology than their competitors. My perception is that Umbria see themselves as providing data while the analysis is left up to the client. This impression was echoed when I talked with year long Umbria customer Shawn Conly of Electronic Arts. So my question is this - if this analysis is done by humans, is it scalable? Can Umbria provide this level of detailed human analysis for every customer?

Don't get me wrong, the data is there. The analysis is possible. But Umbria's focus has been on technology and not human analysis so I'm not sure I'd expect a lot of this from Umbria. While the report is genericized, it is definitely about mobile phone use which is the same application as the widely publicized WPP, G-Whiz US Celluar thing. Umbria's management has a strong background in telecomm which would lend itself to producing this type of analysis. So another question might be - is Umbria even capable of this type of analysis in a vertical in which they don't have specific domain knowledge?

How Accurate is Demographic Profiling?

I'm pretty convinced that fairly reliable demographic profiling is possible using a variety of analytic techniques, especially when looking at lots of posts and aggregate data. Furthermore, I think that Umbria has done more than anyone to turn this technology into serious usable CGM research. That said, I'm still very skeptical of the accuracy and their sample report has added fuel to my skepticism fire. Why? Well, consider this graph from their sample report:

This graph shows people who mentioned Brand A during the week ending March 13, 2005. Now, Umbria defines Gen Y as ages 15 to 27; Gen X as 28 to 40; and Boomers as 41 to 60. So, there are slightly more women ages 15 to 27 discussing cell phone brands then there are men in the same age group. Okay, I'll buy that. But then when women turn 28 they suddenly stop talking about cell phones and there are as many men ages 28 to 40 talking about Brand A as there are men and women ages 15 to 27 combined? If I were a survey researcher, we'd be going back to the field because something is really wrong here... I think. Fact is - we don't know. Maybe I'm wrong and online behaviors between ages groups are just that much different - but I have trouble with this idea. Younger women are more tech savvy than older ones. Yes. But to the tune of 20 to 1? I think that's a stretch.

But maybe this is because we're talking about cell phone? Look at this Umbria demo graph of President Bush mentions from the week ending Nov 7, 2004

The proportions are very nearly the same. (Well, Gen X and Boomer males are flipped, but still...) Does it seem reasonable that the demographic distribution of online posters mentioning politics is that same as those discussing cell phone features? Seems a little fishy to me. Do women over 27 just not use the internet? My personal hypothesis is that female voices are harder to detect as they get older (and wiser?).

So how do we quell the skepticism? I think analytics firms like Umbria need to partner with traditional MR firms and match up data - for example between technology adoption by demographics from traditional survey data versus online mentions by demographics from text analytics. Of course mentioning online is different from adoption but I would think there would be some correlation. You could even do a survey where you ask if you've posted online about your cell phone. Until text analytic demographic profiling is validated as a market research methodology (and not just computer science) it should be read with health skepticism if not a treated as voodoo.

It's also worth noting that my gut feel is that demographic profiling is probably good enough to isolate general sentiment within a particular demographic group - especially for trending. But I think comparing sizes of demographic groups as in the above pie charts is probably misleading. A lot of market research is about finding what number are meaningful and this stuff is no different.

The general message here is, proceed with caution, but proceed. Umbria has done a lot to try to explain their offering and techniques in greater detail and they've done a great job. The more information like this that analytics vendors can provide, the quicker the larger MR community can have these types of important discussions to figure out what to do with this exciting new data. I can't over emphasis the important of transparency here. I think the more that CGM analytics firms can talk responsibly about accuracy and error, the more seriously the research will be taken by traditional MR and marketing folks. Where do these numbers come from? What do they mean? Can we rely on them? These questions all have to be answered before the data can truly become actionable.