The Basement

A new player has popped onto my blog analytics/WOM radar: Umbria Communications.

If you Google Umbria Communications (don't bother Googling "umbria" because you will get stuff about vacations in Italy, the folks at Umbria Communications should read my Branding for Analytics post) you will get lots and lots of hits – mostly business, venture capital and marketing press. For the most part they all say the same thing (paraphrased) – Umbria Communications is a market leader of online consumer content analytics and their methodology is better than anyone else in the industry. While I'd love to believe their marketing story, as far as I can tell, it's hogwash – or at least what Hugh McLeod calls dinosaurspeak.

Umbria Communications first showed up on my radar when Jonathan Carson of BuzzMetrics blogged about them recently. It seems that Umbria has been busy flooding the press with their spin while simultaneously throwing stones at true market leaders such as BuzzMetrics and Intelliseek. Jonathan points out that Matthew Hurst of Intelliseek has also weighed in on this here. Both Jonathan and Matthew have been very diplomatic about this – offering sincere counterpoint that describes the actual philosophies, services and products of their prospective organizations. Thus far this has gone unanswered by Umbria and the press and it hasn't gotten much traction in the blogosphere.

So I thought I'd take up the torch – after all "[Umbria is] more focused on the notion of the 'long tail.'" according to Dave Howlett, VP Product Management at Umbria. So I'm sure they'll read this and respond (he said sarcastically) after all, I am the Long Tail.

What's is Umbria's Story?

Seth Godin tells us that stories work better than fact, they're more persuasive – especially when they fit our desired context. I'm reasonably certain the kids over at Umbria have read Seth's books, because they are GREAT story tellers. So good in fact, that lot's of folks believe them. For example see here, here, here, here, here, here and here.

In contrast, Umbria's website is surprisingly thin – no case studies, no sample deliverables, no management team bios, no press contact, and only the lightest discussion of their research methodologies. And oh, NO FREAKIN' BLOG! What is with these companies that want to sell products based on the power of the blogosphere, but don't blog themselves? There is, however, a list of their 4 member "Advisory Board" – an impressive list – all PhD's with backgrounds that include phrases like congnitive science, natural language processing, computational linguistics, psycholinguistics and automatic speech recognition. This, as it turns out, is a crucial component of Umbria's story.

So what is that story? Well, let's them tell it. After all, they do such a good job...

Umbria Communications collects the unsolicited opinions and perceptions of the online community (blogs, message boards, chat, etc.) and turns it into actionable market research about companies, products, people and topics of interest.

So far so good. This is what their competitors are doing. This is reasonable and plausible.

Using proprietary natural language processing and machine learning technologies, Umbria dissects the who, what and why of online opinion to provide deep insight into the buzz about you, your products and your competition.

Wow, this sounds different and almost unbelievable – wouldn't you need a bunch of PhD's for that? Oh yeah, they do. This sound interesting, I want to know more, perhaps they have a white paper discussing their methodology? In fact they do, but you have to provide your email address to get it a copy. I already have my copy. It's 4 pages long including the cover page and let me tell you, for a bunch of PhD'd academics, this is the weakest white paper I've ever read. On data collection it says:

Umbria collects data from various sources on the Internet. While the company is agnostic in the data it collects and analyzes, the common elements in all information collected is the unsolicited, unbiased and timely human opinion of the nature of the data.

Umbria collects data from various online sources including weblogs, message boards, chat, opinion web sites, Usenet groups, company suggestion and / or comment inboxes, etc.

...the first step in ensuring the sources of data collection match the client's desired objectives.

Once optimal sources have been identified... Every few minutes, the company collects all new postings from more than 10 million weblogs.

Impressive, they must have some killer bandwidth to check 10 millions blogs every few minutes, but I digress. On data analysis they say...

Data analysis is broken into the following two major categories: (1) Speaker Analysis: the analysis of the individual posting his or her opinions and (2) Content Analysis: The analysis of content and context of the postings.

The gist here is that – by using artificial intelligence techniques – Umbria can determine the demographics of the poster (speaker analysis), whether the match on a brand or product name actually refers to the brand or product (context) and if mentions are positive of negative (content). Amazing. Don't we all want to believe this story? Don't we want it to be true? Survey research has just been rendered obsolete. Umbria has invented machines that understand all of the citizen created opinions on the Internet – not just blogs – and it collects all of the new ones every few minutes!!

Digging through the Pudding

The proof, they say, is in the pudding. So, let's dig through the pudding – only there's precious little. I haven't found any serious case studies, or samples of their deliverables. Umbria's PR firm, GroundFloor Media, Inc. of Bolder Colorado, has done an excellent job of ensuring that we all know that Sprint, Electronic Arts, U.S. Cellular and Logitech are all customers of Umbria – although I've seen no quotes from any representatives of these clients. WPP Group used Umbria for their work on an U.S. Cellular teen campaign. They used the research to determine that teens would prefer unlimited minutes to limited minutes. Amazing, who'd of thought? This flurry of press was no doubt the work of WPP Group PR – "hey look at us using the blogosphere for youth trends!" - an not that of U.S. Celluar.

So where's the data? What can we scrutinize to validate Umbria's methodology? Well, in another piece of clever marketing, Umbria has swung a deal with CMO Magazine to provide them with a weekly Blog Meter report. One of the earlier Blog Meter reports (maybe the first, I can't find an archive) covered fast food. This was discussed in the blogosphere on PR Machine and on Dava Vanden Heuval's excellent marketing blog. The fast food report was presented with this graph

The accompanying text failed to mention the base of theses statistics. After all, isn't Umbria's specialty determining demographics? What about context? Are these mentions about the food or the place or simply any mentions? Hmmm. Who knows? But this is interesting none the less and it's believable, it's a good story. But think back to the evening news during this time period. Do you remember any mentions of any of these companies? Maybe this BlogPulse Trend graph will help jar your memory...

Yes, that's right, March 24 was when the "finger in the chili at Wendy's" story broke and stayed in the headlines – and in the blogosphere – through the end of May. But the Umbria graph shows no substantial change in the buzz for Wendy's, and no substantial difference between Wendy's buzz and that of the other restaurants (other than McDonald's of course).

And what's the dip in post volume for the week of 5/8? This is probably a data collection problem – either Umbria's or ping servers or an upsteam provider – but there's no mention of that either. Another interesting point is that Umbria claims they are "identifying hundreds of thousands of new blogs each week." Hmmm. Hundred of thousands of new blogs per week yet the total posting volume about fast food restaurants declined over a 10 week period? Huh! Do you need a PhD to do that complex statistical calculation? Something doesn't add up here. (As a note, these two issues – technical problems and blogosphere expansion – are the reason BlogPulse and IceRocket show their time analysis products in percentages of total post volume, instead of the raw number of posts. This is a double edge sword for Umbria because they would have to answer the questions "Percentage of what?" - I have yet to see a sincere attempt to adequately explain their "proprietary algorithms".)

For a company who's PR machine is working overtime yet providing such little actual data to the public, you'd think Umbria would review their data a little closer – and provide some data that was a little more compelling. The point here is that there is supposed to be new insight in blogs – that facts that teens would rather not pay for minutes and that McDonald's is mentioned more than other fast food chains are hardly groundbreaking. Compare this will BuzzMetrics press release that nearly predicted the bankruptcy of the Atkins Nutritionals, Inc. - this is groundbreaking! Any part-time blogger using free tools like BlogPulse can make more groundbreaking observations than I've seen from Umbria. And shame of CMO Magazine for publishing this garbage.

Don't Look at the Man Behind The Curtain

So what gives? Who is Umbria? Where'd they come from? What are they trying to do? Hmmm. It's pretty easy to find that Umbria opened it's doors in March of 2004 – a little less than a year an a half ago – an incredibly short time to be so critical of the established players in the industry. What is a little harder to get at is who's behind Umbria – as I mentioned earlier their management team doesn't appear on their website – but it appears that it used to. I found this Umbria management team page while Googling them.

Umbria's funder and CEO, Howard Kaushansky, is not a PhD, or a social scientist or even a computer scientist. He is, or rather was, an attorney. From 1992 to 1997 he was VP of Strategic Planning, General Counsel and Secretary of Coral Systems, Inc., which developed fraud detection software for the telecommunications industry. While at Coral, Mr. Kaushansky was named as an inventor on a patent for a data mining process to predict customer dissatisfaction for telecommunications service providers using CRM system data. As the page claims "Mr. Kaushansky played an integral role in the sale of Coral to Lightbridge (NASDAQ: LTBG) in 1997." A transaction in which I would assume Mr. Kaushansky profited.

Mr. Kaushansky then co-founded a company called Athene Software along with fellow inventor Eric Johnson, a firm that provided products to mine CRM data to predict customer dissatisfaction for telecommunications service providers. Hmmmm, sounds familiar. The company was built up to 80 employees or so and was considered a to be a model engineering-friendly workplace according to this article from CIO magazine (March 2001). My favorite quote from the article is from then President and CEO of Athene, Eric Johnson:

Do [employees] want to be a part of building something, or do they just want to draft off others, hoping for an IPO in nine months? You've got to weed those people out.

Six months later, in September of that same year, Athene was sold to CSG Systems. Mr. Johnson, it appears, was weeded out. Mr. Kaushansky's bio implies that he was CEO of Athene at the time of the CSG acquisition, a notion that's enforced in the bio found here, where Mr. Kaushansky's knack for selling companies is almost bragged about. I'm guessing Mr. Kraushansky was put into (or agressively sought after) the CEO position explicitly for the purpose of handling the acquisition – whether that was with or without Mr. Johnson's blessing, I don't know.

Mr. Kaushanky's bio goes on to claims "Prior to starting Umbria Communications, Mr. Kaushansky co-founded and served as the CEO of Green Planet Software, a technology firm focused on predictive analytics for the Internet." This is a huge accomplishment, just look at the website. The domain was registered in 2001 and the page doesn't appear to be updated since 2002. There's no phone number or physical address. There's a leadership team – it just so happens to be the same one Umbria has. And there's a technical advisory board – the same one as Umbria has. Google doesn't have a single link to www.greenplanetsoftware.com and a search for "Green Planet Software" turns up the Umbria management page I linked to above, some bankruptcy document (unrelated I think) and a link to the management page for Tech Guys, a Kansas City technical support company a la The Geek Squad. David Howlett, CMO of TechGuys, formerly of Green Planet Software, formerly of Athene, formerly of Coral, formerly of Sprint – is using Green Planet Software in his bio. David – it's worth mentioning – is now supposedly Umbria's Product Management Vice President. Mr. Howlett describes Green Planet Software as a "maker of client-side personalization software" which is in contrast to Kauhanky's "focused on predictive analytics" especially when the website claims " Green Planet Software is all about revolutionizing the way you interact with the Internet, making it the most relevant experience while protecting your privacy."

I'm not sure I'd want Green Planet Software on my resume – at least not as long as their website is up.

So then, last year, Mr. Kaushanky creates Umbria Communications. Apparently, after brokering a couple of acquisitions, an ex-attorney can collect a bevy of PhDs, hire a stellar PR firm, form a company and become a venture capital darling. Google "Umbria Communications" and "venture capital" and you get lots of hits – most talking about Umbria soliciting or receiving venture capital, including this article about a scandal involving one of Umbria's investors. Not that there is anything wrong with venture capital – but we need to look at the bigger picture. Mr. Kaushanky has sold two companies in the last 10 years. He's built Umbria using lots's of the same folks he's used in the past. He's a venture capital darling. He's hired a PR firm and flooded the press with his "Umbria is the industry leader" palaver when they've been in business less than 18 months and we've yet to see a see a presentation, a case study, a serious white paper or even a scanned image of a cocktail napkin showing real and relevant results. His last "company" had no visible products and no one on the internet ever mentioned them – except for the bios of two ex-executive who are both executives of Umbria.

The point is this – I think Umbria is smoke and mirrors - over hyped dot-com-style vaporware designed to create the appearance of value so that Mr. Kaushanky and crew can sell within a year, or two at the most. This seems to be his established pattern. For Athene Software, the sale was within 4 years of founding and within a year of what looks like a focused PR marketing campaign in industry press.

Where There's Smoke (and Mirrors)...

I should say that I believe in the direction of research in which Umbria's marketing claims they are headed. I think NLP of blogs and other online content is valid, valuable and important. I think that there are probably some really smart, well intentioned folks working directly or indirectly (through Universities and such) for Umbria. These folks are probably doing some remarkable things – they just aren't doing them as well as the marketing message indicates. Why? Because NLP is a really, really hard thing to do. There's a lot of room for error. And when you're dealing with "hundreds of thousands of new blogs per week" – it's hard to human check the machines. Algorithms for English don't work for other languages. Spam strategies evolve daily. Keywords that are used for specific demographics change quickly, etc.

Umbria is measuring word of mouth – although the only place on their website that uses that phrase is in a reprint of a WSJ article. Ironically, it's word-of-mouth that they are fighting as well. I haven't done the research or spoken to someone who has – but I suspect that the internet flattens speech patterns just like it flattens everything else. When 17 year old kids from California are developing software with 50 year olds from North Carolina, I suspect that some writing patterns are transferred – both ways. When Umbria applies NLP to posts to determine demographics, they have to worry about falling off both sides of the distribution – false positives (people that talk like a demographic but don't belong – think of the rapper Eminem) and false negatives (people that belong in a demographic but don't write like it – think of Ben Casnocha, the 17 year old entrepreneur). [Speaking of Ben, Umbria could learn this lesson from him.]

You could determine the nature of the speaker by evaluating several posts in a particular blog, but it's hard to determine if there is one voice or several in a given RSS feed. The list of complications goes on, but the point is, this is really really hard stuff. (Remember the old comic – "Online, no one knows you're a dog." ) If Umbria could identify demographics of any arbitrary post even with 70 or 80% accuracy, it would be world changing and they would not be applying there technology here. They would be going after real money - think Department of Defense, Homeland Security or DOJ. I don't think they can do what they claim. And now that they've claimed it, it will be really really hard to prove because now they've created a credibility problem.

Why Do I Care?

Why am I writing this? For several reasons. 1.) If one web analytics company sells snake oil to a major consumer products company, they poison the well for other legitimate firms. This is bad for the technology, bad for bloggers and bad for the blogosphere. 2.) On their website, Umbria says "Technology is transforming markets and media so much that no company or media outlet can afford not to listen and engage in a conversation with consumers." Well said. Let's talk. This is what WOM is about. 3.) Umbria further says "In the world of the Internet, there's no such thing as controlling the media or the message." Right again! Do you guys ever read you own website? This too is what WOM is about. 4.) Umbria's marketing folk attacked Intelliseek and BuzzMetric – two of the most respected, experienced, generous and involved web analytics companies out there. Both have supported me as a random long tail part-time blogger, so I'm taking it personally. This is also what WOM is about.

Umbria Wouldn't Know the Long Tail if it Whomped 'Em

Now that you have some background information, read again what David Howlett said on ClickZ

Where some blog monitoring applications, including Intelliseek and Buzzmetrics, give more weight to influential bloggers, Umbria takes a more egalitarian approach, giving voice to the so-called "lone bloggers," whose opinions in aggregate can tell a different story than the one told by A-list bloggers.

This brings me to my last point: Umbria is not focused on the long tail; they are all about influence and they are apparently too daft to realize either.

When you are using NLP to analyze the demos of the speaker, you take the folks at the top of the distribution, less you fall off either side. That means you don't get the outliers – you get the people that talk in a voice that your proprietary machine-learning algorithms deem fitting for a particular demographic. This is no where near the long tail, it isn't even close to representative – it's more like stereotypical. (Which might be valuable by itself – so why lie?)

Furthermore, online posters are more likely influentials than the rest of folks. Lot's of folk say this – including me. But more importantly, Umbria says this on their website "Influentials represent a higher portion of the Online Community than society as a whole ". By virtue of monitoring online posters, Umbria is giving more weight to influentials – and they're advertising this fact on their website while throwing rocks at others for it in the media. And remember what they said in their methodology white paper "Once optimal sources have been identified" – they don't look everywhere, just where they think their client's customers will be talking. This is a short-tail influentials view if I've ever heard it.

What I think Umbria means here is that they've put so much value on their NLP technology that they discount the sense of connection – which is a large part of what some others consider an indicator of influence or at least a sense of community or trust. Bloggers that are link to more tend to be read more, are trusted more and therefore have more influence. Umbria's point is that if 2 millions kids in their bedrooms all post about the same thing but no one reads it, it's just as important as if Jeff Jarvis or Robert Scoble blogs something and 2 millions people read it (and lots of them subsequently link to it). This is a little like the tree falling in the woods with no one around. A more accurate statement would therefore be "Umbria applies a lot of the same techniques as others in the industry, except they discount links – because hey, that's a lot of work considering the fact they're just gonna sell in a year or two."

Brand managers, product managers, political campaign managers, marketers, PR folks and others interested in online consumer generated content mining – be skeptical, be very skeptical. Based on their history and their PR machine I suspect Umbria has got some really slick sales folk. Question everything and blog your experience. This space is about to explode (is exploding?) and we as consumers of commercial analytic services have the opportunity to hold vendors accountable for delivering on their marketing promises. Let the buyer vendor beware.