Ignorantium
More reactive than flourine. Funnier than boron.
Some more math problems for social media…

I’ve seen this post bandied about Twitter a lot today with the claim that IBM realized $4.6 million in savings thanks to tagging documents in their electronic document system. I’ve set up systems like that for companies in the past, and for a company like IBM, where knowledge is everything, making a knowledge base more useful is a big, important task. I was all set to trumpet these results as still further proof that social tactics, in this case something as simple as bookmarking, are smart. Since ROI numbers are sometimes difficult to come by, I thought this was a great find.

Unfortunately, I followed the link to the original article and found the numbers to be a little suspect. Don’t get me wrong, I’m not saying anyone is providing intentionally false numbers, but I will say the original post makes a couple of assumptions that most people wouldn’t.

Here’s the problem:  The $4.6 million in savings is calculated based on a survey of users of a knowledge repository. The users claim an average of 12 seconds saved per search. The post then multiplies those 12 seconds times the huge number of searches done every week (Over 286K–remember, this is IBM so they’re probably hitting this database all the time) to come up with 955 hours saved. That’s the first problem. Those 955 hours are made up of 12 second bits of work that it is assumed will be spent doing something else instead of watching a computer churn through a search. That’s not the way people work.

On Twitter I compared this to a company forcing employees to wear zippers on their coats instead of buttons because it saves 12 seconds to take a coat off and on with a zipper instead of buttons. A company with 1000 employees could then claim they saved $552,000 a year based on the aggregated time savings from not buttoning during the six months when coats are required. (See below for the hourly rate I used to figure that out.) That’s not even really an accurate comparison since you really could use 12 seconds not buttoning a coat to do something else. Waiting for a computer to churn through a search still requires your computer, i.e. the tool you’re using, to be occupied. But you get the point.

The second issue is that the 955 hours per week are then said to equal “roughly” $4.6  million in savings. By my calculation, that means that every person making a search is earning over $191,000. ($4.6 million divided by 955 hours x 52 weeks a year.) That’s a pretty high median income. I’m betting it’s based on the average billable hour for a consultant from IBM since it comes close to $92/hour. But that’s the problem: I have to make some assumptions to get to $4.6 million.

What is really interesting about the original post, and the savings claim they make on behalf of bookmarking, is that they mention the system they instituted the program on was not very well liked. By adding tags, standard taxonomy, etc. to it they made it more efficient and more user friendly. As I said, I’ve built things like this before and the changes they put in place are usually considered just good organization of a document system. So it seems to me they’re making the claim that doing the work right saved time over the way it had been done. To me that seems as though they are trying to use work done the wrong way as a benchmark for work done the right way. That’s like claiming you saved time by not getting lost.

As I said, I would love nothing more than the $4.6 million claim to be true, and I’m afraid it will now be a part of social media lore that gets quoted over and over. That’s too bad and won’t help with the perception that social media is not serious about numbers.

PLEASE NOTE: If I’m reading the blog posts wrong, or if the assumptions are clarified/explained somewhere, please let me know. This post has been hastily written and I’m not trying to be combative. I also do not want to imply that blogging and social bookmarking are not great knowledge-sharing tools. They are. I just don’t think they should be oversold with numbers that might not meet the sniff test.  I would love to discuss this with anyone and figure out what the numbers mean. And if no one disputes my numbers, and you use the $4.6 million in a presentation where I’m in the audience, expect to see me during the Q&A asking how they arrived at the numbers.

3 Comments to “Some more math problems for social media…”

  1. [...] This post was mentioned on Twitter by ignorantium and James Wester, Donna Brown. Donna Brown said: RT @jameswester: Thought I'd do a quick post on the $4.6 million IBM claim: "Some more math problems for social media…" http://bit.ly/dsyDP3 [...]

  2. Luis Suarez says:

    (Cross-post from original blog post…)

    Hello James! Thanks a lot for the feedback comments and for putting together such a comprehensive follow-up blog post to the one that I have put together whether you have questioned the validity of those numbers put together. Very insightful and very happy to engage on the discussion as well.
    Rawn’s blog post is by no means the original resource of where the study comes from. It’s actually been blogged and talked about extensively in the past from various other resources (Go here, here, here, here, here, here and over here, amongst several other places…).
    The original resource is actually an article at CIO 100, under the heading 2008 Winner Profile – IBM, where you will see the details being shared based on that selection criteria and what the award was granted for, based on that piece of IBM Research, which is where the original piece comes from.
    I have dug out our internal resources and I have also tracked down the original resource, which comes to explain the numbers with this criteria:

    Cost savings: Using an average hourly rate per employee of US$100/hr, ETS saves IBM US$95,528 each week. Assuming 40 hours per week and 48 weeks per year, ETS brings IBM US$4.6 million in potential productivity gain alone”

    And there are some additional costs savings that were calculated as well, if you would want to find out more on them.
    Now, I am not sure whether the results have been bloated or not, as you mentioned, and I do seriously hope they haven’t, but if those results were part of the selection criteria for that CIO 100 award I doubt they are “fake”, as they didn’t get questioned for their validity over 2 years ago when they first came out. But either way, very good exercise to challenge them, so I would get a chance to dig further on the actual research and survey results shared and, hopefully, with them we have been able to add some more clarity to the data mentioned on my blog post.
    Again, thanks for the great write-up and look forward to further interactions! :)

  3. James says:

    Luis,

    Thanks so much for the fast reply! I’ll go through the links you provided and take a look at where the numbers are coming from. I don’t know that I was clear enough in my post that I’m not doubting the value of the project or the intent of the authors. (In fact, I’m a huge proponent exactly what was done.) I also hope it doesn’t seem that I’m implying they’re faked in any way. My biggest concern, and the reason I wrote about it today, is that I saw a dozen tweets linking to your post all coming from people who are usually regarded as industry leaders in social media. Unfortunately, those same folks have been vocal in the past about the unimportance of ROI measurements in social media. To see them all tweeting about a post talking about such a big number made me suspect that they saw the headline and didn’t bother to understand the numbers.

    Again, thank you so much for responding to my post. I hope you won’t mind if I ask more questions as I go through the other articles.

    James

Leave a Reply