The Basics of Online Community Measurement and Analysis

There are a number of ways to think about measuring the value of your online community, and ROI is only one of them. ROI is, frankly, a head-scratcher, and most people in the online community world throw up their hands when it comes to thinking about how to dollarize their community initiatives. The reasoning goes: "We can't tie community participation to purchases, so instead we'll talk about community participation in terms of branding." I will, however, make a suggestion about how to show ROI that will work for some people in some contexts. 

Online Community Brand Lift Survey

The idea here is to get a sense of how exposure to a branded community experience affects brand perception. There are few on-site behaviors that give reliable qualitative measurements of brand perception, so analyzing visitor behavior isn't the best answer. In the usability lab, the tendency of participants to seek to please the test facilitator (the Hawthorne Effect) casts doubts on measurements of brand perception. 

A series of surveys are the answer. Start with an entry survey that gives you a measurement of brand perception prior to exposure to the community site, and follow up with an exit survey measuring brand perception after exposure to the community site. 

You have to do this right for it to work. Find a qualified researcher in your organization, hire a partner, or get yourself an intern graduate student with a background that includes research using surveys. You want to present the entry and exit surveys to different groups of people, for example, and make sure the survey is written in a way that gathers exactly the information you want to have. It's not rocket science, but it's a little harder than, say, Algebra. 

Another critical consideration for your exit survey is to think about what site behaviors qualify users to participate. In other words, How engaged with the community site, at a minimum, do users need to be for you to expect the experience to influence their perception of your brand? Obviously, folks who land on your home page by accident and immediately leave shouldn't be seeing your exit survey--their perception of your brand hasn't likely been influenced. But how much engagement is reasonable to expect? Over 20 seconds on the home page? Fully completed profile, 5 return visits, posted original content? 

What is engagement, anyway, and how do you know it when you see it? 

Online Community Engagement Index

Here's an approach to measuring participants' degree of engagement with your site. Start by making a list of all the important trackable actions on your site. Here are some examples: 

• Home page view 
• Detail page view 
• Profile view 
• Download file 
• View video 
• Register 
• Add comment

You can also include measures like time on site, number of page views, and more. And you should include important combinations of actions, like registration followed by return visit.

Next, rank the actions in order of degree of engagement. For example, registration usually reflects less engagement than completing a profile, posting video content reflects more engagement than commenting on a video, and so on. 

If you want to get fancy about it, you can do a weighted ranking, where for example uploading a photo for your profile counts 5 times more than filling in a where-do-you-live field. 

For a simpler approach, you can group actions into 3-5 categories reflecting different degrees of engagement. (This is a relative measure, so precise numbers about degree of engagement are less important than consistently measuring engagement over time and above all taking action to increase engagement.) 

Then, you can tally up your users' scores. There are lots of ways to slice up the index--median by month, week, day, quarter, year; median among content contributors, registered users, lurkers; proportion of users above and below certain thresholds. Community engagement is a rich measure that can teach you a lot about your site. 

And where the learning really gets significant is as you look at the changes over time and across iterations of the site. You can get powerful feedback about how your site management affects the experience of your community. 

One could say engagement is a good thing in and of itself, and I think that's generally true. But when it comes time to pitch for budget dollars, "engagement" doesn't necessarily get you past the skeptical VP. 

The Holy Grail: Return on Community Investment

Here's a way to think about ROI in terms of the relationship between engagement and dollarized conversion. The logic is to show conversion relative to engagement, that is, the degree to which people are more valuable to your business the more engaged they are with your community. 

If your community site has an e-commerce component, or your community is woven throughout your larger site experience, it's easy to look at the relationships between engagement and conversion rate, average order value, and so on. If not, the story gets a little more complicated. 

In the consulting business, we're increasingly working with clients to coordinate constellations of web sites, and tracking behavior across those constellations is part of the leading edge in web analysis. Here's a conceptual picture of how some of this works: 

   

Understanding the relationship between community engagement and dollarized conversion enables you to calculate the value of a customer who's engaged with the community compared to the value of a customer who isn't. From there you can look at the total number of community participants, total up their relative value, and subtract the cost of the community. Poof! ROI. 

Some words of caution. This approach doesn't tell you: 

• The long-term value of engagement. 
• The value of brand lift produced by community. 
• The value of community participants' influence on non-participants.

These are important caveats. It's critical to remember that for community, ROI isn't the whole picture.

And, this approach isn't trivial to implement. You'll need solid in-house expertise or a qualified partner.

Nonetheless, contrary to what some might tell you, and despite the complexity, it IS possible to calculate ROI for some community web sites when you need to show the VP that dollar-sign bottom line. It's a powerful basis for goal setting--and for holding yourself accountable.

For additional reading about site analysis and measurement, I highly recommend the blog of my respected colleague, Anil Batra, along with that of analytics luminary Avinash Kaushik. For in-depth reading, check out Actionable Web Analytics, a new book from (full disclosure) colleagues of mine at ZAAZ.

Comments

[Justin]

This is a post for the ages, Ryan.

Defining brand engagement in dollars is always a difficult task. I like the way you are attempting to tie conversion rates with community participation.

One thing that would be interesting to see is a matrix built to define the opposite of what Avinash calls the "most likely to recommend" factor. As we know, using surveys to determine the MLTR factor works, but since we know there are always "influencers"--or highly engaged customers--on community sites, it'd be interesting to see how their influence affects the least engaged to purchase.

I assume the key would be to measure those least engaged customers, determine what user-gen content they visited, and follow-up with a survey to ask what information was helpful.

Maybe there's a better way to do this, but I guess I'm trying to express where my key interest lies, which is the least engaged purchaser that was influenced by the "influencer"--or most engaged community participant.