Web Social Architecture

Site Optimization: Lipstick on a Pig?, Or, Bacon and the Theory of Local Maximum

Jason Carmel is a colleague of mine I learn a ton from. His expertise is in web site optimization--running experiments where he tests versions of web pages against each other to see which performs best. (Not to be confused with search engine optimization, improving a site's visibillity in search engines.)

Jason is a fairly unflappable guy. Nonetheless I recently started making an effort to get his goat. He gives me just enough encouragement that I keep going. The gist of my teasing is that optimization is nothing but a mechanical exercise to determine whether a red button works better than a blue button. "Glad to hear that red button worked out better by 2.84 percent Jason. The sum of your creative energy has produced yet another quarter million in revenue. You must really love your life, man. Hey, have you thought of trying one of those animated GIFs instead of a regular button?"

Fortunately, Jason is twice as nice as me, as well as twice as smart. He takes my ribbing well--and responds thoughtfully to the serious question underlying my teasing: We know optimization can move big numbers in terms of revenue, but can it do more than simply tweak pages to bump up conversion? Can it vet creative concepts? Can it maximize the creation of mutual value between businesses and customers? Can it help create more engaging experiences?

(image credit, found via Lee)

 

The short answer here, according to Jason, is that it depends, partly on what you're trying to achieve. If all you're focused on is moving business value measures, you're probably putting lipstick on a pig. But testing against value creation has the potential to uncover game-changing opportunities.

Here's an email exchange between Jason and me, in which he explains in a little more detail:

 

RYAN:

That whole web site optimization thing—isn’t it really just putting lipstick on a pig?

 

JASON:

I think “I hate you so much” might be a succinct way of responding, but I'll provide a little more detail:

Web Site Optimization is exactly like putting lipstick on a pig, but only if you start out with a pig. And if you are starting out with a pig, your opportunities for improving things are limited, and you’d be using the wrong tool to fix the problem. We are talking here about the concept of a “local maximum” which is a fancy, math term applied to mean “the best something can be within a limited dynamic.” Consider the aforementioned pig’s ability to fly, which, metaphorically speaking, is not particularly developed. We could take a pig and genetically modify it to be more aerodynamic. We could investigate building pig hang-gliders and attempt to train the smartest pigs to use them. But even in the best case, with the most aerodynamic pig, benefiting from the best training, and using the best pig flying technology, it will never fly as well as a bird. The best case flying scenario for a pig (the pig’s local maximum as far as flight is concerned) is nowhere near as effective as a bird’s. In that scenario, you’d be better off exchanging the pig for a bird at the start, rather than waste any time or effort teaching a pig to fly better.

Applied to the web: if a site sucks so much- if the goals and purpose are unclear, if the information architecture looks like my desk (at the moment), if the navigation is counterintuitive and the messaging has absolutely no intersect with the audience, then no amount of optimization in the world will make it right. The local maximum of that crappy site is too low for any optimization to matter. Or (even worse) you’d need the infinite number of monkeys to stop typing Shakespeare and to start applying experiments to your site to get the right combination where testing would make a real difference. Neither is very efficient. If your site is the pixilated equivalent of a pig, you need much more elemental help from a user experience expert first (know any?). Until you fix the fatal flaw(s) in a site, anything else you do will be throwing good money after bad.

Site Side Optimization works well in circumstances where the local maximum is high, but for some reason, the site is not achieving it. This can be due to single points of failure on the site, like a specific conversion path or page underperforming, or because the audience needs to be targeted more specifically, or because the existing content is stale/irrelevant. In each of these cases, experimental testing can make a huge difference. Optimization also works exceptionally well (and this is far more interesting to me) when applied as a method of trying out a new (and potentially risky) idea that could radically change and significantly improve an experience.  In both of these examples, the basic site is healthy, and the optimization program serves as a tool to reach its fullest potential.

 

RYAN:

But what I keep looking for is the way to test birds against pigs, not in the sense of which flies better, because as a user experience expert I do have the capability to predict the winner of that contest—but when I don’t have a clear sense of the best conceptual solution. For example, maybe I just can’t decide between eggs and bacon. Can optimization help design a better breakfast, or only decide between pulp and no-pulp in the OJ?

 

JASON:

Optimization can test more conceptual ideas, but it will be really hard to unpack the WHY after we determine which one wins. Most sites aren’t deciding between bacon and eggs, but rather between the bacon, eggs and hashbrowns with coffee or the granola, fruit and yogurt with yerba matte. If the former wins in a test, I don’t know whether it’s because of the bacon (which will usually win over everything) or the coffee, or because the person deciding had granola for the past three days, and would have taken ANYTHING other than more granola.

The other trick about testing high concepts in a website format is that you would have to build each solution to test them, which is usually more expensive than testing out wireframes or front-end prototypes in front of a more controlled audience.

 

RYAN:

First, you seem to be suggesting that a test win for the bacon breakfast might not imply extensibility for bacon breakfasts in general: That because, lacking control, the results might be idiosyncratic, they might not therefore apply broadly. Next week you might get a different result. My question is, why does that matter? And why does “knowing why” the bacon breakfast worked matter, as long as you know it worked.

 

JASON:

It is definitely a question I get from clients a lot. Why do I care about the individual elements of a variant- if the variant as a whole makes us more money, let’s just launch it and move on. I can’t fault the sentiment, but knowing why the bacon worked could lead to better tests, more focused messaging and (even more) cash money. I want to know that it’s the bacon by itself that is the motivating factor outside of all the other influences. Let’s say that we ran a test breakfast against a bowl of Total cereal and we tested bacon with powdered eggs as the experimental variant. Now let’s assume that bacon and powdered eggs lost to the control by 1%. You could take the position that we would do better to serve Total because we want to avoid losing that 1%, and you would be right. But what if you knew that the bacon by itself actually improved the breakfast by 15%, but the powdered eggs were so crappy that they hurt the breakfast by 16%, so you netted the 1% loss? If I controlled for all the variables in my breakfast, that knowledge would a) help me make a better breakfast overall (just serve bacon), and b) will also prevent me from throwing away a positive variable simply because it was paired with a really negative one.

I imagine you run into that a lot with prototyping and wonder how you deal with it in the UX world.  If a subject totally fails at a task, are you ever afraid of overcorrecting a prototype to account for it? Do you ever throw the baby out with the bathwater? How do you control for that?

 

RYAN:

A great question. One of the answers is that in usability testing, you're looking for usability problems. So as long as your test participants are representative of your users as a whole, major failures are, practically speaking, never anomalous. If your user population is one million, and one of the eight people participating in your test has trouble understanding some aspect of the interface, what are the odds they're the only person who's going to have that problem?

The other thing I wonder about is what happens when what you’re trying to accomplish is harder to measure than conversion (e.g. brand lift) or if you want to measure it over time (e.g. engagement). Especially in social media, it’s quality that matters, not quantity. You want to know how valuable your user-generated videos are more than you want to know how many of them you have. Can web site optimization help you get to answers?

 

JASON:

Ah, you and your social media. When are you going to come to terms with the fact that this whole thing is a fad? The future is in email, Ryan, and lots of ‘em. Mark my words.

Absent a more qualitative tie-in with optimization (surveys, satisfaction scores, etc.) you will be hard-pressed to get good data about branding or the impact of social media. But I’m not saying you shouldn’t optimize for branding or social media. I’m saying you need to get that qualitative kicker. I’ve done a few branding tests, and I think they provide some interesting feedback. But I’ve never optimized where a KPI has to be judged on quality (e.g., good comments vs. troll comments) or off the site entirely (e.g., buzz in the blogosphere). Sounds fun.

Mining the Business Value of the Social Web: Behavioral Metadata

As an information architect, I work with metadata a lot. I help define interfaces based on information about the content. For example, an object on a home page might be the "newest" object in a system, or it might be a rotating series of "newest" objects by each "author" with "home page" authority. See my other posts on quality and self-organization in the presentation layer if you want more on that.

I work at a company with a strong focus on performance-driven design, and so I work constantly with web analysts, who specialize in understanding behavior on web sites and using that behavioral data to drive business decisions, and I work constantly with designers, researchers, and information architects whose focus is creating innovative, valuable stuff people want to use.

I'm working where I work partly because I think there's tremendous potential in the intersection of information architecture and web analytics, in particular on the social web.

While some companies have been implementing sophisticated web analytics programs for years, many are just starting to measure behavior on their sites. Very frequently, there's lots of data collected but no real plan for taking action based on it.

At the same time, many companies are just beginning to experiment with social media marketing. They're getting out there and participating, and their customers are creating content, but that content is hard to present meaningfully, and partly because of that most social media marketing efforts fail to realize their full promise.

And naturally, the new focus on measurement and data raises the question of how the success of those social media marketing efforts should be measured. I've posted in the past about measuring communities, and there are a lot of people out there contributing to that discussion. What I think is frequently overlooked, though, is behavioral data helping to create value in social systems. Not data about value, but data creating value.

I made this handy chart to help explain what I mean. Click the image for a more-legible view:

 

When users interact with web sites, information about their behavior accumulates--information attached both to the user as object and content as object. That information about people and the things they do--what's called behavioral or tacit metadata--tells a lot about people and content. And in complex, emergent and self-organizing systems, behavioral metadata is especially powerful, because in those kinds of systems it's a challenge on the one hand to present content in meaningful ways and on the other hand to generate business intelligence to drive smart decisions.

As content and people increasingly cross freely between domains in the emerging standards-based social universe, I think we'll see a shift in business models from today's focus on owning content to a new focus on owning the metadata about the content and the people interacting with the content. I'll risk a prediction: In a short time, behavioral metadata, the information about what people do with content, will be more valuable than the content itself.

What do you think?

Web Professionals: Social Media is about to Change Your Job. Are You Ready?

Let me set the stage for what I mean to discuss here by sharing Ryan's Two-Minute History of Web Marketing:

The web has been defined by three revolutions over the past 20 years.

Prior to the involvement of business, the web was a connected ring of online communities. People interacted with each other and shared information. But the threshold of entry was relatively high, and there was no business model in place to drive innovation. The web kind of stayed the same for a long time.

In the early 90's, businesses started to get involved with the web, and driven by advertising dollars, they applied the same approach to the web as to broadcast media: This was the Broadcast Web. Web as TV commercial, billboard, brochure.

In the late 90's, e-commerce emerged. Amazon started kicking peoples' butts. The web was now a storefront (minus most of the customer support, but that's another story), complete with data-driven pages and integration with business and financial channels. Give me your credit card number, and I ship you a book: This was the Transactive Web. Remember? The one with the bubble.

We're now at the flashpoint of another revolution, driven by the convergence of three forces: Widespread availability of broadband, the lowering of thresholds of entry to publishing (especially for photos and video), and popular awareness of social media (especially MySpace and YouTube). We're only beginning to see the impacts of this change. We're at the bottom of the curve of the Participatory Web. In some ways we've come full circle, returning to a web of connected people sharing information. But in other ways, the emerging web is a new beast entirely: Bigger, faster, with exponentially more people and institutions involved, and with more at stake.

My point with all that is to suggest that most of the way we do our work today in the web-site-making profession reflects the demands of the Broadcast and Transactive web ages. Our planning, processes, tools, and tasks are undergoing a shift—and we’ll be well-served to get out in front of the demands of the Participatory Web.

So what's changing? Most obviously, a greater proportion of the total work of the web site happens after launch. And in a world of limited budgets, budgets built on the idea that the site is mostly done at launch, we're challenged to be faster before launch and smarter after.

Here are some ideas about the disciplines of the people I work closely with day-to-day at ZAAZ. I'm not an expert in all of this, but I sent an early version of this post internally as a memo and got some great feedback. And I'd really love to hear what you think. What have I got wrong, or missed entirely?

Client Services will shift toward helping clients plan for ongoing engagement with customers through the web channel. We’ll be supporting policy development, leadership, and moderation, and we’ll be helping clients transition over set timeframes to self-sustaining community management. We’ll be scoping and planning iterative work responsive to user communities.

User Research professionals will shift toward working to understand issues like value, trust, identity, reputation, motivation, and social ecosystems--online and off. 

Information Architects will shift toward organizing user-generated content; participatory interaction; and emergent and self-organizing content.

Designers will shift toward faster prototyping, with an emphasis on developing brand-aligned affordances for social and participatory spaces.

Developers will shift toward increased reuse and customization of third-party tools, vendor assessment, portability standards, and integration with third-party sites. We’ll be supporting prototyping and iterating live sites. Standards will become more important for client work.

Search professionals will be supporting on-site metadata definition for user-generated content and driving awareness beyond the site. Findability is a key differentiator in the crowded social media marketplace.

Web Site Analytics will shift toward measuring engagement, affinity, and cross-domain activity. Analytics reports will become thermometers for measuring the health of communities and guiding site iterations. The convergence of ad network power and data portability will have huge impacts. It's not about what happens on only your own site anymore.

Web Site Optimization folks will work in closer partnership with analytics, design, and UX to improve engagement, raise trust, and build motivation to participate--conversions that address customer lifetime value. It’s no longer going to be all about pushing more purchasers through the funnel.

Every web discipline is affected by the revolution at hand. To be successful, we need to define how to talk about our capabilities in that light, to frame up those capabilities as offerings, and integrate them into an overall story about our ability to execute social media--because clients are asking, and very soon they're going to want proof.

 

I've got plenty of ideas about this stuff, but I'd really love to hear from you and write a follow-up post summarizing your perspectives. Do share your thoughts about your discipline and how you see it changing in the next few years!

Separating the Wheat from the Chaff: Quality in Emergent Social Content

Users create tons of terrible content!

We must not say so. We'll say instead, users create content that ranges widely in terms of its degree of interest to other users.

I've been doing a ton of work on quality the past few weeks, in several different contexts. Consider the following challenges:

1. Our design for a client's corporate group blog includes a prominent list of "Most Popular" posts. In this context, where the blog is fully integrated with the main site via primary navigation, we expect many non-regular blog readers to wander over to the blog landing page, and we want to increase the likelihood that they'll see something both compelling and fresh.
2. Another client's technical support forums include thousands of posts, many of them addressing the same or similar issues, and only one of them providing the single best solution for a particular user's problem at a particular time. The most common handful of solutions address 90% of user's issues.
3. A client's social media site uses a set of algorithms based on user behavior to rank user-created content objects, and the best are exposed on the home page of the site. However, their exposure on the home page garners them more attention than average, strengthening their ranking and producing static "best" lists. The rich get richer.
4. A client's community site is wide-open, allowing posting on a range of subject areas. The home page of the site shows the most popular posts, but the popular posts have tended to be less than germane to the business rationale behind the site.

Let me take a step back, though, and talk a bit about the general approaches to organizing emergent user-generated content:

 

Emergent Content

Emergent content is content that shows up automatically, based on a set of rules. No single person decides what content is visible and what content isn't. Instead, content objects appear in various contexts based on attributes encoded in their metadata. Sometimes this means the good stuff shows up, sometimes it means stuff shows up based on similarity to other stuff, sometimes it means complentariness, and so on.

In the business, we call it "The Magic of the Web."

 

Algorithmic Approaches

In many contexts, especially on web sites with huge volumes of user-created content, quality is derived algorithmically, based on user behavior. And user behavior can be thought of as including explicit behavior like voting and rating and implicit behavior as well. For example, the number of people viewing a video object can reflect its quality--and beyond that, quality measures can include views of the entire video, comments on the video, time on page for the video, qualified page views, and more. And other measures can reflect negatively on the video's quality--incomplete views, time on page below a qualifying threshold, bounce rate, and more. 

Bits and pieces of behavioral metadata stick to content objects--less like fingerprints than like the folded corners, weakened binding, marginalia, and coffee stains on an often-loaned book--and provide a basis for us to construct the rules that govern how the objects behave.

Algorithmic approaches to measuring quality take into account a combination of these measures in ways appropriate to the type of object and culture of the site. Which measures are more important (and how much more important) depend on the context of user needs and business requirements--every site is unique. "Quality" is quantifiable only with the right formula.

The most bad-ass example of algorithmically-derived quality is, no doubt, the Explore category on Flickr. Based on its concept of "Interestingness," Flickr finds photos that are interesting and shows them in views filterable by date, camera, location, tag, and tag cluster.

What's so amazing about the Flickr Explore feature is that it's constantly refreshed and the photos are always, indeed, interesting. Flickr has an advantage: Millions of content objects. Naturally, the best hundred are pretty darn good.

 

Editorial Approaches

As appealing as is the idea the bottom-up, self-organizing content architecture, editorial participation has a key role to play in many branded online communities. First and foremost, an editorial presence can provide a human face for the company, a point of social contact between the community and the brand. An editorial presence can also model appropriate behavior, highlight the kind of contributions that best serve the needs of the community, and speak for the brand without being perceived as a constraining or intervening voice. It just has to be done with a delicate touch.

A while back I wrote about a quality problem on the social media site Treemo. Since I wrote that post, Treemo has rearchitected its home page, which used to highlight popular content, to feature editorially-selected content. The result is a much stronger affordance on the home page for content that aligns with the site's mission. Over time, that affordance will help improve the overall quality of content on the site--though Treemo's too-simplistic separate display of most viewed, favorited, and commented still render horrors upon the unsuspecting eye.

 

Hybrid Approaches

Combination approaches can offer the benefits of both algorithmic and editorial quality measures. Typically, an initial level of quality valuation emerges from user behavior, qualifying a top tier of content for editorial review.

A great example of the hybrid approach is JPGmag.com. JPG is a social media site, which announces a theme for a period of time and invites users to submit photos related to the theme. Users view photos one at a time and vote whether each photos is "good for (theme)." The experience is compelling, partly because the emergence of photos is affected by the votes of other users, so the photos that aren't voted up tend to disappear quickly. On the surface, the interaction is extremely simple. But there's some good, solid magic going on behind the scenes.

Photos that rise to the top based on user voting are qualified for review by the editorial panel, which selects photos for inclusion in the theme-based print edition of the magazine.

 

 

Beyond the Top-10 List

Ranking content objects based on quality makes possible some powerful uses of quality content--including but greatly improving on the typical idea of the "top 10" list. Quality can be superimposed on other metadata to produce content presentation that are both highly faceted and of high quality. For example, you can search Flickr for photos tagged "gorilla" within a geographic range limited to Rwanda, then sort the results by interestingness, and even filter by the type of camera you own. You can see the best photos of Rwanda's gorillas taken with your particular camera model, and see how your own adventure travel pictures measure up.

Amazon's customer reviews page is another great example. Customer-written product reviews are ranked based on a simple "usefulness" vote by users. Amazon highlights the most helpful reviews above and below a rating threshold. The most helpful favorable and unfavorable reviews appear side by side, an instant conversation that socializes the shopping experience, adds credibility to the buying process, and creates a unique value for online shoppers.

 

 

Quality in Action

All of the challenges I described earlier require a solution with some kind of quality measure. Over the past three or four years, I've found that my information architecture work is less about labeling and organizing content and more about creating mechanisms to apply descriptors and rules to govern emergence. As user-created content becomes more important, the IA work happens at a remove. It also feels more critical to get it right and to design flexible systems that can adapt to unpredictable user behavior.

Likewise, new possibilities to create value have emerged--and quality is just the beginning. Let me know what you think, and stay tuned for more posts on this topic.

Measuring Community Success: Two Contrarian KPIs for Web 2.0

We've seen a shift recently in how some of the major players in Web measurement are evaluating success. Nielsen/Netratings switched from counting page views to counting time spent--to the derision of some in the analytics community.

Time spent has problems, it's true. But I haven't heard a lot of great suggestions about how you do measure success on participatory web sites, social media sites, or online communities. The truth is, the devil's in the details, there is no silver bullet, and most of all, it depends what you're trying to accomplish.

Yes, everything in web analysis should track back to your goals. Repeat that ten times.

Nonetheless, I will offer up a couple ideas about how measurement in online community is different than on brochure sites, and how standard measures, to the extent there are any, fall short in typical online communities.

My suggestion is basically that we need to think differently about success, and abandon some of the old-school, quantity-focused measures, like number of posts, number of comments, and number of registered users, for an approach that emphasizes quality. Here are two examples of what I mean:

 

Return Visitor Sign-In, Not New User Registrations

Sign-in is more important than return visit--it's even more important than registration. Participants in online communities typically register to access content they can't access otherwise, to take an action they can't take without registering, or to streamline a future process (e.g. saving credit card information).

So, participants typically register for the promise of a reward. Sign-in is different. When a registered user returns to the site and authenticates, you are a little closer to knowing you gave that person something valuable. In other words, when the cousins get up for more, you can be pretty sure they like your mashed potatoes. If you want to gauge the success of your meal, measure the number of second helpings, not the number of people at the table.

 

Qualified Content Consumption, Not Number of Posts

I've written in some detail about the value of so-called "passive" participation (I personally wouldn't call information-seeking online a "passive" activity). In short, content consumption tends to be greatly undervalued relative to content creation. While content creation introduces the potential for value, content consumption represents the realization of that value. Tracking content consumption, then, helps you learn not just how much stuff you have, but, more importantly, how good is your stuff.

Part of the problem with quantity measures in social media is that all content is not created equal. The best content, while it's created only once, creates value many times over, while the worst content can actually be detrimental to the participant experience. So the number of content objects on your site tells you much less than does the number of times those objects are consumed--in total, on average, and individually.

But how do you know content consumption when you see it? Content detail page views can be misleading, but I differ from those who'll tell you it's all about time on site in Web 2.0. Instead, count page views qualified by either a time-based or action-based threshold. 15 seconds on a detail page is a decent place to start, but it doesn't account for the effect of tabbed browsing, which can skew time-based measures upward.

Here are some examples of action-based measures for qualified content consumption:

• file downloads
• play embedded media
• copy embed code
• forward via email
• scroll to bottom of page
• print
• favorite

There are many more, and there's a lot more to say on this topic. Drop me a note in the comments with your thoughts, ideas, and questions.

Social Network Marketing: Think Differently about "Placement"

"Social networking" is a term I hear used interchangeably with "online community," "Web 2.0," and, most discouragingly, "viral marketing." In the latest intallment of The Commoncraft Show, Lee has done a nice job clarifying the term for those unfamiliar with it: 

What I like here is that Lee describes social networking in a platform-independent way. Obviously, there are lots of different kinds of web site that could serve as a social network, including blogs, which aren't typically considered social networks. 

Online social networking is a human activity, not a technology type. While there are some very familiar sites like MySpace, LinkedIn, and Match.com that are typically lumped into a category as "social networking sites," I'm much more inclined to think about social networks in architectural terms. 

From a social architecture perspective, online social networking sites are clusters of linked identity-centric web spaces. While on the one hand, there's no real hard-and-fast line between "social networking" sites like MySpace, which are structured around the individual identities of users, and "social media" sites like YouTube, which are structured primarily around individual pieces of content and secondarily around "channels" (groups of content objects with a common author), on the other hand this difference of emphasis has a huge influence on how these sites are used. 

Most significantly, social networks increasingly cross boundaries between web sites. Behind the power of web services, APIs, widgets, and portable content (like Lee's video above), relationships and content move effortlessly across domains. 

Even more interestingly, social networks are beginning to cross boundaries between our physical and digital lives. Add-on services like Moo's 3rd-party-based business cards herald a new generation of real-world artifacts of our online experience. 

Marketers wanting to tap into "the all-powerful magic of" online social networks need to shift from thinking about (and budgeting for) site based-initiatives and campaigns ("Let's do something on MySpace this fiscal year") and begin viewing social networks as a relational architecture, independent from and superimposed against what I'd call the superficial architecture of mere web sites.

The Lurker Myth: Measuring the Value of Passive Participation in Community

In conversations about measurement of community sites, I frequently find myself championing the value of passive participation. Intuitively, it's easy to feel like "lurkers" are somehow taking advantage of a community without doing their fair share. In fact, not only do I believe so-called passive participation is the very lifeblood of community, but that there are better ways to think about whom your users are.

My basic philosophy on this point boils down to this axiom:

Active participation creates potential value, and passive participation realizes that value. Most people do both.

In other words, active and passive participation in and of themselves are equally valuable. While it's tempting to "reward" content contributions, which often reflect a higher degree of engagement, with a higher valuation, active and passive participation actually have a symbiotic relationship. In some situations, such as on a technical support forum, "passive" participation by people who need help accessing helpful content is the whole point.

 

 

It's a nice way of making the point that you need to measure, value, and optimize for passive participation. But of course it's not quite so simple. A single piece of content can be accessed many times, returning value over and over. In terms of individual actions, adding content is more important than accessing it. And some contributions create more potential value than others.

 

 

In this view, you can rank actions like posting a video highest, commenting on the video next highest, rating a video one notch lower, and viewing a video one notch lower still. That kind of engagement-based ranking can help you compile an overall index of community engagement, as I discussed in an earlier post.

This kind of relative view of the value of actions also provides a different lens to look at your users. While it's tempting to think of users as either content contributors or content consumers, participants or lurkers, the truth is nowhere near that tidy. 

This is the old-school way of looking at user segments in online community:

 

 

The basic idea here is that a core group of highly engaged users adds the content that fuels the community. The problem is that this model isn't accurate.

In fact, the majority of your online community users probably both produce and consume content:

 

 

So it's not necessarily useful to think about your participants as either "contributors" or "lurkers." It makes much more sense to think of them in terms of their degree of engagement. Using the community engagement index, you can identify and reward highly-engaged participants. You can reach out to less engaged participants with invitations to get more involved. Using behavioral targeting, you can even serve up content reflecting users' degrees of engagement to orient new participants, deepen engagement, and encourage exemplary behavior.

The Basics of Online Community Measurement and Analysis

There are a number of ways to think about measuring the value of your online community, and ROI is only one of them. ROI is, frankly, a head-scratcher, and most people in the online community world throw up their hands when it comes to thinking about how to dollarize their community initiatives. The reasoning goes: "We can't tie community participation to purchases, so instead we'll talk about community participation in terms of branding." I will, however, make a suggestion about how to show ROI that will work for some people in some contexts. 

Online Community Brand Lift Survey

The idea here is to get a sense of how exposure to a branded community experience affects brand perception. There are few on-site behaviors that give reliable qualitative measurements of brand perception, so analyzing visitor behavior isn't the best answer. In the usability lab, the tendency of participants to seek to please the test facilitator (the Hawthorne Effect) casts doubts on measurements of brand perception. 

A series of surveys are the answer. Start with an entry survey that gives you a measurement of brand perception prior to exposure to the community site, and follow up with an exit survey measuring brand perception after exposure to the community site. 

You have to do this right for it to work. Find a qualified researcher in your organization, hire a partner, or get yourself an intern graduate student with a background that includes research using surveys. You want to present the entry and exit surveys to different groups of people, for example, and make sure the survey is written in a way that gathers exactly the information you want to have. It's not rocket science, but it's a little harder than, say, Algebra. 

Another critical consideration for your exit survey is to think about what site behaviors qualify users to participate. In other words, How engaged with the community site, at a minimum, do users need to be for you to expect the experience to influence their perception of your brand? Obviously, folks who land on your home page by accident and immediately leave shouldn't be seeing your exit survey--their perception of your brand hasn't likely been influenced. But how much engagement is reasonable to expect? Over 20 seconds on the home page? Fully completed profile, 5 return visits, posted original content? 

What is engagement, anyway, and how do you know it when you see it? 

Online Community Engagement Index

Here's an approach to measuring participants' degree of engagement with your site. Start by making a list of all the important trackable actions on your site. Here are some examples: 

• Home page view 
• Detail page view 
• Profile view 
• Download file 
• View video 
• Register 
• Add comment

You can also include measures like time on site, number of page views, and more. And you should include important combinations of actions, like registration followed by return visit.

Next, rank the actions in order of degree of engagement. For example, registration usually reflects less engagement than completing a profile, posting video content reflects more engagement than commenting on a video, and so on. 

If you want to get fancy about it, you can do a weighted ranking, where for example uploading a photo for your profile counts 5 times more than filling in a where-do-you-live field. 

For a simpler approach, you can group actions into 3-5 categories reflecting different degrees of engagement. (This is a relative measure, so precise numbers about degree of engagement are less important than consistently measuring engagement over time and above all taking action to increase engagement.) 

Then, you can tally up your users' scores. There are lots of ways to slice up the index--median by month, week, day, quarter, year; median among content contributors, registered users, lurkers; proportion of users above and below certain thresholds. Community engagement is a rich measure that can teach you a lot about your site. 

And where the learning really gets significant is as you look at the changes over time and across iterations of the site. You can get powerful feedback about how your site management affects the experience of your community. 

One could say engagement is a good thing in and of itself, and I think that's generally true. But when it comes time to pitch for budget dollars, "engagement" doesn't necessarily get you past the skeptical VP. 

The Holy Grail: Return on Community Investment

Here's a way to think about ROI in terms of the relationship between engagement and dollarized conversion. The logic is to show conversion relative to engagement, that is, the degree to which people are more valuable to your business the more engaged they are with your community. 

If your community site has an e-commerce component, or your community is woven throughout your larger site experience, it's easy to look at the relationships between engagement and conversion rate, average order value, and so on. If not, the story gets a little more complicated. 

In the consulting business, we're increasingly working with clients to coordinate constellations of web sites, and tracking behavior across those constellations is part of the leading edge in web analysis. Here's a conceptual picture of how some of this works: 

   

Understanding the relationship between community engagement and dollarized conversion enables you to calculate the value of a customer who's engaged with the community compared to the value of a customer who isn't. From there you can look at the total number of community participants, total up their relative value, and subtract the cost of the community. Poof! ROI. 

Some words of caution. This approach doesn't tell you: 

• The long-term value of engagement. 
• The value of brand lift produced by community. 
• The value of community participants' influence on non-participants.

These are important caveats. It's critical to remember that for community, ROI isn't the whole picture.

And, this approach isn't trivial to implement. You'll need solid in-house expertise or a qualified partner.

Nonetheless, contrary to what some might tell you, and despite the complexity, it IS possible to calculate ROI for some community web sites when you need to show the VP that dollar-sign bottom line. It's a powerful basis for goal setting--and for holding yourself accountable.

For additional reading about site analysis and measurement, I highly recommend the blog of my respected colleague, Anil Batra, along with that of analytics luminary Avinash Kaushik. For in-depth reading, check out Actionable Web Analytics, a new book from (full disclosure) colleagues of mine at ZAAZ.

A Sea of Flowers: Feevy's Map of the Blogosphere Challenges the Long Tail

In a previous post about Feevy, I wondered aloud what they'd do with the metadata accumulated by their product. David de Ugarte pinged me today with the first hint.

Cool! I have to say, I find this view of the blogosphere much more interesting than the ones provided by Twingly, which shows what's now, and Technorati, which shows what's big.

There is a distinctive quality of this map that strongly interests me:

Within the central "nucleus" of connections, there tends to be a relatively even distribution of connections. That is, there is no small handful of blogs that are substantially more linked-to than others. This is a big surprise, because conventional wisdom says links should create a power curve of influence with a long tail of isolated blogs.

But something different seems to be happening here, and I'll float two theories about it:

• Blogs seem to link to other blogs of equivalent connectedness

because:

• Networks of linkages form loosely around discrete and related topics.

This observation suggests a change of emphasis for businesses wanting to engage the blogosphere from seeking exposure on the top blogs to seeking engagement with the most relevant blogs. Whereas the traditional PR model might seek out the biggest, most widely read blogs, and a newer, targeting-based PR approach might look to respond to specific mentions of a brand or exec, there could well be a Middle Way:

Find the people who care the most about your stuff, engage with them, and stay engaged over time. In other words, focus resources on targeted, relatively narrow sections of the blogosphere, rather than trying to curry favor with the so-called A-List or scattering ephemeral attention across the whole Internet. You might have heard of this. It used to be called relationship marketing.