Mr Koster, I am wondering if you can help me with a bit of online Mythbusting? I have an interest in statistics with relation to gamers and beta testers. This quote: “Something like 90% of the people playing an MMO never post in the forums.” was recently made here and attributed to you and Rich Vogel here. (please read the thread to see why I am interested) So, I was wondering if you could confirm this? How did you collect this data? And if so, what other data can you share? Do you know of any other sources of this sort of data? And, yes. I would be very happy to see this email posted and commented on in your blog. Thank you for your time and effort.
Regards, Guy Russon
Well, as far as how that stat comes about (and it does vary game to game — don’t take 10% as gospel, becaus eyou are right it’s a “whisper stat” at this point), you simply measure your subscribers, measure your active forum posters, and derive a ratio. 🙂 In the case of forums where they require a game registration in order to register for the forum, this is pretty easy.
Like Joe in that thread, I can’t share specific stats for specific games, but I will mention that community managers who have experience across different sorts of virtual worlds (gamey worlds, worldy games, kids’ worlds, social worlds and so on) tell me that there are very different levels of engagement on forums. Worldy games apparently have a much higher forum activity level than gamey worlds, for example. I am told that kids’ worlds have a lot of challenging drama, from a community management perspective, and so on.
There are many stats that get measured around stuff like this — and to address some that come up in the thread you pointed to:
- Most users who quit do in fact disappear silently. The poster who bothers to say goodbye is far less common than the person who simply reduces playtime until they just don’t show up, and then lets their account lapse. The #1 predictor of a given user churning out is decreasing playtime.
- Closed beta churn is a huge issue, and these days the pool of possible beta testers is full of “looky-loos,” people who just want to sample a game for free early. Many people underestimate the bugginess and general unplayability and lack of usability of an early beta or an alpha, and just bounce off.
- Very few people ever submit bug reports, as a percentage of the testing pool. (And fewer yet of those are reproducible, comprehensive, and comprehensible). In general, I don’t think that we tend to make it easy enough, either.
You can dig through some of the presentations on this site — I may have posted some other stats before. And there’s certainly been many postmortems and conference talks about MMOs with random stats mentioned. But I think you may find it hard to come by hard stats for different games out there, because by and large a lot of this stuff isn’t shared publicly. Also, the game industry is not nearly as obsessive about metrics and analytics as the web industry is, so sometimes some of this stuff is not instrumented as well as it should be prior to the game’s testing period beginning.
It’s generally considered best practice to take forum feedback as only one channel of input in terms of community sentiment. Forum posters tend to be passionate, highly informed about the game and its details, connected in the community, and are often thought-leaders who have strong influence over the overall game community. But this can be a drawback as much as a benefit, and needs to be balanced with input from the “silent majority” that plays but is less plugged in, is less devoted and more casual, and who doesn’t even read the website.
Other channels include exit surveys, user polling (SOE has had great results with log-in polls), gameplay metrics data, and gathering info from targeted focus groups that are intentionally made more representative of the playerbase as a whole.
25 Responses to “(Tuesday) Mailbag: forum posters, churn, betas”
Sorry, the comment form is closed at this time.
It sounds like politics, to me. Ie, millions live in America, half of us vote, and of the ones who vote, maybe 10% actually do any research. Probably a lesson in there, somewhere…
Funny, you said pretty much what I did. Hopefully Guy isn’t too frustrated as a result. 🙂
Forum goers are the squeaky wheel. 😉
I was going to reply on the Pirates forum with my Station account, but someone apparently decided to bar nonsubscribers from participating in discussions.
I can think of several marketing reasons why that policy is stupid. I could probably even show how Flying Labs is losing sales due to just that requirement!
Joe Ludwig wrote:
Guy Russon wrote:
Any market data as related to specific products is strongly protected by individual firms as trade secrets. The rationale is inherently and mostly competitive; however, the rationale can also be related to stocks and trader confidence.
As a frequent beta-tester and rare bug-reporter, I want to say that a lot of games make it very difficult to report bugs. Many games I’ve beta tested require users to exit the game and write an e-mail describing the bug and steps to reproduce it. This is a lot of work for voulenteers. Of course, many people don’t go through the steps of repeating the bug themselves before they report it either. The other bug-report system I’ve seen is an in-game ‘ticket’ system where you fill in a text field and categorize the bug; but I have NEVER seen a text field large enough to accurately describe a bug in detail, let alone how to reproduce it.
This is common practice, and the reasons are basically related to cost and practicality. Having forums with no account requirement vastly increases the amount of moderation required, as spammers, ex-players with a grudge, inter-game fan wars, and so on pollute the forums.
In my experience, posters to game forums are disproportionately hardcore, and the atmosphere is often condescending (when not actively hostile) towards new players and self-identified casual gamers. The net effect is to discourage participation by anybody outside an insular core of posters. Relying on this cadre for primary feedback on development issues could potentially cripple your ability to recruit and retain new players, if the newbie/casual game experience is neglected in favor of developing for high-end raid groups and pvp.
I’ve been tracking this about once a year now for seven years on the MMO I produce (WWIIOL) and 10% is about the right number for posters with more than just a handful of posts. Those were generally support issues or one-offs that come from official discussion threads related to a production note or announcement.
10% is best applied to paying subscribers who post regularly. That comes across as the players who are actively engaging in the forum community outside the game. In our particular case some of these players are primarily tied to the forum community and in fact rarely play the game. They are particularly prevelant in the OT forums. I would think (any one have input) that anyone with a very long running community (15% of our subscribers are from year 1) would see something similar.
@morgan
We also bar non subscribers from posting on several of our forums (community discussions), restrict to read only on others (dev and announce) and give full access on a few (support). I don;t think this approach is unique to either subscription based or game related forums especially when those communities are hosted by the developer.
There are probably marketing reasons why this is “stupid” but there are several why it is important. Value add propositions come readily to mind and in particular for us the number of customers to our small company who pay to chat in OT. Many of these players are the same long term friends of the company who have been here for years and support us primarily because they like us and like what we do and how we do it. Hundreds of these players have joined our builder program by giving us up to a thousand dollars for a lifetime account. I believe we were the first to do that sort of program. It was a natural given our devoted following.
It is also a good idea to restrict passersby. Meaning that if I ban you from the forums and you can just continue to create accounts I have to increase my moderating budget. I am also then exposing potential customers to the rantings of people who have no interest in paying for my service. If I restrict where new accounts have access I can concentrate my message and moderation resources more on those places. Flying Labs, being free to play may alter these dynamics, but I doubt it.
Sorry for the hijack Raph. If Guy thinks that our data is useful I’lll be happy to discuss it with the team and see if we can easily put some numbers together for him.
Raph wrote:
I understand the rationale. As a marketer though, I read that as an excuse from community relations, “We’re not willing to invest in generating sales. We think marketing is solely responsible for acquiring customers. We’re just here to retain them.” Ironically, that’s not what most community professionals think.
It’s been my experience, too, that many users who DO say goodbye usually say hello again within 7 days because they’re just having a short-term spurt of anger.
Morgan, perhaps… but do remember that the increase in moderation difficulties is not linear with population increases, it’s closer to exponential. At a point it becomes a cost/benefit study, and the costs balloon very quickly due to the difficulty in managing increasingly large populations. The benefits are not so tangible.
Tying forum access to gameplay accounts have other benefits too: if your forum account is tied to your game account there’s an inherent decrease in moderation issues simply because people don’t want to be banned from the game; there’s an actual consequence to their actions other than losing the 10 minutes it takes to roll up a new forum account. Also, considering how horribly stressful the general CSR’s job is, I’m not sure I’d want to go trying to make it that much harder for them to begin with. You’d result in a lot less direct communication with the dev team as the CSRs would increasingly have to fall back to pure moderation instead of answering questions, and in an increasingly hostile environment they’d be vastly less able to maintain a healthy relationship with the playerbase. Noise increases too, making it that much harder to for the devs to get accurate feedback. Forums are there as a method of communication with the devs too, not just for forming a player community. So I don’t think it’s as simple as them not being willing to use those venues as a method of acquiring new users as much as that doing so would be a rather large trade off, and not just in cost for paying CSRs, but also in long term retention.
‘Sides, the net isn’t so monolithic that the official forums need to worry about that directly, acquisition via forums and other internet communities can occur via non-official venues. Fansite listings are usually provided via official sites too, so it’s not like it’s hard to find a forum to ask questions on. Of course, if the devs wanted to have a direct hand in this, they could easily have an open sub-forum for non-account people. The question is whether or not that would really provide much more of a benefit than a fansite’s forum would.
Eolirin the Contrarian wrote:
Avoiding work and controlling costs are different activities. In this and similar cases, professionals have simply decided to avoid working!
Any company that voids a service agreement with a customer (e.g., online game service subscription) due to the customer’s misuse of an extrinsic benefit of the agreement (e.g., online community access) needs to revise their policies.
You’re assuming that the customer service function is poorly managed. FYI: CS is Customer Service. CM is Community Management. CSR/TSR ≠ community relations.
You’re assuming that human resources are fixed.
You’re assuming that “increasingly hostile” is a condition of being for communities.
You’re assuming that there’s only a single channel for customer feedback.
You’re assuming that direct communication with developers is actually a goal.
I’m a member of the Community Managers Group. I’ve read and discussed community managers’ thoughts on integrating community relations with marketing, and in fact, I was scheduled to write a chapter on the subject for a book.
Most community professionals recognize the advantages of integration, they’re enthusiastic about integration, they want integration, but regardless, integration is difficult to make happen whether due to budgetary and cultural constraints or simply because some people want to avoid doing more work.
According to the “Improving Game Marketing” study conducted by Frank N. Magid & Associates for the MI6 Game Marketing Conference,
— official websites are vital to providing information that consumers need to make purchase decisions;
— consumer opinion leaders are more likely to use official websites as sources of information;
— official websites are the most important source of information for self-buyers (as opposed to gifters);
— the official website tends to be more significant to PC game buyers;
— and a 1/4 of consumers don’t even notice advertising/marketing campaigns.
Both official and fan communities are of great importance to the success of interactive entertainment products and services.
I think one of the hurdles you’re facing in understanding why restricting access to an entire product community to only buyers is such an awful tactic is that you’re approaching the issue as though there aren’t real solutions as well as solutions to the problems doing so attempts to avoid.
For example, there don’t have to be extraneous techical support and quality assurance costs due to, say, consumers with pirated copies of games [which was a big issue for the now-defunct Ritual Entertainment] if access to technical support was limited to buyers/subscribers.
Nonsubscribers should be able to participate in discussions about the product — their feedback is just as important as feedback from subscribers! — and they should be able to participate in general discussions so that they can be pulled into the community and therefore influenced to subscribe. There are more reasons, but I think that’s enough vertical scrolling for now.
We’re a regular old subscription MMO, actually.
And FWIW, I’m the “someone” who decided random dudes from the internet, people we banned, and angry ex-customers shouldn’t be able to rant at the world using our megaphone. 🙂
We’ve considered having a forum that’s open to everyone to allow people to ask questions and the like. It wouldn’t apply to all the forums, but would address some of the concerns that Morgan has. We haven’t actually DONE it yet, but may one day.
Before I say anything else, I want to make it very very clear that I’m not against integration of community and marketing. I think it’s a wonderful idea. And I don’t think there are unsolvable problems stopping it either, but I don’t think it’s even remotely fair of you to simplify the reason down to simply being “people aren’t willing to work.” It’s not easy to do. And there’re a lot of issues that require a lot of resources to be dedicated to them before they can be resolved. Either in staff or in research and development. I’m not entirely sure that the tangible benefits are actually worth those costs depending on the particular game and the particular situation of the company making it. It’s great if you can do it, but not doing it doesn’t make you lazy.
I would agree that being banned for an extrinsic activity would be a bad way to do business. But in the case of community forums it’s not extrinsic. At least in terms of exclusive to subscriber forums, the official community forums are part of the service being provided, not something separate from it. They’re there as a way of broadening the experience being offered by the game world and really shouldn’t be viewed as being a separate entity. As an analogy, a country club can have golf courses and a restaurant. If you act out at the restaurant and get kicked out, you shouldn’t expect to be able to use the golf courses, even if golf was what you were there for.
Management and staff size has nothing to do with it. I’m assuming, and I’ve got far too much evidence behind me to not, that any official forum is going to eventually become laden with a huge amount of negativity, and that the mods are going to have to constantly be dealing with it. This has little to do with the number of people involved running the positions, or with how those positions are managed, but more with the fact that it’s a completely thankless job that involves the people who interact with these groups being constantly misrepresented, insulted, and raged about. Even increasing the numbers of mods does not resolve this problem, as there’s an extreme amount of diminishing returns there. Heavy application of the ban hammer does not tend to resolve forum flare ups, so increasing the amount of staff doesn’t necessarily allow things to be dealt with easier. And this also applies to devs who post and community managers who post even if they’re not involved in moderation, because the tenor of the boards directly impacts their ability to function on them. Remember, problems tend to increase exponentially with size, and not linearly.
I’m not assuming anything with this one, though I’m not generalizing out to “communities” even remotely. When applied to specifically official game forums, increasingly hostile *is* a state of being, though. The longer a game remains in operation, and/or the larger it’s population becomes, the more hostile the members of it’s forums will tend to be. This tends to be progressive too. Hostility rarely decreases long term on the forums. Walk around in any official forum for a game that’s been around for a while and this becomes extremely obvious.
I never said anything about forums being the ONLY form of feedback or communication, but it is *a* channel of feedback, and invariably it’s one of the largest. And it’s usually the only two-way one. Players of MMOGs tend to like to be able to hear back from the devs, and they tend to like to be able to at least have the illusion that there’s an ongoing conversation. Dialogue is important. So if it’s not a goal for there to be direct communication of some sort, well… I think that company really “needs to revise their policies.” Forums are a big big part of having that communication framework; increasing noise makes it much harder to manage.
And the last bit that you’re mentioning conflates Official Website with Official Forums. Regardless of the import of the official website as an information source (I’m not questioning that, I know it’s true), the official forums don’t necessarily have the same pull anyway, even if they are open and accessible. One is an information source, and the other is a community structure. Communities aren’t necessarily viral, they can be insular too. And game forums tend to be just as likely to be hostile (if not more) as they do to be helpful to players looking for genuine information.
I’m willing to bet that there wouldn’t be that much in the way of people swimming up stream from non-subscriber to subscriber just because of official community forums, at least without completely redefining what an official forum looks like. Again, forums do not exist as information sources. They’re feedback and community devices. Communities are great, but without some sort of “in” they’re not very helpful a device for someone who isn’t already part of them. This can be dealt with, yes, but it’s not so simple. Because remember, the bulk of people who are on your community forums are there because they’re they’ve got something to complain about or are having an issue with something. Most of the people who are absolutely happy and content with the game tend to spend more time actually *playing*. So there’s a greater tendency for the forums to tend toward hostile to begin with. And a hostile forum population doesn’t really help when it comes to adding members to the community.
Eolirin wrote:
I don’t care if I’m being “fair.” That’s the way the world works. Humans are naturally inclined to avoid complexity and thus avoid work. That’s why we have tools: to simplify complex tasks, sometimes to the point of automation. Avoiding work is a strong motivation, particularly in poorly managed organizations where everyone hates their jobs.
You’re saying, “Marketing isn’t worth customer service costs.” That’s the same dumb attitude that accounting-driven organizations have towards marketing. “We need to cut costs. Let’s cut our capacity to generate revenue first!”
We just had a huge debate about this subject. Access to official community forums is an extrinsic benefit of the provider’s agreement to provide customers with an online entertainment service. The debate we had was about the importance of communicating the extrinsic value of such benefits to consumers to influence their decision making.
If you need to be thanked for doing your job, join the military. You’ll even get yellow-ribbon bumper stickers.
Again, you’re assuming that traditional resolutions are the only tools at the disposal of community management. There are many, many different ways to handle community problems. Sometimes the community software needs to be customized. Other times the policies and procedures within the community team need to adapt to ever-evolving people.
Your assumptions still lead you to baseless conclusions. Guess what: as a product matures, so do the resources, which are increasingly allocated to other projects. As resources allocated to a product dwindle, so does the performance of the product’s managing team. I just love when people try to blame consumers for the failures of management. “Our customers are getting hostile because they’re assholes!” Whenever something goes awry in a community, management screwed up. Whenever management screws up, someone should step up to fix the resulting problems.
I’m reminded of what Guy Kawasaki said about marketing new products:
Bad managers blame the wrong people. They should be blaming themselves.
One of the primary responsibilities of community management is to serve as a liasion between the consumers and the company. A liasion is a middleman. When you go through a middleman to provide feedback to, say, developers, you’re not providing feedback directly to developers. You’re providing feedback to a filter who relays all, part, or none of your feedback to the right people. Moreover, anyone who’s not the CEO, the PR representative to the consumers, the community relations team, and the technical support team really shouldn’t even be posting in community forums until they’ve had public and media relations training.
Official forums are part of official websites. For the purpose of market research, when you ask consumers about official websites, their opinion is based on their experience with all or some elements of official websites, including official forums.
That’s just not true. When a community is managed properly and effectively, you don’t create aggregious problems. You don’t create hostile members. In the same way, when you run a board of directors properly and effectively, you don’t have dissent, ill will, and drops in productivity. Communities are not necessarily highly negative and hostile organizations, but if you approach them that way, you’ll make them that way. Genesis 1:27 as a management principle applies here.
I have to come down on the side of one Richard “Lowtax” Kyanka and say that a forum that’s pay-to-play is, from a business perspective, superior to one that’s completely free. Not because it generates revenue (though that’s certainly a good thing in business) but because the barrier to entry will make the forum a better place for those that do use it.
Having said that, what I would beg for is a “Messageboard-only” subscription option at some small fraction of full subscription price (does $1.99/month sound bad?) along with at least one “Guest” messageboard that anyone can post on, subscribed or not. I think that would strike the right balance between the anarchists and the stalinists, as it were. 🙂
Peter S. wrote:
I don’t think his example, Something Awful, proves his point. I’m actually registered at Something Awful, but I’ve never made a post. Browsing the Leper’s Colony — the list of recently banned users — you can apparently have your account suspended or banned for any reason, including not posting but whatever. Something Awful is a better example of how mysterious rules and strict moderation, without regard for individual investment of either time or money, per those mysterious rules can subjugate a community and bend their participation to the will of the community operators.
Thank you for your interesting reply and thanks to all those who have taken the time to respond.
There are a lot of issues here – so many you probably could start another couple of separate topics!?
Am I frustrated by the response?
Yes and no.
Morgan Ramsey said:
Where stocks and shareholders are concerned there is a requirement for company accounts to be available. In that case it becomes impossible to hide real data on the number of subscribers – so I dispute that trader confidence is a motivation. Player confidence is another matter however?
But in any case, this only applies to games post retail release and does not explain the secrecy surrounding data related to Alpha, Beta and Open Testing. Shareholders also probably don’t care too much how many subscribers post on the forums unless there is an expense involved?
I do appreciate that some of this data is commercially sensitive but it actually suggests to me that the reluctance to release this data openly is more because it is “embarrassing” than commercial?
As Raph said:
This is where I believe the truth may lie. As I said in the original thread
http://www.burningsea.com/forums/showpost.php?p=361015&postcount=10
In fact, just last week the MMORPG.com newletter (#128, 19th May 2008) led with 2 of the 3 Feature Articles relating to Beta Signups. (1,000,000 for AoC and 100,000 for Stargate Worlds.)
http://mmorpg.com/gamelist.cfm?READ=10719&VIEW=news&GAME=191&bhcp=1
http://mmorpg.com/gamelist.cfm/game/242/view/news/read/10750
WAR actually had a visible counter and PotBS also made a big deal of offering 10,000 new beta positions too IIRC?
Why is this a concern?
My interest in this is as a player. I want to play good games.
I am a firm believer that the end consumer should be presented with a “polished” and “finished” product. Advertising your product is one thing but pre-release testing should be just that. I don’t care if you sign up 1,000,000 beta testers – I want to know that the “testers” you select are doing their best to see that I get that product and that the development companies are also addressing the issues those testers are reporting (Again, this is another issue – lately I have my doubts about this too.)
And so I am left with a conundrum. Development companies that are only too keen to release info on how many people have applied to become Beta Testers and release data on how many Beta Testing invites they are issuing – and yet when presented with the simple question of “what percentage actually ‘test’?” claim that the data is “sensitive”?
How would it be a “trade secret”? Unless, of course, the percentage is so low it shows that this sort of “Beta Testing” is ineffective except as a marketing tool?
That said, I have been an advocate of allowing members of the public to test games in this way. I do appreciate that games have become incredibly more complex with time and a modern MMORPG almost certainly requires a “Monte Carlo” style of testing at some point.
The issue appears to be that if the testing goes on for too long then the players will have played the game for free and move on when the game goes retail and paying subscription starts?
Or maybe Beta Testers need to be policed and ‘culled’ if they are not contributing reports on a regular basis?
The discussion on forums is a whole new issue which I also have an interest in for a number of reasons. But, way too much to discuss in one go (sadly).
@Dana “Gophur” Baldwin
Yes, I would be very interested in any data you care to release. Of course, WWIIoL is unusual in a few ways (I am an active subscriber BTW).
Which brings me to a few interesting points raised by other posters:
Banning a Forum account is one thing – banning an entire account for forum conduct is entirely another. The two things should be kept separate – I would not be surprised to see legal challenges to this in the future.
Besides which, players should never feel they cannot express their views (even the negative ones) freely (if you want good games that is!). People really only get passionate when they actually care. Also, the Moderation has to be very ‘professional’ and even handed for this to work.
WWIIoL is actually a very good example of this. Players are free to even argue with the Devs provided it is done in a mature way. Players can be cruel at times, but most mature people on both sides realize that it is not personal.
And
From a business and legal point of view taking money also elevates a forum from an extrinsic benefit to a primary service with all the added liabilities that implies.
In terms of Community Forums there is a lot more that could be discussed but I am interested to note that many of you appear to imply that the forums are completely separate from the game and result from the game.
I challenge both counts.
Forum Communities are often well established even before the game gets to retail and I contest that the atmosphere of the official forum community can and will ultimately reflect the attitude in game and vice versa.
There is a whole new discussion right there.
That will do for now – it’s late and I have Pirates to sink and Aircraft to shoot down… 😉
Having been a game board moderator, I can tell you that it is relatively easy to tell people who have legitimate concerns (no matter how passionately they express those concerns) and those who like to stir up drama for the joy of rolling around in the resulting chaos. The former rarely need anything more than a gentle reminder to watch their language now and again. The latter need to be excised… and if you have the ability to excise them from the game as well as the boards, do it early and save your other players some grief.
If the marketing department wants an open forum for anybody and everybody to drop by and visit, give it to them. And let them moderate it. And see how long it lasts, and how many units it moves. And if it’s a cost-effective model of civil discourse and a brilliant promotional tool, I’ll go back to my armchair and hush up.
It’s not just those that make negative comments and the “trolls” that are a problem though. Moderation needs to be even-handed because a board over-run with “fanbois” can be just as hostile.
In the case of MMOs the board is very important in building the community you want IMHO.
If you don’t want your MMORPG dominated by a bunch of smack talking “leet” speakers then discourage this behaviour from the beginning – even when they flatter you.
@Morgan
Well, certainly, I wasn’t planning to bring up Lowtax’s specific iron-fisted rule of his forums. Rather, you might want to watch the university presentation he gave (University of Kentucky, I believe). Actual execution of the presentation aside (I don’t think he was kidding when he said he put it together at 3:00 AM the night before), he really seems to know what he’s talking about.
Yukon Sam wrote:
Marketing ≠ sales.
I suppose the critical issue here is what kind of tool do you think a forum is?
Is it a place to let people vent so they’ll get over it and go back to playing? (answering machine to scream at)
Is it a place to discuss concerns and give feedback?
Is it there to look good and be full of cheery posts about how wonderful the game is and you should rush out and buy it too?
That’ll probably tell you what kind of access and moderation policy to pursue.
Hello Again,
I just thought I would drop by and bump this to see if anyone here has any further thoughts or comments to make?
Also, I would like to check (Raph / Admin) if comments made here are quoteable?
I have done a bit more research since my original question and have come up with some more interesting information 😉
At present I am still collecting info and trying to knock togther an article.
Oh and in response to Kerri Knight:
This is a good point and I would suggest that this depends very much on what sort of game you are developing and what stage of development you are at?