Measuring MMOs
With the release of the latest edition of SirBruce’s chart of MMO populations, the debates have once again erupted over not only the validity of the numbers, but also over the how to even talk about the populations of these virtual worlds. It’s a discussion that has come up before, of course.
With recent versions of the charts, Bruce has started to include stats such as “peak concurrent users” and “average concurrent users.” With the rise of non-subscription services, the industry is having to adapt to different metrics altogether. Services such as Blizzard’s have had to attach statements to their press releases to gain credibility for their figures, after much skepticism from competitors.
World of Warcraft’s Paying Customer Definition
World of Warcraft customers include individuals who have paid a subscription fee or purchased a prepaid card to play World of Warcraft, as well as those who have purchased the installation box bundled with one free month access. Internet Game Room players having accessed the game over the last seven days are also counted as customers. The above definition excludes all players under free promotional subscriptions, expired or canceled subscriptions, and expired pre-paid cards. Customers in licensees’ territories are defined along the same rules.
The original metric people used, of course, was subscriptions. This is a great metric for assessing how much revenue a game is getting, but it does have a few oddities in terms of determining the popularity of a game.
- Players often have multiple accounts. And when I say “often,” I mean “really often.” While we know that price sensitivity for users can get pretty high, enough so that a client purchase and any fee at all is a significant barrier to most people, we also know that that among the hardcore, it’s almost nonexistent. It’s not uncommon to hear anecdotes about the people who own 10 or more accounts; this leads to an overcounting of actual people on the service.
- Even game operators have a fair amount of uncertainty around the actual incidence of this, because accounts get mapped to credit cards. The average US household has many credit card numbers — I have heard figures from 7 to 14 cards per household. On top of this, uncertainty goes in the other directions as well: multiple accounts on one card, with no sense of how many people it maps to — one head of household paying for three accounts for her kids, or maintaining them for herself? You can try minor cleanup on this by mapping against addresses as well, but even a minor typo will noise up the data.
Because of this, there have been estimates that the average accountholder in subscription based MMOs actually accounts for two or more accounts. In other words, half or more of the subscriber base isn’t real. This is something that operators need to take into account when they assess price sensitivity and commitment to a game; a change might cause those players who run 10 accounts to choose to axe 9 of them, while retaining one to keep a toe in the water.
- Quasi-subscribers also complicate matters. At any given time, assuming a game with decent inflow, there is a fairly large proportion of the userbase that is playing for free, based on promotions, loyalty programs, free months provided by the box, and so on. These are folks who are able to play the game, but aren’t necessarily paying for it, or paying for it directly. The most obvious one of these, and the one that causes Blizzard to have so much stuff in their subscriber definition, is game time cards (which in Asia are often sold on shorter blocks of time than a month), but there’s also PC baang license arrangements and other such complications.
- One example of this is aggregator packages which offer access to multiple games for one fee; you can break apart the numbers to try to determine which game they were playing, but at what point do you count them as equivalent to a subscriber?
- The most interesting quasi-subscriber figure is that surrounding people who just don’t log in very often. At any given time, you can expect the number of unique logins over a month to be drastically lower than the number of subscribers that month — by as much as 20%. In other words, at any given time, 1/5th of the subscriber base won’t have logged in during the last month.
- A lot of people forget to cancel. It’s hard to tell how many of these there are either. The 20% figure mentioned above is not really representative, because a lot of those are folks who maintain a sub, but simply doesn’t use it very often (or extra accounts for muling or special purposes, or whatever). The number of people who simply forget to cancel is likely in the single-digit percentages, I’d guess.
- Plain old noisy data also has a small effect. For example, someone can enter the service, get registered, but bail before putting in a credit card number. Are they a subscriber? They’re not actually able to play the game, but they are an account. There’s a surprising number of people who register, pay, and then don’t log in — their machine couldn’t handle the game, perhaps.
Of course, subscriber metrics, despite these problems, are still a damn sight better than registered users, which typically
- include everyone who ever created an account, whether or not they have ever even logged in. This is a decent metric for penetration of sampling, but lousy for determining the population of a given world.
- include people who stopped playing, and therefore are always going o rise, because they are an accumulation stat.
- frequently double-count expansion account keys.
A lot of the early skepticism of Korean figures arose because registered users was a common figure used.
Korea in fact has a fairly open numbers policy — there are ratings systems that measure peak concurrency for the various games kind of like Nielsen measures concurrency for television shows in the US. This has led to popularity charts being fairly available, but it of course ignores the fact that concurrency is also a bad metric in a number of ways, even for measuring popularity.
- First off, concurrency follows predictable weekly patterns, fluctuating up and down noticeably based on the day of the week. Because of this, an individual peak isn’t really reflective of the overall usage.
- When people do use average concurrency numbers, they vary the trailing date length. For example, Second Life uses 60 days, other companies use 30 days, and yet others in Korea use 7 days.
- Most importantly, concurrency ignores session length.
Do the thought experiment. Let’s say that there is a game with 24 users. The game permits only one session per day, and it mandates that said session last exactly one hour.
If the users all log in at peak time (unlikely), the game will show a peak of 24. But more likely they will log in spread out across the hours of the day, with peak time showing a bulge — perhaps of 10.
Now change the session length to 24 hours. All of a sudden, the peak is 24 again, because everyone overlaps.
Now change the session length to 1 second. The peak drops to 1, because nobody overlaps.
This is shown most dramatically in the difference between MMORPGs and more casual games. Peak concurrency for the RPGs, which demand multiple hours of play time in a session, is always significantly higher than that for the new breed of “casual online games” such as Kart Rider. Even more casual RPGs will exhibit this; SWG and Planetside, for example, had a much shorter session length than the EverQuest games, and so would not get similar peaks.
- This means that our tie ratio rules of thumb are obsolete. Once upon a time, the rule of thumb was to take your peak concurrency and multiply it by 4 or 5 to get an estimate of the userbase’s size. But in the more casual games, such as Kart Rider, you actually need to multiply by ten because of the brevity of session lengths. The fact that Kart Rider held the #1 spot on the Korean charts despite its userbase being underrepresented by half speaks to its popularity.
Now, this is of course the point at which the more business-oriented developer chimes in and suggests using revenue instead, such as “ARPU,” which stands for Average Revenue Per User. This is often broken into ARPU/Hour, ARPU/Month, and so on.
But even this figure is often misleading. For example,
- the casual games boast figures much higher than the figures for subscription games; it’s not uncommon to hear of APRU in the $30/month range. But that’s often “average revenue per paying user,” because the free games get millions of players who don’t pay a dime. Here the revenue per user is being inflated by not counting every registered or active user.
- Yet they still incur costs on the many players who are playing but not paying. Without access to profit and loss figures, we cannot tell whether a high ARPU is really a success or not, and it’s entirely possible for a low ARPU to be immensely profitable, if the operational costs are low enough.
- Of course, there’s also the sunk costs in development that ARPU does not take into account. You could have a great ARPU, but if you overspent prior to launch, still not be very successful.
- Lastly, you could be a game like Achaea, which has a relatively small userbase and a great ARPU. Very successful business, but we’re not learning a lot about how to compare it to other games; it mines its niche very well, but doesn’t scale (and doesn’t particularly want to). It’s going to be axiomatic that with a niche audience, you can mine it for more dollars because the fans are likely to be more hardcore and you can target their desires very precisely. But you aren’t going to get a good sense of the total revenue picture from ARPU alone; a lower ARPU traded for massively more users may well be a better business (and in fact, generally seems to be).
Revenue in general is a tricky thing. People keep taking the 6 or 6.5m users reported by Blizzard and multiplying it by $15 to arrive at astronomical revenue figures. But in practice, they probably only get less than a dollar per user in China. Their partner, The9, only reported total revenue for a quarter at $22m last year; comparing that to the frequently cited figure of “well, take 2m times $15 a month” and you can see the gap. Blizzard also doesn’t get all of the revenue, of course, some of its goes to the partner.
This business equation is going to vary in every territory. Revenue is, of course, immensely important ot measuring the success of virtual worlds as businesses, but just as in any other business, revenue is not the only story: reach, costs, growth trends, and so on also matter a lot.
The figure I prefer isn’t publicly given out by anyone, and that’s average weekly uniques. Basically, instead of counting how many people are online at one time, or counting how many are paying, doesn’t tell you how many are actually playing. But counting how many people logged in each given day, and then averaging that out across the week to smooth out the daily fluctuations will give you a very good sense of how many people are actually playing.
The other nice thing about uniques is that you can treat it as a percentage of the paying subscriber base to determine the stickiness of the game.
One of the flaws with the other metrics is that they are often difficult to compare across multiple games. Peak concurrency doesn’t work because of the session lengths. Subscribers are counted differently by different providers — and sometimes even by the same provider within a stable of offerings. But you can compare uniques percentage to determine which is the game that kepss more of its users coming back day after day.
A fall in uniques will also presage a fall in subscribers, often by weeks; you can see when people are losing interest and failing to log in regularly, and that will generally be a very good leading indicator of cancellation. A high uniques percentage in a week (80% or more) is likely to indicate a growing game, whereas one with only, say, 50%, is likely to be shrinking.
Most telling is graphing uniques over time, and then marking on it the dates of significant events in the game’s history: every patch, every expansion, every promo, every marketing push, every scandal. You’ll be able to objectively assess whether those changes made players want to play more or less, and use that knowledge to guide your future decisions.
Of course, Bruce isn’t likely to be given the uniques percentage by any company, which is a shame, because I think it’s the only really good way of measuring MMO populations.
80 Responses to “Measuring MMOs”
Sorry, the comment form is closed at this time.
Original post:Measuring MMOs by at Google Blog Search: credits cards people bad credit
Wars has sold over 2 million copies by this point. And while that number is somewhat circumspect because it doesn’t account for people who’ve bought multiple copies and the like the numbers for “subscribers” to an online game is likewise an imperfect measure of how many people are actually playing. My admittedly brief investigation reveals nothing further about other metrics from the world of GW. Still, it’s useful as a comparison, if nothing else, and were there actually anything close to half that
M Show 132 – Second Life Population, Evening Harder Intro – Christopher Penn from the Financial Aid Podcast News: Sirius XM merger? Second Life population more around 200k Raph Koster on Measuring MMOS Talk: Working like a dog, everone’s sick and it’s 70 degrees Impossible Dilemma [IMG]CAPOW – Marketing, PR and Advertising Podcasts – Eric Schwartzman Entertainment: Dead Rising Kevin Smith
for Second Life? When the media announces that x virtual world has a high number of subscribers/residents/foobars, doesn’t that influence the numbers by causing a bunch of new sign-ups and virtual tourists? How do you calculate meaningful numbers? Raph’s Website – Measuring MMOs
limitations of the design, so far, although time will tell. (I know! It’s a beta!) [IMG Barbiecolour_2] Course, this is just registered users, not returning players, and as Raph will tell you, quite rightly, registered users hopeless for measuring popularity. It’s returning users you want to measure, and the article over at Scientific American doesn’t mention that…
Celebrity Heights – How tall are Celebrities � SoundManager 2: Javascript Sound for the Web � How To Make Noise Blocking Headphones (Technology: Gadgets) � Uncanny valley – Wikipedia, the free encyclopedia �Raph’s Website >> Measuring MMOs� Ballardian: the World of J.G. Ballard >> The Ballardian Primer: Surveillance Cameras
Is the number of multiple-account holders high enough to be statistically significant? More than the +/-? I’m not challenging your stats, I’m sincerely curious.
[…] Revenue in general is a tricky thing. People keep taking the 6 or 6.5m users reported by Blizzard and multiplying it by $15 to arrive at astronomical revenue figures. But in practice, they probably only get less than a dollar per user in China. Their partner, The9, only reported total revenue for a quarter at $22m last year; comparing that to the frequently cited figure of �well, take 2m times $15 a month� and you can see the gap. Blizzard also doesn�t get all of the revenue, of course, some of its goes to the partner. This business equation is going to vary in every territory. Revenue is, of course, immensely important ot measuring the success of virtual worlds as businesses, but just as in any other business, revenue is not the only story: reach, costs, growth trends, and so on also matter a lot. The figure I prefer isn�t publicly given out by anyone, and that�s average weekly uniques. Basically, instead of counting how many people are online at one time, or counting how many are paying, doesn�t tell you how many are actually playing. But counting how many people logged in each given day, and then averaging that out across the week to smooth out the daily fluctuations will give you a very good sense of how many people are actually playing. The other nice thing about uniques is that you can treat it as a percentage of the paying subscriber base to determine the stickiness of the game. One of the flaws with the other metrics is that they are often difficult to compare across multiple games. Peak concurrency doesn�t work because of the session lengths. Subscribers are counted differently by different providers � and sometimes even by the same provider within a stable of offerings. But you can compare uniques percentage to determine which is the game that kepss more of its users coming back day after day. A fall in uniques will also presage a fall in subscribers, often by weeks; you can see when people are losing interest and failing to log in regularly, and that will generally be a very good leading indicator of cancellation. A high uniques percentage in a week (80% or more) is likely to indicate a growing game, whereas one with only, say, 50%, is likely to be shrinking. Most telling is graphing uniques over time, and then marking on it the dates of significant events in the game�s history: every patch, every expansion, every promo, every marketing push, every scandal. You�ll be able to objectively assess whether those changes made players want to play more or less, and use that knowledge to guide your future decisions. Of course, Bruce isn�t likely to be given the uniques percentage by any company, which is a shame, because I think it�s the only really good way of measuring MMO populations. Link: Measuring MMOs 150)?150:this.scrollHeight)”> __________________ The tools suck! — Raph Koster […]
There have been public surveys done on MMO populations asking how many accounts they have, and usually the answers show that a largish number have 2 or 3 accounts. I am not at liberty to reveal any confidential data, of course.
It’s also very hard to get a good handle on even internally, because usually, multiple account holders do not want to be found. But do the math — Assume a game of 200,000 subscribers. Let’s say only 2% of players have 10 accounts — the extreme hardcore, perhaps using them for gold farming. That’s 4000 people. Those guys then represent 10% of the total paying userbase, since they pay for 40,000 accounts. I’d guess the commoner case is 2 and 3 account holders.
Even worse, this can severely distort your perceptions of a game’s size. Let’s say you have one game that (tacitly, for sure) encourages farming, and another that doesn’t. The farming-prone one is the one we describe above. The other one is smaller, only has 150,000 users, but there’s no reason to run multiple accounts — perhaps they give lots of alts, permit running bots in groups, and so on. Odds are that the 150,000 user game actually has more people playing it, even though it derives less revenue and has fewer “subscribers.”
Raph –
Why reduce things to one number? These various metrics each highlight different facets of an online game and affect its health and success. While you may discount accounts where the player is not logging in, these are virtually perfect from a business perspective (as long as they last). After all, they add no infrastructure cost… this is the entire premise of the health club business.
Blizzard’s abuse of the subscriber number is fairly deceptive, it hides the number of people who have “converted” to a recurring subscription.
It certainly would make sense to break out these numbers by country for the partnership factor as well as the distinct business models.
Average concurrent users and peak concurrent users are certainly of interest if you are looking at modeling infrastructure costs. As is average session time.
It would be good for the industry to move towards a sensible, common set of metrics. There is still more to be learned from sharing than from hiding data.
If I recall, the primary intent of the MMOGchart effort was not really to chart the rise and fall of individual titles/services, but rather more of an attempt to get a feel for overall MMO market penetration: in which case, the multiple account issue comes back and complicates things yet again.
My own situation is probably a good (if extreme) example: I have a Station Access account (for EQ2 and SWG), a CoH/CoV joint account, a DDO account, a Horizons account, and a WoW account: 5 (or 6) total… and I’ll be adding Auto Assault soon. A year ago, add Lineage 2 and drop DDO. 2 years ago, drop the Lineage 2 and add 2 Shadowbane accounts… and so on.
So even as a measure of “total audience growth”, there is likely plenty of subjectivity in the numbers, an issue which even title-by-title uniques would not address.
First of all what an awesome post Raph, this seriously breaks down the means by which the industry is measuring its success (valid and flawed)as far as measuring its consumer base
As a data miner of course I’m biased in that I believe the value of data is to not only understand ones business but to make it better.
“While you may discount accounts where the player is not logging in, these are virtually perfect from a business perspective (as long as they last). After all, they add no infrastructure cost… this is the entire premise of the health club business”
Steven thats a great point, but I think whats more important is the longevity and enjoyment the player base is getting from the game. Just as in a health club, sure its good for business, from a pure profit standpoint, but are the members living longer and enjoying it?
The thing that pisses players off is when the company treats them like some unit, or just another “script”. Thats when companies start losing players goodwill, economic and otherwise….
I dont think any business model that Ive heard of succeeeds in the long term when the proprieter of the business takes thier money then gives them bad service (boring or just plain bad gameing experiance)
I believe a reason for measuring this would be to ensure your organization is proactive rather than reactive. Because people not logging in regularly generally means somethings going on( playerbase is bored or unhappy), and when there is a lag time on cancellations and a game design supports multiple accounts as mentioned above one might consider finding out WHY the playerbase is unhappy/bored.
Thats the value of data….proactive decision making and business longevity
Again great post..
Claiming Blizzard earns $90,000,00 USD per month in revenue sounds a lot better as a response than "well, it’s really difficult to estimate from an outside perspective". 😉
Now, you said that in practice they might get less than a dollar per user in China. Is that profit or revenue?
[…] Raph Koster wrote an interesting post talking about the numbers on MMOGChart.com. He points out how different companies will use different metrics, the weaknesses of the different metrics, and what he thinks would be the ideal metric (unique connections per week). […]
*sigh* Manual trackback: Measuring Success
Great post, Raph.
[…] Kressilac Smack-Fu Master, in training Tribus: Louisville Registered: February 24, 2004 Posts: 138 Posted document.write(”+ myTimeZone(‘Thu, 01 Jun 2006 20:16:49 GMT-0700’, ‘June 01, 2006 23:16’)+”); June 01, 2006 23:16 Interesting blog post by an insider that gives insight into just how complicated things really are.Raph Koster’s Blog […]
[…] Comments […]
You know what I find interesting — UO still has as many or more subscribers than when I played it back in 1998. I think that’s Fantastic. 🙂
Even if farming-encouraged games may get more revenue through subscriptions, doesn’t that cut down on the total ARPU? I mean you’ve got people willing to pay for secondary goods (ingame cash / items) relating to your game, but the portion of their budget they’re pointing at your game goes to the Farmers, not to the guys running the game.
Excellent remarks, Raph, and very comprehensive. I agree with just about everything you said, although I think perhaps monthly uniques might be a better measurement. It’s probably too easy to overreact to week-to-week fluctuations. I also agree with the poster that all of the metrics have their own advantages and disadvantages, and what we really need is a basket of data for every game. But as you pointed out, I have a hard enough time getting subscription figures. Even if I could move to a different metric tomorrow and get some companies to agree to it, there’s still the wealth of historic data that wouldn’t be available.
I’m still trying to pin down’s Blizzard’s numbers more precisely, and there may be a revision downward on my charts in the future. It’s clear there are over 4 million accounts in China, but how many of those are unexpired, or were active in the last month, or in the last 7 days, is not entirely clear, since WoW did not explicitly quote that customer definition in their recent press releases. Still, the “truth” is still going to be quite impressive, far more than any other subscription MMOG.
[…] Raph Koster weighs in on how to measure MMOs. Great stuff. Bruce […]
*sigh* Manual trackback:
The mechanism Raph has for “Trackbacks & Pingbacks” seems to be based on incoming links. I’d recommend that you create a token link in your entry to this post, and then click on it. It should show up, then.
[…] MMOG guru Raph Koster has posted an excellent write-up on his blog on how to measure MMOs.� I wanted to provide a link to here because it provides a very comprehensive overview of both the advantages and shortcomings of using subscription numbers to track the market.� Not only are these numbers imperfect, but they also don�t provide a complete picture.� They are only one of a variety of metrics that could be used, if such data was actually made available by those in the industry.� We may even be moving towards a point where subscription-based MMOs will be a thing of the past. […]
[…] Raph has chimed in on the problems with comparing MMO populations to each other. It’s excellent, and touches (and improves upon) many things I was thinking about posting, so I’ll just thank him for saving my time. […]
I always thought the metrics were a bit on the odd side. What I’d really like to see, though, are metrics for certain continents.
For instance, just the ones for North America. Of course, games that are doing well in, say, China or Korea, won’t like that idea. That includes Blizzard, which purportedly has somewhere between 4 and 5 million of its WoW subscriptions in China. Lineage 1 and 2 would also be heavily hit, as the majority of their subscriptions are in Korea.
Whether or not you count people with multiple subscriptions as separate accounts depends on which metric you’re going for. If you’re looking for the money amount, then they would need be be counted as separate. If you’re looking for the total number of people that play your game, I wouldn’t count them.
There’s also the problem where they may let other people use their accounts while they aren’t using them.
Just something to think about.
Good article, Raph!
I’m still a little fuzzy on using uniques as the best metric when it comes to games where multiple accounts is the norm (or at least higher in comparison). Doesn’t the problem which multiple accounts contributes to subscription numbers still exist in uniques, or is it nullified in some way?
To use SWG as an example (the only game I’ve owned multiple accounts), I only bought one copy of JTL initially. The result was that for several months one account got logged daily while the other would only get logged when there was a resource shift. To expound a bit, a member of my guild was one of those guys who had 10 accounts; once he started his jedi grind, you rarely saw the other 9 accounts.
Conversely, when my wife and I played WoW, we both used the same account (because we could 🙂
Does counting uniques solve those problems or is it simply a better way of cutting through the noise?
Daily uniques does still fall prey to multiple account holders. But overall, people do not seem to tend to log in every account every day, so it reduces the “noise” factor somewhat. In your example, the JTL account would show up regularly, whereas the harvester account would show up rarely.
Its bigger weakness is the WoW case you described, which is shared accounts. But ALL the systems fail to handle that one.
First of all, great article.
I operate a couple of very popular MMOs in Asia. As an operator (I license games, localize them, and bring them to my market) long ago I gave up trying to estimate how many physical people play the game. It’s hard to get that number, so I rely on other measures. Every different measure has its nuances, and has to be interpreted properly.
I look at many numbers every day, here are the most important single for me:
* PCU helps me with server and datacenter planning. They’re also fun at parties. PCUs are the macho comparison, and game operators and developers love to talk about them. At parties, people also like to talk about paid subscribers, total registered accounts. It’s all hype 😉 But indicative of…something.
* If I look at one number every day, it’s average unique accounts (Ralph’s favorite measure, and rightfully so). I’ve been using that for a long time to assess traction.
* Monthly I also pay attention to paid gamer drop-offs. These are gamers who paid at least once who, for the first time have not logged in during the past month.
I’d also add:
* account sharing is rampant in Asia
* with many early titles (2D and 2.5D) in Asia, autoplay bots are common, and can contribute as much as a third to PCU numbers
* ftp (free to play, or item-selling) MMOs and casual games are becoming the dominant model in Asia, as you noted
* average play time per day (the sum of all sessions in one day) is important for MMOs, as it relates to gamer promiscuity 😉
As for comparing MMOs with each other worldwide, it depends again on what you’re driving at. I’d agree w/Ralph that average uniques is the best (though still imperfect) single measure.
[…] Again, as Bruce pointed out at Slashdot, these numbers are very stretchy. Any way you slice it, though, Blizzard has won the Massive market. Raph talks more about these numbers, of course. Weekly uniques would be nice. My big fear is that in winning the war, Blizzard has broken the battlefield. With that kind of success … what’s the point? […]
bpez, can I ask where you’re based and what games you operate? Just curious.
Well, I don’t know about other games, but I’m playing Dark Age of Camelot, and most (if not all) players I know use a “buff bot” on a secondary account, which they log in about every time they play their main account.
Very nice read, btw.
Taken on the whole, and seasoned with Raph’s thinking on numbers, the picture for MMO’s isn’t all that rosy. WoW seem the be an industry anomalie. Get out of the top ten and you have a tough row to hoe.
As to Bruce’s numbers, I can accept tham as general guidance, but at certain points he tips of the reader that he is merely guessing. An instance would be WWII Online, where his graph shows no change, yet in the text analysis he conjurs a large gain in subscribership due to a euro re-release, and does so with no data, and no evidence, not even anecdotal.
Well, I provide some rumor in the analysis so people can get the context that goes along with the number. I’d say my comments about the European release for WWII Online count as “anecdotal” but it’s true that I haven’t any numbers to release yet. But those guesses don’t generally appear on the chart. I do have a couple of derived numbers on the chart, and they’re rated a “C” in the analysis; they’re not so much guesses as data points with larger than usual error bars. (I could put error bars on every number on the chart, but how big that error bar would be would just be a guess as well…)
Blizzard doesn’t just give “some” of the revenues to The9. The9 gets the lion’s share. I saw an industry analysis that would have more recent numbers but I think its behind a pay-gate somewhere and couldn’t find it again, but it was reported here (http://www.dmeurope.com/default.asp?ArticleID=4676) among other places that the deal was originally $3 million + 22% of sales, with a minimum value of about $50 million over a 4 year period.
What I find it amazing is that The9 is totally dependent on Blizzard in providing an accurate count of user hours, which in turn determines how much Blizzard has to pay The9. Even still, Blizzard is looking to get out of their deal, because the upside of their success in China has been much larger than they anticipated. We do know there are millions of WoW players in China, but exactly how many were active in the past 7 days or the past month or are still on unexpired game cards is uncertain. But I’m trying to get clarification from both Blizzard and The9 on this point.
It’s funny, of course, because AOL had to deal with this kind of definition anxiety of “active_paying” vs. “trial” vs. “prepaid” etc. years ago. Sarbanes-Oxley just made it mandatory for everybody, and reminded AOL to keep doing it.
I always figured that the easiest thing to do would be to understand where a user is in the billing lifecycle vs. worrying about whether they actually use the service regularly. And how they may or may not use it (e.g. alts). But I suppose you have to be able to know whether a person has just renewed, or whether they are N days into their renewal, or N days away from having to renew, etc. That is, you have to document (and be able to present) your definitions for what constitutes a sub. And surprise, getting those definitions and reporting on them is not straightforward. And bigger surprise, different providers may have slightly different definitons. But I think SOX is supposed to remedy some of that.
Listen.
I do not Care!
Because you and your companies will not be upfront and public about subscription numbers. Your Marketing people practice deceptive wordings.
Most every MMO has bad Customer service with regards to what is going on.
This really doesn’t mean anything. Whether something is "deceptive" to an individual depends on individual comprehension of that something. After all, honesty can be deceptive too. The most marketers can do with regard to deceptive practices is adhere to fair commerce laws and regulations.
The people who are employed as customer service representatives are often gamers and citizens of the virtual world they service. When their shift ends, they’ll return home and likely play the game as normal players. They know the game inside-out. They usually know "what’s going on". In addition, they are far more knowledgeable about problems players experience daily than players who are not employed as customer service representatives. Their perspective on the game enjoys a breadth unmatched by the typically selfish concerns of gamers who complain about "bad" customer service. Regardless, their responsibilities include helping players succeed and they’ll do what they can to help.
These people should be respected, not admonished.
[…] menus = document.getElementsByClassName(‘menu_container’,’site_menu’); menus.each( function(element) { new Dropdown(element.id,{width:150,display_trigger:’mouseover’,group:’site_menu’,use_submenu:0}); }); Measuring MMOs Araman posted on 6 Jun 2006, 12:14 PM As a followup to our previous story covering the fact that WoW has secured 50% of the MMO market share, Raph Koster, a name now synonymous with the MMO gaming industry, has a very interesting and detailed writeup on the mechanics behind the metrics in measuring MMO revenue and popularity.Those following the trends of todays MMO gaming industry, and even people who are interested in a little more insight as to what makes up the playerbase should head on over to Raph’s official website. News from WarCry.com […]
[…] https://www.raphkoster.com/2006/06/01/measuring-mmos/ […]
Ralph , You have gain the respect of those who have been shun by SOE by leaving the company itself but to quote Smedley about how bad news travels faster then any good news is a little elementry for such a sage of this industry as yourself.
Of course , My problem is game ban for alleged forum misuse (I will always say alleged and say anything that SOE want to hear concerning my DEAD accounts) only SOE has this rule and as I have pointed out in my blog – http://www.myspace.com/lovethatjestor – it was a new policy slipped into the EULA and TOS at the end of 2004 and quickly enforced on myself and a selected few.
After witnessing the self destruction of SWG with NGE , there is nothing to go back to. ( Althought the CU was not well received). The only thing I have gained is the fellowship of thousands done wrong by SOE.
There is no one correct metric for MMO’s. The right metric to use depends on the question you’re trying to answer.
If you want to know who’s making the most money, you don’t care about multiple account holders, you don’t care about people who forgot to cancel, and you don’t care about dormant logins. You do care about the number of paid subscriptions in each price bracket. And if you know the cost per player, you may want to know about peak and average logins.
If you want to know who’s got the biggest crowd on the weekend, you don’t care about free vs. paid subscriptions, you don’t care about price brackets or Station Access prorating or any of that. You want peak and average concurrent logins.
If you want to know who’s doing the best marketing, you don’t care about logins and you don’t care about legacy subscribers, but you do care about new subscribers per month and boxes sold for games that sell boxes.
Having said all this, my suspicion is that once a game “draws a curve” on the SirBruce chart using one metric, it would probably draw a similar curve using any other metric you succeeded in determining.
This is all flimflammery by the nutcase Koster who is largely responsible for the debacle that is Star Wars: Galaxies. As a business leader, the measure for MMORPG’s is called Customer Satisfaction, and it is measured as follows:
Total Number of Players Logged In Daily (divided by) Total Time Logged in Daily.
That is your Customer Satisfaction Index (CSI) and includes a volume element balanced with a time online (satisfaction) element. I’m sure that the CSI measure for Star Wars: Galaxies has dropped through the floor over the last year, and at other points of botched management over the last 3 years.
You can balance the CSI satisfaction measure with trending and comparing between games, Average Players Online Daily, and Average Time Online Daily.
Leaving aside your inflammatory first paragraph, I do agree that a satisfaction index is useful. However, you cannot compare total time logged in across different games. They simply have different “time it takes to get things done” scales.
A web-based turn-based game might have a very high satisfaction index, but not permit you to make more than one turn a day, and thus show a very low “hours per day” time. A Live Arcade game may make your thumbs ache and force you to stop after an hour. A chat-heavy game may encourage people to stay online for hours chatting. A large-group game like Lineage or raiding may require hours of online coordination time. For that matter, it may require hours of offline coordination time that don’t show up in your metric (nobody has good metrics for “hours spent on the hobby & fan community” which would be vital to considering a CSI).
This is without even getting to the issues of bots, of course.
Interesting post Raph. I would think the metric is a lot easier from a revenue perspective. Like a previous posted had mentioned, in the US, Sarbane-Oxley should be providing the guidlines for what a customer is.
When you try to add in the other requirements of running an MMO, like resource planning and customer satisfaction, then your calculations require a lot more work.
I am curious about the bundled service offering that SOE has. I think this kind of bundling skews the true state of a subscriber base for a given title and makes it all but impossible to estimate the acutal number of subscribers for a a given title included in the bundle. I am wondering what you think of this practice and do you see other companies becomming gaming portals or aggregators?
Thanks,
Joe
.ps Where have you landed?
This depends a lot on the max. number of characters per account allowed. While SWG used to allow 1 char per server, WoW allows 10 (? – at least more than most people are able to manage) on one server. I knew a lot of people who had multiple accounts on SWG – I don’t know anyone at WoW who does.
EQ allowed 8 per server, and still had extremely high incidence of multiple accounts. I have certainly heard plenty of stories about multiple WoW accounts as well.
Multiple accounts isn’t driven just by alt-play, it’s also driven by simultaneous play. The games that center more on farming and online presence exhibit it more, is my impression.
[…] Raph Koster wrote an interesting post talking about the numbers on MMOGChart.com. He points out how different companies will use different metrics, the weaknesses of the different metrics, and what he thinks would be the ideal metric (unique connections per week). […]
I find it ironic how Raph singles out Blizzard and Korean figures for possibly deceptive metrics, but doesn’t mention SOE’s same practice while he was still on board. I remember SOE playing the exact same games with numbers for SWG. In fact, Raph specifically criticizes the use of “registered users”, but here is a press release from 2003: “Star Wars Galaxies now has a total of 275,000 registered users. Released at the end of June, Galaxies ranks as the second largest online game in the US, in terms of registered users”
If registered users is such a poor metric, why did you use it Raph? If you are going to single out other companies for using misleading metrics to boost their numbers, at least have the courage to admit you have done it yourself.
Joe, it’s not like I usually write press releases. But yeah, everyone plays those games.
That said, registered users is also not too bad a metric for assessing the velocity of a launch. It basically means number of trials. It’s a poor metric for ongoing measurement, though.
“Joe, it’s not like I usually write press releases.”
You’re right. You didn’t write that particular press release.
But you did say this in an interview on Oct 1, 2003: “We’ve surpassed 300K registered accounts and we’re pretty happy with the progress.”
Source: http://pc.ign.com/articles/452/452616p1.html
IMO, it all comes down to this:
When you buy a single-player RPG, it doesn’t really matter how many other people are playing it since that fact will not significantly impact my personal enjoyment of the game.
When you buy a multiplayer RPG, then it DOES really matter how many other people are playing it. If the game is slowly dying, then is it worth $15/month for me to build up a character that may not be around 6 months from now? Will I even have enough other people to play with since most MMORPGs require some level of grouping? Will I have enough of a market to sell player-created goods?
Unlike with single-player RPGs, the health of the game community is a vital statistic to someone deciding to purchase and spend HUNDREDS of hours on a character in a muliplayer game. Therefore, it really boils my blood when MMORPG companies and executives (such as yourself) do everything they can to hide the true facts from prospective buyers.
Bluntly, Joe, I give out the figures I’m allowed to give out. Would I use registered users as a preferred metric? No, as I said in the article. But I am not allowed to give out whatever figure I want — it’s confidential information. Before an interview, I sit down the marketing and we prep on what stats are releasable. That’s how the business world works.
FWIW, SWG was growing at a nice clip through most of 2003. There was definitely a dropoff during that year, but cities, vehicles, and mounts, plus some content starting to flow in, made a difference. I can’t tell you how many users it had, because that too is confidential.
Personally, I’d be quite comfortable with all stats for all games being public, but I don’t think that most companies would agree.
[…] Register Measuring MMOs https://www.raphkoster.com/2006/06/01/measuring-mmos/ With the release of the latest edition of SirBruce’s chart of MMO populations, the debates have once again erupted over not only the validity of the numbers, but also over the how to even talk about the populations of these virtual worlds. It’s a discussion that has come up before, of course. Submitted by prognosticator on Jun 02, 2006 11:25:02 CST (more) (comments) karma: 0 / clicks: 75 / comments: 0 To post comments, please login. […]
[…] by randal2k on 6/02/06 [comment buried, show commenthide comment] + 2 diggs take frequent crashes, 100% CPU usage, problems with Video cards (Nvidia and ATI), random disconnects, cheating and “Warden” and you have the biggest MMORPG with the MOST problems. I have been with WOW for over a year and have ended my subscription. However, not before they charged me for a month after removing my account, they said “no refunds”. There warden program causes my CPU to rage 100% during the game, lending me to believe that IT IS what is causing lag and so forth in the Game. Warden is also uninhibited in what information is sent back to blizzard… anything they want. With all of this said, i am shocked that so many people play it. then i realized… there using the same tactic as AOL. AOL, said if you signed up for a free trial your a subscriber, and they would leave you as a subscriber for awhile after you didn’t even use it.. this cooking there numbers. I don’t have an account anymore, but, i do have 14 days left on the account till it’s nonfunctional. this is considered as me having an account… then take the “Free” accounts.. and you get more players… also, do they count people who played, and now don’t since they never delete accounts? No, i think that this is just more nonsense, and not accurate numbers. this says allot about what there looking at and how it’s not 100% accurate https://www.raphkoster.com/2006/06/01/measuring-mmos/ even if it is, i still don’t get it… something that buggy … says allot for continual beta programs. [reply] […]
[…] Speaking of games that keep on trucking and show growth when you don’t expect it, Puzzle Pirates has announced 2m registered users. Arr, Daniel, ye knows that registered users be a landlubber’s metric! Still, congratulations are in order, as it’s a significant achievement, and speaks well for the business model that Three Rings has adopted with micropayment doubloons. […]
[…] Speaking of games that keep on trucking and show growth when you don’t expect it, Puzzle Pirates has announced 2m registered users. Arr, Daniel, ye knows that registered users be a landlubber’s metric! Still, congratulations are in order, as it’s a significant achievement, and speaks well for the business model that Three Rings has adopted with micropayment doubloons. […]
[…] charliesangel – Glad to have you join us. You know what was surprising to me … is that SirBruce’s latest Subscription Numbers just came out … and Ultima Online has had as many or more subscribers as it had when I was playing back in 1998. (WOW is, of course, kicking the everloving crap out of everybody else.) Graphs here. Of course, Raph points out some of the misleading bits, but still… interesting data, nonetheless. __________________ I may have been married since 9/24/05, but I’ve been Janey since 1998. Woot! Baby Incoming 2/24/07 :: Blog […]
[…] I’ve mentioned before that weekly uniques is my preferred metricf or measuring virtual world populations. You need to have an SL account to see the stats, but the announcement is here. […]
[…] Life posts weekly uniques Second Life posts weekly uniques: “I’ve mentioned before that weekly uniques is my preferred metric for measuringvirtual world populations. You need to have an SL account to see the stats, but the announcement is here. […]
[…] For those that don’t know Raph Koster, he was UO’s lead designer, creative director for SWG, and chief creative officer for EQ2. This guy has been in the MMO business a long time and knows a thing or two. https://www.raphkoster.com/2006/06/01/measuring-mmos/ […]
[…] So why the controversy? Critics contend that journalists don’t understand the difference between “registered accounts” and “active users,” and that execs at Linden Lab haven’t exactly gone out of their way to clear up the confusion. Because Linden Lab prominently reports more than 2 million residents, many articles have said that Second Life has more than 2 million users. Ondrejka argues that this is still accurate. “The total residents, which we have always talked about, is the right number to represent users,” Ondrejka said. He pointed out that Linden Lab also prominently reports the “trailing” number of residents who logged in within the last 60 days, which as of Wednesday was 844,310. That means, of course, that at best, fully 63 percent of registered accounts have not logged in within 60 days. There’s another issue with that figure: the game industry typically reports users logged in within the last 30 days, not 60. The 30-day number, as of Wednesday, was 534,738 for Second Life. That means 77 percent did not log in within 30 days. Ondrejka said that Second Life is different from traditional online games. He pointed to a recent study Linden Lab conducted over six months in which the company examined member usage. He said that 30 percent of users had gaps of one or two months between log-ins. And that’s why the company prefers to put the 60-day trailing number on the SecondLife.com front page. Ondrejka said it is nearly impossible to arrive at a true number of active users because of the vagaries of the credit card numbers and IP addresses employed by users, and the fact that users can have multiple accounts. “I’m open to any Internet service that has a solution to the Internet identity problem,” said Ondrejka. “We don’t know who’s at the other end of the keyboard.” Still, because Second Life users frequently return after long times away, he said Linden Lab sticks to its “resident” definition. Other virtual-world and online-game veterans acknowledge that it’s very hard to figure out how many real users there are. “With the rise of non-subscription services, the industry is having to adapt to different metrics altogether,” Raph Koster, a designer of Ultima Online and former chief creative officer of EverQuest and Star Wars Galaxies publisher Sony Online Entertainment, wrote on his blog. “The original metric people used, of course, was subscriptions. This is a great metric for assessing how much revenue a game is getting, but it does have a few oddities in terms of determining the popularity of a game.” […]
[…] 2007/01/08 ���� 09:40:34 ���� �÷� ���� ��� ���������� ��������α���� ��? Daniel Terdiman2007.01.08 ���� ���� ����� �������� ��� ����� ��� 200������ �����ߴ�. 8�� �� ��ġ�� 100�������ٴ� �� �����ϸ� �̴� ��� ���� ��ġ��. ���� ����� ����� ������ ����� 7�ڸ��� �����ߴٴ� �Ϳ� �ǹ�� ����ϰ� �ִ�. �� ��� ������ ������ ��ΰŵ�� ���̸ӵ鵵 ���鿡 �ռ��� ����� �������� ��� �����ڼ� ���϶�� ���� ��(Linden Lab)� �й��ϱ� �����ߴ�. ���� ����� �������� ����� ��ó�� �������� ���� ���� ������ ����� �ִ�. CNET Networks. […]
[…] SecondLife.com ù ������� ����ڼ� ��� ����� 60�Ϸ� �� �͵� �� �����̶�� �����. �µ巹ī�� �ſ�ī�� ��ȣ ������ �����ڰ� ����ϴ� IP �ּ�, ���� ����ڰ� ������ ���� ���� �� �ִٴ� ��� ������ ��� Ȱ�� ����ڼ� ����ϴ� ��� ���� �Ұ����ϴٰ� ���ٿ���.�״� �����ͳ� �ü�� ����� ���� �ش�� ���� �ִ� ���ͳ� ����� � ���̶� ȯ���Ѵ١��� ���츮�� Ű������ �ݴ��� ���� �ִ��� ���� �١��� ����ߴ�.�״� ���� ����� ������ ����ڵ��� ����� ��� �Ⱓ�� �� �Ŀ��� ���ƿ�� ��쵵 ���� ������ ���� ������ֹΡ���ǿ� ����ϴٰ� �����.�ٸ� ���� ���� �� �¶��� ���� ����鵵 ��� ����ڼ� �ľ��ϱ�� �ſ� ��ƴٴ� �� ����Ѵ�.��Ƽ���¶���(Ultima Online) �����̳����� �� ������Ʈ(EverQuest) �� ��Ÿ���� ������ ���߾�ü�� �Ҵ� �¶��� �������θ�Ʈ CCO(chief creative officer) ���� �ڽ���(Raph Koster)�� �ڽ��� ��α� �������� �ʿ� ��� ���� �����ϰ� �ֱ� ������ ��迡���� ���� �ٸ� ������� ��� �Բ� ������ �ʿ䰡 �ִ�. ���� ����� ����ؿ� ��� ���� �����̴�. �� ���� �ϳ��� ������ ��� ����� ����� �ø����� ���ϴ� ���� ������� �ش� ������ �α⸦ �����Ѵٴ� ��鿡���� ���� �ʴ� �κ��� �ִ١���� ���.�ڽ��ʹ� �ڽ��� ���� ����ϴ� ��� �������ְ� ��� ���(average weekly unique)
[…] over 6 months old. Here is a page discussing Measuring MMOs – it links to that guys page as well. https://www.raphkoster.com/2006/06/01/measuring-mmos/ __________________ Bow down before the one you serve… VQ2 character : RokX VQ+ character : […]
[…] registered etc.. For a WONDERFUL article on how many MMO’s fudge their numbers, read this: https://www.raphkoster.com/2006/06/01/measuring-mmos/ If you are reading mmogchart.com, note that he only measures NORTH AMERICAN subscriptions in his […]
[…] Raph has a great post about measuring player numbers in MMO games. […]
[…] News: Sirius XM merger? Second Life population more around 200k Raph Koster on Measuring MMOS […]
[…] and most of them are seriously flawed. The best discussion I’ve seen to date can be found here if you care. (We’ll talk more about Monsieur Koster in a future […]
[…] wrote at length about how to measure MMOs, if you recall — I quite agree that registrations is not a useful […]
[…] to an article on Raph Koster’s website, multi-account users in mmo’s are extremely prevalant. Assuming POTBS is […]
[…] Raph Koster weighs in on how to measure MMOs. Great stuff. Bruce […]
[…] stats; Uniques to userbase ratio; Puzzle Pirates; Puzzle Pirates ARPU from VG Summit 2007 […]
[…] Uniques to userbase ratio […]
[…] them, Raph Koster (main designer of both UO & SWG) wrote an excellent piece on his blog here:Click HereIts a good read, but to quote the part relevant to this debate on how bad using registered users […]
[…] people, more than the population of all of North America). So far, so good — after all, I have complained about this […]
[…] Club at Big Fish Games! Sponsored by: http://www.BigFishGames.com/ [Found on Ads by Google] 8. Raphs Website » Measuring MMOs Of course, subscriber metrics, despite these problems, are still a damn sight …. Raph has […]
[…] guru Raph Koster has posted an excellent write-up on his blog on how to measure MMOs. I wanted to provide a link to here because it provides a very comprehensive overview of both the […]
[…] no sensible way of answering the question literally. A couple of years ago, Raph Koster did an updated version of the explanation for this problem (it needs updating again by now to take account of how the industry has continued to evolve since […]
[…] (this is rather the opposite end of interpretation to “Over 1 billion people play online games” – and make sure you read Raph Koster’s thoughts before trying to interpret these figures) […]
[…] Uniques to userbase ratio […]