Lessons Learned, and gaming folks using web ways
Lessons Learned is a really good blog that gives a lot of insight into exactly all the sorts of things that I was referencing in that AGDC panel on how we have to adapt to web ways of doing things. Be sure to read through the archives.
I commented on that panel that the notion of doing things like A/B split testing is sort of foreign to game developers. But it’s something incredibly powerful when correctly applied. But it’s a little hard to conceive of saying “the combat rules are different from fight to fight.” The commonest form of game A/B testing is alternate rule shards, which is a huge investment and is far too large and game-adjusting an approach to really be used as a split testing factor.
It’d be worth pondering what sorts of split testing could be fruitfully brought into the game world, given how useful a tool it is. Of course, part of the barrier for game teams in the large-scale MMO world is that clients are updated via patches, not streaming, and there is no good mechanism typically for patching only half the clients, or some such. But imagine having taken the original SWG UI icons and the more colorful NGE revamp, and upgrading only half the users — and then seeing whether the control or the new design have better metrics.
There’s a level of commitment that we feel when making game changes that we should try to avoid. After all, successful game design in the early stages of prototyping and platytesting involves killing lots of sacred cows, often making big shifts in how things are played. But we tend to see patches as a case where “it’s a big deal if we have to revert.” Part of the web way is acknolwedging the frequency of mistaken hypotheses.
10 Responses to “Lessons Learned, and gaming folks using web ways”
Sorry, the comment form is closed at this time.
Dear Mr. Koster,
Please stop giving away my secrets before I even have a chance to launch my game. It’s bad enough that you wrote abouts lots of my clever ideas years ago, but this is going to far. Cease and desist immediately or I shall sic my lawyers on you for these egregious acts of chronometric piracy.
Respectfully,
Peter Harkins
“Clients are updated via patches, not streaming” … not true of all games. Guild Wars does streaming updates in the background while you’re playing; it’s not unknown for them to release several patches in one day. From the player’s point of view all we see is an occasional “a new client is available, please restart the game” message in our chat windows. You don’t have to restart immediately, you can just keep playing on the old client until it’s convenient to restart. (Although on one memorable occasion this led to an economic mini-meltdown, when some clever bugger found a way to make craploads of gold by arbitrage between players running the old and new game mechanics.)
As far as I know they don’t do split testing (at least not on purpose), although they do often try out changes for a week or so before deciding whether to keep or revert them.
Yeah, the GW team has been very good at the whole “test it and revert it if it doesn’t work out.”
It’s honestly one of the most impressive things about their balancing efforts. Heck, about a month ago they reverted a complete reworking of a skill because their metrics showed it was being used less often than before. Granted, a single skill doesn’t have as huge of an impact on the game as say, the entire combat system, but it works really well with the small granular stuff like that.
Some games have public test servers which is a way to do split testing. Scrubbing production database and using them in stage and dev environments is always better then generating data from scratch. Client recorders can be use to generate data to test with if you do not want to us your customers. Rollback, server deploys, client updates are all features and if one forgets to built it or take shortcuts it does effect the cost and quality of the service that is being provided. Games are a great place to do automation but anything but beyond button pushers in the game industry is rare. No need to write a strat guide just export the existing game data into another format.
Gaming folks need to get current with how technology is being used and built. How something is built is as important as what is build since it cannot be static for more then a few days, weeks, or month before being out of date and replaced with something new and better. Much of the web way is a active and continueous feedback loop in hours or a day. If one cannot see something make a change and everyone see the change in a few hours or a day the process is to slow.
I’m waiting for users to be able to access in-game journals written by players via RSS feed. I’m waiting for Web Services to be taken seriously. Why isn’t the auction house offered as a Web Service outside of the game for anyone to download and use the data? In all, the webifying of MMOs is really applying more openness to the platform which is something the web accomplished years ago.
Web Services.
Web Browsers as a legitimate alternate gameplay client where the actions on the web immediately become available in game and vice versa.
RSS Feeds.
Just setting up RSS feeds and Web Services will put IIS/Apache logs in the pathway for analysis by any number of tools.
Here’s a radical thought. Use IIS 7 on Server 2008 as the MMO engine for everything but necessary functionality to run the 3D simulation (combat and collision detection). Basically, run anything that can tollerate lag in the transactionby building it on a web server. Accept a quest, there’s an ASP.Net MVC application that has a view embedded within the game client. Need to view the AH data; fine there’s an MVC object framework that has viewers for RSS, Web Services and the in-game AH dialog.
Usually most people bring up scalability as the reason why not to but I’m not buying it with today’s hardware and infrastructure. Web sites handle larger volumes of traffic on more standardized server software every day. If you have to, overide the IIS7 pipeline for serving a document by creating your own document type and writing the entire guts yourself. You still get the benefit of all sorts of built-in functionality like load balancing, health monitoring and statistics logging.
That’s what I think about when I think about Webifying MMOs.
The problem with reverting patches, is that to the casual uninformed, it appears as if you’re admitting error and having wasted their time for X period of days the patch was in place. Granted, that’s probably better than having shoddy product, but there may be a better option.
On PTS, instead of making changes, fixing and then committing to live, consider instead making changes, gathering feedback, reverting them on PTS, making different changes, gather feedback, and compare. It’s like sharding, but with the cost applied to time, rather than system overhead, and allows you to utilize a far smaller population.
What Guild Wars is reportedly (in this thread) doing is arguably better than split testing, since it pays in the player-trust currency. Part of the point of split testing is that it’s seamless and invisible to the end user and results in improvements without talking about it. Maybe that’s not a good thing, sometimes.
@Michael Chui,
I can echo that sentiment to a degree, as another thing that’s really top notch about how Arena Net goes about things is that they tend to tell you *exactly* why they’re making a change to a specific skill and what the changes are supposed to bring about on the official wiki in advance of the changes going permanently live. And when they revert a change, they tell you why too. There is a high level of transparency in the process that’s extremely refreshing.
Wow, thanks for the link and the kind words. I’m honored.
I’d like to underscore the importance of transparency. I don’t have much experience in Guild Wars, but I do play City of Heroes / Villains from time to time, and out of all the big MMOs they probably have one of the best relationships between the “Devs” and the players. This is founded largely on transparency and open dialogue, and is really something else.
Granted, it took a (long) while to get to the point where one of the Devs could say “we’re considering changing this, for this reason” and not have reams of knee-jerk “OMG NERF!!!” doom-posts. But, their cultivation of an open, two-way atmosphere has paid off.