The Art of LiveOps: Raph Koster

This podcast was released on August 8th, 2019.

We are excited to announce the launch of a new podcast that we are hosting, The Art of LiveOps. In each bi-weekly episode, we will be interviewing game creators working at the cutting edge of LiveOps to share valuable lessons, perspectives, and yes, disaster stories of LiveOps gone wrong.

We’ve created this podcast because we believe effective LiveOps is critical to the success of modern games, but there’s simply too little knowledge and information out there about how to do it right. We all know we should be engaging our players on a continual basis, to better grow our game communities, but what is the best way? What techniques work and don’t work? What changes do you need to make to culture and process? Where should you start?

In our first episode, we talk with Raph Koster, a legendary game designer who has worked in online games since 1993 and was the lead designer behind Ultima Online as well as the creative director for Star Wars Galaxies. Raph was there at the birth of modern LiveOps in early MUDs and MMORPG games, so what better guest for our first episode?

— https://developer.microsoft.com/en-us/games/blog/introducing-the-art-of-liveops-podcast

Audio

Transcript

Hello, I’m James Gwertzman. I founded PlayFab five years ago because I saw a huge gap in the kinds of access to live ops technologies game studios needed to be successful.

I’m Crystin Cox. I’ve been making games for 16 years as a designer and a game director focused on LiveOps.

James: We’ve put this podcast together because a lot of the information on how to do LiveOps effectively just isn’t out there. There’s way more information about how to optimize your graphics pipeline or how to put together effective game design than there is on how to do effective live ops

Crystin: And since it’s pretty tough to go around yourself and just find everyone that does live ops and ask them questions. We did it for you.

James: We’ve picked together a list of some of the top practitioners in the industry, folks who we think are really pushing the boundaries and doing some cutting edge work.

Crystin: And we interview a different one every week and ask them about their experiences running live games, doing live ops and even having live ops disasters.

James: Yeah, train wrecks are the best. That’s where you really learn how to do things effectively.

Crystin: So tune in. We have another interview for you today and be sure to stay tuned after we’re finished with the interview. If you want to learn more about live ops academy and PlayFab.

James: On with the show.

Crystin: Hi James. How’s it going?

James: Hey Crystin, good. So who do we have today?

Crystin: Today we are talking to Raph Koster.

James: Raph Koster… huh… I’m just kidding. Everyone knows who Raph Koster is.

Crystin: Yeah, no, it’s, this one’s very exciting. Raph has, you know, been sort of a fixture in the live game space for almost, you know, 25 years. I don’t know, maybe he’d be like, don’t make me sound so old!

James: We were lucky enough to corner him here at GDC.

Crystin: Yup.

James: Where he’s speaking as he has at probably every GDC going back as far as I can remember.

Crystin: Right.

James: And we’re going to chat with him about, uh, about live ops in games and, and probably a whole lot more.

Crystin: Yeah. Especially, I think history and his, you know, he’s got a lot of cool insights on community and the way live ops has changed over time.

James: All right, well let’s, uh, let’s go right to it.

Crystin Cox: So thanks for joining us. Why don’t you, uh, introduce yourself?

Raph Koster: Sure. My name’s Raph Koster and I’m a game designer who has been working in online games since about 1993, um, in one way or another, um, ranging from the old text MUDs to massively multiplayer online RPGs like Ultima Online, Star Wars Galaxies to Facebook and social games and mobile games, and even multiplayer augmented reality.

Crystin: Nice. So you’ve been working in the industry while we’ve seen it change quite a lot, especially the online space. When you hear the term live ops, what does that mean to you?

Raph: I flashed back on a day in a hallway at Origin where Rich Vogel, who was a producer on Ultima Online and I, he stopped me in the hallway and said, hey, I think we need a name for the team that’s going to stay and keep running the game. And we kept batting around something, and finally we said, why don’t we call it the live team.

Crystin: Right.

Raph: And so I flashed back on that!

Crystin: Yes, and so now there have been now dozens of live teams on MMOs ever since.

Raph: Well there have been live teams long before that, right? I think people tend to forget that online games, you know, the earliest, the earliest computer games were multiplayer, not single player because we couldn’t do AI. And networking came along really early. Right. So even though it wasn’t available in homes, online games were around throughout the 70s on networks like PLATO, then the first text MUD and shortly thereafter the birth of the commercial online game industry. That was 1978. So there’s really been live teams since then, although we may not have called them that.

Crystin: I know for me personally, actually, when I look back on that era, the MUD era, I actually think there’s some things that we were doing then as an industry that were really good as far as social structures and social design, that we sort of lost along the way?

Raph: I could not agree more. Um, yeah, I could certainly tell stories. I don’t know which ones you want to hear, but yes.

Crystin: Well, I mean I think that everyone likes to hear mostly the disaster stories as we’ve found.

Raph: Oh, disaster stories!

Crystin: I was asking people for a live ops disaster stories…

James: But origin stories are good too!

Raph: Yeah. I think, um, you know, early on in online games, the line between the team that kept the game running and the people who played was really, really thin and that was an intentional design choice, right? In the early games, uh, they were running on code written by students mostly, on university computers and therefore they were volunteer run, which meant if you got good at the game or advanced really high and you happen to know how to delete files from a file system, or reboot an executable, great, you could join the team! Right?

That meant that an awful lot of things that we would consider about professionalism weren’t a thing. So I still vividly remember the first person I ever fired from an online game was actually an admin on a mud that I was working on who thought it was a perk of the job — and this was a common view — a perk of the job to invisibly lurk in the game, hovering over people invisibly and watching them do things including, you know, personal things. Um, and uh, you know, just basically invade people’s privacy.

And when I went to him and said, what are you doing? He said, “it’s a perk of the job. Why else would we have the power to be invisible?” And back then we had tools that actually, some of which I miss today. One of them was a command called “snoop” — terrible name. But what it meant was have your view of the game exactly match that of another player.

Crystin: Right.

Raph: It literally directly cloned their network stream so that you could see exactly what was happening to them, which is a phenomenally useful tool if you’re trying to debug a problem. But oh my gosh, is it invasive!

Crystin: Yes!

Raph: …because you could see their private messages. You can, it’s everything. So, so yeah, that was the first guy I fired. And those sorts of things gradually led particularly over on the commercial side, but even on the text mods that gradually led to, um, the notion that we really needed to develop codes of conduct, not just for players — which were common — but codes of conduct for admins.

Raph: And that was sort of a radical thought. Um, I mean early admins were called Gods and Wizards. Right? Right. So they had… you know, one of the most famous admin manuals was about how to turn players into toads!

James: I think that we’ve lost a lot as we’ve, as we’ve gone along. Because one of the themes that we’ve been developing over the interviews we’ve been doing this week alone really has been the focus in live ops and how that has shifted game design around the player and how, you know, the, the player’s really re-emerging now as the center of the design. Not necessarily a designers grand vision, but how do you design for that continuous engagement. And social structures I think are really key to that continual engagement. And, and you know, now the peaks seems to be lots of players running around shooting each other, but it seems like there’s a lot of those more richer interactions that maybe are coming back.

Raph: I think so. What we ended up back then, uh, in order to deal with those sorts of issues, what we ended up doing was taking the admin staff and splitting it in four. Which today, I think this will probably sound more familiar, but we split it into sort of the team of programmers who today we might call either Live or DevOps. Right? Um, so these folks were, we had their home phone numbers so he could call them if the mud went down, you know, they are the ones who propped changes, right? And all of that. And so that’s a team that more or less exist today.

They were also though the game’s dev team for code, but, um, the games back then were arguably more data driven than they are now!

Crystin: Right, right.

Raph: Which was another thing that I think is coming back. And, um, and that meant they weren’t necessarily needed all the time. Features didn’t actually come along that often, so they weren’t continuously pushing that sort of changes. It was mostly bug fixes.

Then a second group was Admin, and Admin actually — the cops, right? Um, and we intentionally split that role from community

Crystin: Right?

Raph: Which actually we call PR back then, um, player relations, public relations.

Crystin: That makes sense.

Raph: And so Admin were the ones who dealt with problems, who provided assistance as well, right? Whereas Player Relations was about cruise directing. It was about, you know, how do we engage, how do we engage the users? So super common in those days, to do things like, “hey everybody, it’s Thursday night, 8:00 PM, let’s do a trivia contest on live channels.” Right? Or Hey everybody, we’re going to play tag. You know, there were, you could play tag with like these magic wands and stuff. Um, and so on. Right? Uh, they’d build custom areas that could be tag arenas and things like that.

Um, and then the last was content creators or Builders, right? And those folks were the ones who are constantly adding to and updating the game.

Crystin: Right.

So, uh, segmented admin powers. So that you know, different people have different permissions and all of that sort of thing. And I think a lot of those practices, and particularly I think of the sharp division between the Admin and the player relations was particularly powerful and something that isn’t always as clear cut today but I think is a best practice for player engagement.

Crystin: I definitely agree. We had a really similar split on MapleStory and it was very helpful not just in managing the way people interacted with the game, but also managing the perception of the player base. And you know, I think they were more comfortable knowing that there was some kind of balance of power that there was someone that could appeal to. Who was kind of “on their side.” And we never asked the community team to come down hard, you know, that was separated out for a while. GMs handle that and that’s something that they’re going to do. I think that actually did really help.

So you also worked in the social space specifically and in social games. How did you see the earlier sort of mud and MMOs impact how we approached that sort of new platform?

Raph: Not nearly enough is basically the bottom line.

I look back on the social gaming boom as possibly the greatest missed opportunity in video game history. Um, there we had the single largest connected platform of people in history, that we ever, ever had. The potential to make, back then, the first million player concurrent game, you know, that kind of thing. And we didn’t [inaudible] just didn’t even, we didn’t even take a serious try. It is the way I’d put it. Um, so we ended up making games that were not actually that social.

Of course, I think it’s fair to say that we eventually discovered the networks themselves are not necessarily that social in some of their intrinsic design elements. I think it turned out there was a lot for everybody to learn.

Crystin: Right.

Raph: But, um, yeah, so I see it as a, as a huge missed opportunity. I think a lot of the lessons around… oh, so many lessons, um, around emerging content about building live community, like very few social games actually supported players talking to one another…

Crystin: Which seemed crazy at the time! Like for me, I was, when the boom was just starting, I was working on MapleStory and I’d come from working on like Toontown Online. So mostly with kids, but those games are all so focused, or I mean they were practically just 3d chat rooms and they were really focused on like one on one chatting and physically seeing each other. It was kind of strange to see social games lag so far behind getting just chatting.

Raph: Yeah. And that’s… I mean, fundamentally it’s because their architecture was completely web based and so… Basically, you know, a Farmville client or one of these other things is really — even though it was in Flash — it was really just a display of static data in a database.

Crystin: Right.

Raph: So there wasn’t really a game server there. Uh, so there’s no way for people to talk to one another. There was nothing in the way, or nothing in between. Literally you played your game, you just updated the database directly and got back results. And just when we updated a row or column it, you know, it happened to look like a cabbage or whatever.

Crystin: Right.

Raph: Uh, we actually, I was doing social games at a studio of mine called MetaPlace, which was acquired by Playdom and Disney. Metaplace prior to that had been a web based virtual world engine. So our farming games, actually, you could log in, see other people, chat with them, hang out. And we had things like professors running classes and in the vineyard farming game we had wine tasting parties and you know, all of these social things and it dramatically increased retention, because community is the number one predictor of retention.

Crystin: And that has been something I’ve noticed, very interesting, especially having spent a lot of my years more focused on Asia. That we were really interested in log in retention numbers, how long the relationship was with players. And it seems like it took the western free to play mobile and social communities a little longer to get to that point. They were more interested in acquisition was really, really a big key KPI. And Monetization was a big, uh, sort of leading KPI. Why do you think that took longer for us to sort of come around on?

Raph: Yeah, I think that’s a complex story and a lot of it is actually tied in with metrics, right? So you know, there’s the whole, you become, what you measure is, is sort of an a classic lesson. Um, there were early metric systems, even in the text muds, like I even have a slide, I pull up some times and talks showing graphs of frequency of kills and game balance. You know, “who managed to take out this monster, is it under or overpowered?” for LegendMUD that dates back to 1994. Right? A graph, by the way, that far too few modern games could pull up that to be honest!

Crystin: Which is funny because game designers love numbers. Like how do you stop us from looking at that stuff.

Raph: Yeah. And then the answer is it depends on what you’re choosing to log and what you choose to dashboard.

So MMOs, because of their business model, it was a monthly recurring subscription; that meant you lived and died on your churn rate. Right? Um, it was all about retention. And even once the games went to free to play, which not all of them did, but most of them did. Right. It was still about retention because there was a very clear sense of “the revenue is the area under the curve.”

Crystin: Right. Right.

Raph: And so the longer we make the curve the better. And that was just a, it was just a mentality.

Crystin: Yeah.

Raph: We paid attention to activity numbers, but not closely.

Crystin: Right.

Raph: Right? So… I mean we would have like a brag wall with an LED display or something, like how many people who are online, but we weren’t obsessively, like, we weren’t dashboarding that at five minute intervals or anything because it just wasn’t relevant.

We mostly wanted to know are we hitting our 10% of the user base logged on at peak concurrent time kinds of questions.

Crystin: Right. Yeah.

Raph: Because those would be leading indicators, but you only need to measure that basically once a day.

Crystin: Right.

Raph: The rest of the time it’s not as critical. It’s not as crucial. Um, so our real time monitoring tended to be for emergency things. Right? Right. “Oh, something’s crashed.”

Crystin: Yeah.

Raph: In social games, the mentality was very different. And the realization that I eventually came to after talking to a bunch of the analysts was that a big part of it was how the database queries and how the databases were even literally structured. You know, depending on how you arrange a database, a query can either be really cheap or really expensive.

Crystin: Right.

Raph: And sort of, you know, think querying an Excel sheet along a row or a column.

Crystin: Right.

Raph: And basically the analogy I tend to use is that most of that industry was really, really good at building dashboards that were the equivalent of a traffic camera sitting above an intersection. At any given moment, they could tell you how many cars are in their intersection, how many people are crossing the street, how many red cars are there.

Crystin: Right.

Raph: They could tell you everything you wanted to know about that instant.

Crystin: Right.

Raph: And they had no idea where the car came from, where it’s going, how long it had been there.

Crystin: Right, right. No clue. It was all snapshotting!

Raph: Which made for wonderful, cool looking dashboards. Sure. But if you started trying to ask a question, hey, I’d like to run a query on how, uh, “how many of this cohort who are regular chatters with players, how much is that impacting a, their lifetime in the game and their lifetime value versus players who don’t have friends?”

I got back, “Wow, we’ll get back to you in like three weeks?”

Crystin: Right, right.

Raph: And on an MMO, we could run that analysis relatively easily. It’s like, “oh, well let’s pull up their sheet. How many friends do they have? You know, ta daa, we’re done. It was super easy.

Crystin: Yeah.

Raph: So that was a lot of it. It was, you know, you become what you measure to an even deeper degree, and then it seems your tools end up shaping your practices.

James: What I’ve also noticed a lot of the early mobile social games when you know, there is a, they were hiring a lot of folks that didn’t have to have game backgrounds per se, but more kind of analyst systems backgrounds, right? So they would take a problem like retention and say, well, what are we measuring? Okay, day one, our attention our day, setting our attention. Okay, let’s put in systems that reward you for logging every day and treat it with this insurance bond, little escalating jackpot type type mechanic as opposed to an underlying social mechanic, is way harder to measure, way harder to design, but maybe it is ultimately more, more sticky.

Raph: Yeah. We actually, in the MMO business, there was no such thing as day one retention. We actually used the term “conversion” to discuss pretty much everything in the entire first month because player’s got a free month and so on. I mean that conditions everything you do, you’re already thinking one day who cares about one day? If I don’t make it to day 32 right, I’ve wasted $15 and you know that, I mean that was our, that was what we were doing, right?

So for us, it was all about building an emotional connection with the player in the span of 30 days in order to convert them to a paying customer. And during the social– and even actually, even today, you know, when I’ve worked with mobile clients– I’ve urged them to think similarly and say, look, this is an ongoing relationship.

So, you know, in the first however long, you’re dating, right? And there is no not yet a “conversion” to going steady. There’s no “put a ring on it” yet. Right? That’s not what’s happened yet. And so you have to actually get the consumer to convert to a customer rather than basically a trial. Right. And that’s an emotional journey on the part of the player, right? Not a, yeah, not a day. One thing. Very rare. You know, that that would be love at first sight. It happens, but we tell fairytales about it.

Crystin: It does happen… Yeah. I mean, I think it’s also challenging because it is much easier. And if you’re not someone who’s done a lot of sort of, uh, you’ve not spent a lot of cycles on learning to do really deeply engaging game design, it’s very simple to — and I still hear this all the time from game teams — where they’re like, “well, I want people to log in so I’m going to reward them. I’m just going to reward them for doing the thing that I want them to do.” And that is so, so quickly doesn’t work anymore. Right? We, we’re humans, just our baselines get reset all the time. Like we don’t really that motivated by extrinsic value. Um, how do we deal with that in a world where — I mean, cause I mean sometimes I like to look back at, at building MMOs 10, 15 years ago, and I’m sort of nostalgic, because it felt like we had more time. It felt like we, we had more opportunity to build these deeper relationships with players. How do we deal with it now? There are, there are thousands of games and you do need to make an impression quickly, but you don’t want to just be — what I call it, paying players to play your game.

Raph: Yeah. You know, there are the techniques that Walmart uses that, uh, you know, a lot of people make fun of, but they pay people to say hello. Right. And, and it turns out that the bottom line impact of having a human greeter in that, in that service role is massive double digit returns on, on effectively conversion to being a Walmart shopper. And, uh, we saw the same in online games, uh, that, that having people, uh, you know, at first it was the player relations folks would show up and say, “hi, welcome.” You know, “what, are you confused or do you need any help?”

Or, you know, now obviously that doesn’t scale very well today. Um, it didn’t scale back then!

Crystin: Yeah.

Raph: So, um, on the very first mud I played, uh, I was asked by one of the admins there who, uh, has also ended up in the video game industry. Uh, he ended up as working on, uh, designing on Meridian 59, Shadowbane, Star Wars: The Old Republic, a fellow named Damion Schubert.

Um, he said, hey, we need a, you know, I’ve been playing for a while. He said, we need a, the guild leader for one of the guilds in our game. Back then, guilds were hardcoded players did not create them. And it was called the Golden Pyramid. I don’t remember why, but the guild’s charter was newbie helping. Right? And the, the deal was, “hey, you guys are a guild just like any other guild. But when a new player shows up, the admins are going to tell you and tell you their name and you can go find them and offered to help.” Right?

And so our entire group was sort of like a Lion’s Club or something. Right? And, and people joined the guild who enjoyed doing that later on, much later when we were doing, um, Star Wars Galaxies, there was simply a command you could type in, /helper, and it popped up a flag over your head.

And the people who chose to do it, we’re advertising players could look around, see, oh, that person, somebody who’s volunteered, right. To help. And you know what, it made a massive difference, right, to the community. And that scaled because it was free.

Crystin: Yes.

Raph: And we had actually almost no issues, right. There was, it was completely voluntary as players who enjoyed doing it because they liked often gilts would do it because they needed to recruit members. Right? So it was often to their benefit. It was such a simple technique.

Cyrstin: Right. I think that speaks to something too that I’ve seen a lot working on Guild Wars 2 where we were really adamant that we wanted our community to be friendly. And I think people were always surprised when we talked to them about it is we can say actually we have very few problems. Basically we set expectations. We built the game around the idea of you’re always happy to see another player. We tell people to be friendly and then they mostly are, I mean this not that hard. I mean, I don’t want to downplay it because it can be very challenging to deal with toxic behavior online, but I think we sometimes shy away from just setting the expectation of offering people the opportunity to be helped.

Raph: The most important thing you can do in return to reduce toxic behavior is expectation setting. Behavior gets the most toxic when no expectations are set of any sort and people come in with a mentality of anything goes. The more you can kind of provide a framework for people to operate in, the more likely they are to behave. There’s peer pressure to do it.

There’s, you know, social standing reasons. There’s lots of reasons why. One of the results out of psychology for a long time now — and we probably didn’t need the results from psychology because it’s, it’s such a common practice, It’s actually enshrined in law, in most cultures, right? — when you are about to take on something important, where you’re making a commitment, say to tell the truth or to deliver something or whatever, you have to take an affirmative stance, sign your name or state and oath, right? This is a piece of human culture…

Crystin: …going back thousands of years.

Raph: We have no idea where it started, but we know it works, like it works! Well. Guess what? You can very easily ask people to make an affirmative statement, and I don’t … — clicking through terms of service is not it!

Crystin: Yeah, right.

Raph: But it was…

James: EULAs don’t fit that.

Raph: Right, EULAs don’t fit that. But if you, when you logged in, a message, a stark message, popped up on your screen that said, “Hey, this is a friendly community, we’re not going to bury you in rules… but be nice to one another. Be excellent to one another! [“deedle deedle” air guitar sounds]

Crystin: Right. Right.

Raph: And then when you click it signs your name.

Crystin: Right.

Raph: Yeah. That’s been shown to noticeably reduce the incidents of bad behavior.

Crystin: Yeah, that’s cool.

Raph: It’s human psychology.

Crystin: Yeah. Switching gears a little bit, you’ve been doing some form of live ops for most of your career, but there are a lot of people in the industry today who have spent a decade or more in the industry and not ever have been touched LiveOps, but they’re starting to have to dabble in it or go full on indeed. Do you have advice for them?

Raph: Patience. Um, my experience is that an organization learning to run a game as a service is typically a three to five year long learning curve. It is not something you can just flip a switch. Um, there are temperament issues around it demands, um, marathons, not sprints, and all these other cliches.

Uh, it demands a certain level of emotional investment in gardening and tending a user base, which is a very different mentality from making cool stuff, and making new stuff in particular, so it isn’t actually suited for everybody.

Some people prefer making lots of new stuff and constantly doing something different. This involves the, that sort of slow effort of “okay, it’s going to take 20 years to shape this into a proper English garden. Let’s start.” Right?

And that’s a very different mindset and it can be very, very challenging because studio cultures are not necessarily aligned and you start getting developers who want to move off. You start getting, “when are we making another game?”

And if business pressures are, “when are making another game as a service,” then there’s no escape. Games as a service eats your studio, right? Like, you don’t end up with spare people afterwards. In fact, you end up at people deficit. And so that’s a, a structural thing about them.

Right. Um, so yeah, it’s, it’s a very, very challenging shift to make. And I think probably the biggest shift, and I don’t want to sound too forceful about it, but the biggest shift is that the customer, the player, the, your guest becomes the king in a very particular way. It doesn’t mean they’re always right. It doesn’t mean they’re perfect.

But the orientation is around serving.

Crystin: Yeah. Right.

Raph: I mean the word has a root, “to serve” is absolutely crucial and that is not the root of packaged goods, that’s not at the root of that experience. And that’s a difficult fundamental mindset shift for people.

Crystin: Yeah. As I’ve been talking to teams that are about to take the plunge, a couple of them asked me, “what’s the thing I don’t know, like I’ve heard a bunch of stuff, but what is it that will blindside me?” And my answer is always customer service. You’re not thinking enough about it. You don’t have enough people. And you probably haven’t trained them correctly because if you’ve come from packaged goods, what it means to do customer support is totally different. It’s not, “I had a problem, answer it.” It’s not a lot of form responses. It is a lot of individual personal experiences. It’s going to be a lot of volume and you have to have that mentality of whatever I need to do to take care of you.

Raph: Yeah. I and, and it cuts across, I mean, even if we just get super concrete on numbers, right? Even if you’re, you’re trying to get away with, you know, minimum wage frontline people, which I don’t recommend, but let’s say you’re trying to get away with that. You’re still in a position where you know, you’re wiping away, taking one call might erase your entire revenue from that user.

Crystin: Yeah.

Raph: So one problem that they had means they now got to play for free. And so it starts creating a very different mentality around, okay, what level of “bug free” can I handle? Right? Um, if I’m losing a month revenue from talking to this person, then I better get two months or three months lifetime out of the interaction.

Crystin: Right.

Raph: And that means the interaction can’t be “go away,” obviously. No. Um, yeah, I think, I think you’re absolutely right. There’s an enormous amount of things around customer service people need to learn. And one of the fastest ways that I’ve always seen to teach it is — make absolutely everybody in the org sit at a customer service desk, and have them do that periodically. Like, Hey, every quarter you gotta do your shift.

Crystin: Yeah. You can’t run away from the players.

Raph: That’s right.

James: I want to, I want us just to talk a little bit about, uh, the comment you made earlier about being cruise director. You know, that notion of, you know, a team is really thinking about programming the content, programming the events, um, what, especially in the world in mobile and social, where maybe the community structures are, the social structures are weaker than a classic MMO, what advice do you have for us as designers, thinking about your live events? And those sorts of games.

Raph: Mobile is particularly challenging. We’re walking around with, you know, the most powerful communications device ever invented. And there are virtually no channels for developer to talk to a player. It’s actually appalling how bad the infrastructure there is, no offense to all giant hardware companies. But, um, the fact that in the mobile world we primarily communicate our users via reviews and, and comments on reviews and maybe sometimes patch notes that aren’t even visible to the user. Most of the time is dreadful. Like actively dreadful. I, I honestly believe as big as mobile hits have been, they have probably left an additional digit’s worth of revenue on the floor because we can’t even talk to the users.

Uh, and that’s, I mean, some of that’s on the storefronts. It’s on the carriers, it’s on the operators. Some of it is on the developers as well, right? Because in theory, you can build those channels into your game, right? It’s additional expense. It’s hard, but it can be very, very worth it, right. As we’ve seen with those mobile games that then do end up doing periodic events and so on.

So, yeah, mobile, they’re actually, it’s a, it’s a bad place. It’s very hard. Um, so I would love to recommend that people, you know, change that practice, but I also recognize that it’s actually probably too big a hill to climb for quite a lot of studios because platform support for doing it is basically absent. Yeah. Tough, right?

Crystin: Yeah. I want to see if you can hear that. Let me see though. The uh, the lovely, that’s the noises of San Francisco. This is the ambiance of where this has happened to. We’re fine. Yeah. It’s not that big a deal. Um, so I think that is also something that’s interesting about the era that we’re in right now, which is that it used to be in a lot of ways, a lot more work to get a live game off the ground because you did have to build a lot of infrastructure. Now we are relying more and more on platform holders to support games. Do you think that that’s going to stay true? I mean, I think there’s some interesting things happening, you know, with, with Epic specifically in Fortnite and pulling away from the Google Play store and things like that. Do you think we actually will start to see, and this has been true in Asia for basically ever right, that there are these really well developed platforms specifically for games. Do you think we’ll see that over here?

Raph: I think so. I think, uh, you know, certainly, um, Steam for has an, I’ve actually seen the complaint from independent developers that one of the reasons they don’t want to be on Steam is because Steam forces them into doing a bunch of community work.

Crystin: Yeah.

Raph: Right. Suddenly, wait, I just wanted to sell my game and now I have to moderate forums.

Crystin: Yup.

Raph: Or my or my sales will suffer because my front page will look awful. Right. Things like that. So yes, I think we are going to see more awareness of that. Um, and I think there’s a lot infrastructure popping up all over the place. Um, not just storefronts, but also, you know, I mean, well, I mean there’s what Microsoft is doing, you know, PlayFab…

Crystin: …interesting that it comes back! Yeah.

Raph: But I mean, but you know, as you know, you’re not the only ones. I mean there’s, there’s a lot of stuff like that that’s out there now. So in a lot of ways I think the soft expertise is more important. But, uh, that said, I also think, and this is just my general advice to developers, is that they need to not be beholden to any given platform. Don’t hand your community over to a storefront. You know, you need to own your own community, particularly if you’re one of the smaller developers who, uh, moves from platform to platform when there’s opportunity, right. You know, it has to go look for the best development deal and so on. Right. There are a lot of reasons why a developer like that will not, should not want to have their community tied down.

So I think it’s important for them to think in terms of owning their relationship and not letting it get, um, disintermediated or whatever by somebody else.

One practice that we’ve lost over time, I think it’s actually a really long term, not a great idea to outsource your community to Twitter or Reddit. Um, because you, you know, let players have a community on Reddit, that’s fantastic. But if you don’t have a means of communication and a bridge to the users and the only one belongs to third parties, I mean, not only might Reddit go away, theoretically it can happen. We’ve seen that sort of thing happen. But also you can’t moderate Reddit. It, you can’t, you know, that’s the front page for your community then you don’t control it as a developer.

Crystin: Absolutely. There’s actually been a lot of debate around this in the video industry in the last couple of years. So I think that there is some real concerns from developers being asked to wade in, like sort of wade in to the community. And I think it’s been interesting to see a lot of developers pushed back on the idea of actually getting out there and having one on one relationships with players because like the feeling from a lot of developers is that the players are too needy, or they’re too entitled, or they’re too demanding. And I tend to think of it as a, we are not doing a great job of training and supporting our devs in doing community management. What do you think about those, those concerns?

Raph: Yeah, it’s, it’s sort of one of those situations where everything can be true at once, right? Like, it is true there in any given population of players, there’s going to be some that are demanding or expensive or trolls or whatever. That’s just statistical. It’s going to happen.

Um, it is true that it is a ton of work and an expensive small studio may not be able to afford to allocate the headcount.

It is true that, um, you’d need to do it from a business expense.

Like all of these things can be true at once and there is no easy way to cut through it. I completely agree that developers are — I mean everybody is increasingly in the public eye. I’m not even sure it’s safe to say developers. I kind of feel like people need a license to Twitter where it’s like, okay, let’s put you through some basic training on how not to end up getting flashmobbed on Twitter.

Crystin: Right?

Raph: Because it’s very easy to fall into those traps. One unwary remark and you might suddenly be trending worldwide for no good reason and destroy your life. And that’s pretty terrifying. So I think there’s plenty of blame to chalk up on the platforms themselves again, there’s lots of social design there that I don’t think is optimal. I’ve given talks about that. So people who want to can go look on my website or whatever and find them.

But I think it is also incumbent, particularly on companies, to take care of their employees and protect them. And that means helping the employee by giving them the tools to cope, which means things like a certain amount of social media training, you know, um, when and how to use conciliatory language…

James: How not to escalate, you know..

Raph: …how to deescalate tension. Um, you know, how to respond to criticisms in a way that doesn’t inflame things.

Crystin: It’s hard. I mean, I know me personally, when I started getting death threats, like our studio just didn’t really know what to do. Right. I mean, you know, like they were like, well, we’d love to help, but what do we do about this? And I think that it’s a new world. Like we don’t always know what to do about it.

Raph: Yeah. I remember it was clear back UO that I got my first anonymous package in brown paper wrapping at to my home address, right? Yeah, it was, it was back on UO, the first time that some podcast called my home phone number in the middle of the X-Files. Actually, I was really mad at them.

Crystin: Um, would you call back in thirty minutes please?

Raph: Exactly. Um, yeah, and I, and I’ve been unlisted ever since. Right. But it’s, uh, yeah, it’s, it’s a, it’s a challenge and I think it comes down to the players see the developers out there.

Developers are answering and engaging. Yes. Twitter is not a private platform in any way. Everything you do is public. There’s no way to avoid that. Yep. You might think you skate with a security through obscurity for, you know, when you have 500 users or a thousand users or whatever. But you know, yeah, you’re, you’re in public and that’s it. Deal with it.

And which is unfortunate, and I hate to say it that way, but then it also means that the, the company, the employer has a couple obligations. Training is one, right? But also if their employee missteps and the community goes after the employee, part of it is the company saying, “we have community standards. If our employee said or did something wrong, that is an internal disciplinary matter for us to deal with. Right. And our community values are respect, civility, politeness. Therefore, even if the employee did something wrong, which we’re not going to discuss, don’t.”

Crystin: Right.

Raph: Right. And that’s a community standards thing. And I think it’s super important that anybody in a service organization understand that community standards like that have to be reinforced daily, in advance, so that when you need them, the culture is there already. Right. And that’s a difficult lesson.

Crystin: It is. I had some of it’s framing. I used to tell my team that we should think of ourselves more as a professional sports team. Um, and that one of the services we offer to our fans is a safe thing to be angry about. So they were going to get angry. Right. They were getting mad at us. Right. They were going to get on the forums.

Raph: “How could you miss that shot!” such shot.

Crystin: Absolutely. Right. But it’s not personal. It really, that’s really what it’s about. It’s not that they hate you or they’re really out for blood. This is heart of a hobby. It’s a safe thing too for a lot of people just debate some people to yell and scream and get very fiery about.

Raph: Yeah. It’s a thing to get emotionally invested in. Right, right. Passion is actually the good sign.

Crystin: Yes.

Raph: Right. What your enemy is indifference. Indifference is death for a service.

Crystin: People on the forums screaming every day are playing the most, a lot of the time they are spending the most too. They’re very invested.

Raph: They’re very invested.

James: I’d like to take this in a completely new direction. So let me throw this out there… We had a very interesting business conversation yesterday with a, a Korean developer and they have been making huge investments in machine learning, AI on the live ops side. And they were very proudly saying that, you know, cause we’re talking about costs and they were very proud to saying that they’d gotten in one of their games or live ops DNS may be hyperbole, their live ops team from a hundred people down to four people.

Raph: Wow.

James: Thanks to you know, really rigorous use of essential optimization techniques. Very curious to hear your thoughts. You know, either a) are you seeing or hearing similar stories and b) you know, where you think this could go and you know is a good thing or bad thing. And cause I think it’s an inevitable thing. I’m curious to see how we, cause we have, we have some role to help in guiding where that could go.

Raph: Of course. Um, uh, this is certainly the first time I’ve ever heard of a 96% reduction. So that sounds like a, an enormous outlier to me. And my gut reaction is unwise, but, uh, is machine learning assistive? Absolutely. And um, it’s really… alright. It’s important that everybody know machine learning is a really nice buzz word right now. But I mean, and it’s really cool and awesome and it is not nearly as fancy as people think. Right? Like it is not magic. It is not even AI. Okay. Right. And I think, you know, probably your listeners are sophisticated enough to know that, but I think it’s really important that people understand machine learning is essentially, it’s, it’s a fancy pattern recognizer engine and um, or alternatively it is a fancy pattern variation generation engine and that’s about it.

Now it’s a giant brute force one of those. But once you frame it in that context, you start realizing that, well, but particularly for something like this, pattern recognition is probably what we’re talking about more. You know, generation, there’s tons of interesting things down that direction, probably not live ops, directly live ops related.

Um, as soon as you frame it that way, then you go, okay, what are things that live ops has used pattern recognition engines for before? We just have a better one now. And suddenly it’s like, oh, okay, chat filtering jumps out at you. Um, you know, recognizing troublesome, uh, service histories over time jumps out at you. You know, there’s a whole bunch of things like that, where machine learning could immediately apply value. And not just a reactive value like chat filtering, but uh, because it can detect subtler patterns, it can offer predictive value. Like you can, you can analyze ’em, you can analyze chat in order to detect “this person is going to explode in profanity soon” rather than “they already did.” Right? That sort of thing is of course, incredibly valuable.

Um, uh, churn. You know, churn is a great example. We have been able to predict churn to very high degrees of accuracy for like a decade and a half. Right. And machine learning is going to let us do it… marginally better basically

James: We can react to it dynamically a little bit…

Raph: Yeah. And we’ll be able to respond faster to it, you know, so the efficiency gained will be consequential, but we’re still talking about going from like even a crude model of “how often has this user chatted with other people.” Right. That by without doing anything else will get you about 80%. Yeah. Like literally 80% accuracy on predictions. So we have a pretty good starting point, right. So there’s enormous value in going from 97 to 98. Don’t get me wrong, but, uh, but I think it’s super important that people realize, you know, this is another step along the way.

Um, it cannot replace human connection.

James: Yeah.

Raph: That would be the generative side. And we’re still not good enough at the generative side. And if anything, I’d be actively concerned. You know, we’ve all had this experience. We, especially the spam calls these days, you know, he gets a phone and somebody who, you know, it rings and “Hello? Is Frank there? Wait, no, oh, well I’m glad I got you. I’m here to talk to you about this.” Click right here. Right. Um, and I, and I think most people are angrier at that than I am at an actual person calling saying, “I’d like to interstate, are you interested in solar?”

Crystin: Right.

Raph: Um, and the same has been true, like the more impersonal the contact is, and part of what we resent is the fact that it feels like fakery and trickery. Right? And so there’s this temptation to go use, let’s use the generative side and procedurally answer all these questions. Yeah. That actually really worries me. I think if any valley exists for more than just a visual, exactly. You know, it will come across as they don’t actually care.

Crystin: Right. Right.

Raph: And so that’s why a 96% reduction would worry me. Sure. But is there enormous value? Absolutely. There’s a number of things out there. We are, we are seeing entire companies forming our own middleware for doing things like, um, uh, AI based moderation, right. With machine learning of various sorts, often in really clever ways. Right. Um, you know, having NPCs react to bad behavior, things like that.

Crystin: Cool.

Raph: But, um, you know, there was a bartender in Celtic Ireland on legend mud that when a player was in the tavern and cursed, she would march over frog marched him out of the tavern and wash their mouth out with soap. Right? So yeah, you know, simple RegEx can go a long way and the power of that is actually in the frog marching and the soap, right. Not the detection. Right. I think that’s super important. Right? It’s really important and machine learning isn’t going to provide that kind of magical spark, right. That that made that moment where like everybody laughs and stops cursing. Yeah.

James: Well this looks like it is really important. Yeah. And I think, you know, tools like the ones that we provide allow games and capture a ridiculous amount of data if they choose to. And yet we, one of the common things again from these we’ve had this week is it’s hard. Analytics is hard and it’s hard to know what to capture. It’s hard to know if it’s right. It’s hard to know to do with it. Love to hear again. You’re kind of just musings on, you know, analytics and data, both. Other advice for developers just getting started or lessons you’ve learned about, you know, using it effectively.

Raph: I think the first and biggest thing is people need to think about what are their true KPIs. Right? And you know, I mean that’s kind of an old business school lesson, but, um, but I think people still get it wrong all the time. You know, when they, when they start thinking about, let’s say it’s gameplay analytics, it’s still about, okay, what are the things I actually care about? Right, right. Like, it can be awesome to measure how many player weddings were held in my shooter or whatever, but there’s not going to be a KPI probably.

Um, it’s really easy to fall into the trap of activity versus retention or you know, what one of those really big ones. And picking the wrong one of those, right. That is a really easy trap. Um, the, the, the, the thing I always tell people is that, look, if you really take the step back, I’ve had access to an awful lot of granular data about usage, acquisition and churn for a lot of online games. And the shape of the curve on all of them is exactly the same year.

They all have exactly the same curve. A J curve and an s curve are just slices of the same curve. And if you run enough regression analysis on the curve, you can actually predict your peak usage from the first like eight hours of data. It’s actually shocking how accurate it is. Um, and, uh, if you, you know, you can go look up things like the Bass diffusion curve and you know, all of this stuff and it really makes you realize that the, the, the thing that separates the fad from the viable service is whether or not the curve stretches out.

Crystin: Right.

Raph: And again, it’s area under the curve and the area under the curve is going to be driven by what are your key metrics. There are business situations, absolutely. If your key performance metric is I need to get this much money in this quarter, okay, that is fine. Right? Like, some businesses are like that. Um, let’s say it’s a, even in the game as a service context, maybe it’s a DLC or an expansion, in which case you don’t need the curve to be long. Right. You just need it to peak high. That is okay. Even in a game as a service context, sometimes that is the business need, right? Uh, you know, we’ll run short on cash. Okay. I mean, you’ve got a need, you need to work against it. Um, that is the kind of of careful thinking, right, beause we might easily fall into the trap of “it always has to be stretched out as long as possible.” Right. Or we’ve, we had just as often, “it’s gotta be tall.” Right. And No, it doesn’t have to be tall. If it’s really long.

Crystin: Yeah. There’s, uh, you know, what are you trying to do, right?

Raph: What do you, what is are you trying to do? That’s the key to data. What are you trying to do? And actually thinking about it because I think we go on autopilot a lot. That should tell you then what is it that I, what are the questions I should be asking? How do I dashboard and then be very careful about what you dashboard. Don’t, don’t dashboard too much and don’t dashboard things that don’t serve the point.

James: Yeah. So back in back in the days in, you were in a, in live ops are guiding off the tools, almost got like a pilot’s scan rotation on a dashboard. What are the things that you would look at each day? You’d come in the morning and what were the things you’d want to see to sort of check the health of your game?

Raph: Uh, it varied by era.

For early MMOs, the, the things that were most important. Uptime, um, lag latency performance, right? Because both of those then drove population, and for synchronous participation games, if you concurrency falls low, the whole thing falls apart. So population, concurrency, but uh, sentiment. And honestly the first thing I would check every day when I sat down was not the dashboards. It was the forums.

Crystin: Yeah.

Raph: Sentiment was more important. Right. And sentiment is the slowest to move. Well rather — it’s the slowest to move up. It can go down really fast. And so it because of that it’s, it’s extremely sensitive. It’s a predictor of all the others. So if something starts to go south, it’s the first thing you have to work on, right? The server will come back up. You know, people are going to get that back up and it’s not going to take a week, but it might very well take you a week to fix the sentiment. A week might be quick actually.

So back in those days where you were in it very clearly for the long haul. Um, you know, an an MMO kind of thing, where that the moment your, your sentiment, uh, predictor is running on a month out because it’s, what’s the resubscribe moment, right. You know, so there are shelf moments before that, but the true decision point is when they cancel, right? So that’s, we had to think of it that way.

As we move forward into more modern days, some of those are always there. Performance doesn’t go away. Um, concurrency. So on, uh, those, those don’t change. Um, there’s uh, measuring engagement with other users becomes even more important because many of the games are not structured around it. Right? And in an MMO, you could pretty much assume players are getting together and chatting. It was so intrinsic to the genre.

If you’re making a game where that’s not the case, then it’s something you need to actively tend and actively garden. And even in the Facebook games, this was around things like how many friends, how many active friends are, uh, you know, are the little viral requests for assistance? Are they getting listened to? Are they responding, right? That was our proxy for is there actually a community of people actively engaged here.

One of the common early mistakes in social games was to build giant friend libraries and it turned the process of doing the invites into a chore, which led to less engagement, which led to the game failing. And we found no, that bar cannot be too big. Right? Right. It needs to be something people can use. So it shifts over time. It shifts with, with the genres. Um, but you know, there’s always the truisms of uptime, community sentiment, performance

Crystin: As we wrap up, do you have any disaster stories you want to share with us?

Raph: All kinds? Uh, probably one of the most famous in my case was Christmas in Ultima Online. We always gave away Christmas presents to the players every year and early on one year we wanted to give them a present. We hadn’t budgeted development time or art time for a new piece of art. And we were, you know, we just misjudged, you know, there was a cycle for patching art into the game back then. And it’s like, oh no, we forgot to have something in there.

So, um, I took a pine tree and we already had, we had pine trees and we also already had a little colored rocks and you know, on Ultima Online we had a s an extremely powerful server side game scripting system, uh, which is an enormous amounts of the entire game were built in that.

It was, by the way, something I highly recommend for all lives off teams code as data server side updates, not client updates you shouldn’t need to do builds to keep the game fresh.

Um, so what I quickly hacked together was a little, you got a little deed in your backpack and you could take it out and place a Christmas tree in your house, or outside or whatever, and it would now place the pine tree and then spawn all the little rocks and each of the rocks had a little callback timer to turn it to go invisible and come back. So they were all blinking. It was a really pretty little Christmas tree.

James: Yeah.

Raph: It was a really beautiful little Christmas tree. Until I overloaded the message callback queue for the entire service. And because of course, everybody got the deed on Christmas morning, I brought down the entire service on Christmas morning.

Crystin: Destroyed the service with Christmas trees.

Raph: Yeah. Completely. Um, fortunately because of that style of backend, it was a one character fix. Right. Like literally comment out the blink and we were back up. Yup. But boy was my producer, angry, boy where the engineer’s mad because it took them a while to figure out what it was.

Um, I have many, many more embarrassing stories like that. Oh, we could go an hour on just those. Absolutely.

Crystin: We could, yeah. I’d love to hear. But I think it, it makes people feel better knowing we all make mistakes. Yeah.

Raph: It’s live. It’s live. I mean, also, I’m not sure why it was always Christmas. Uh, but, um, we spawned Santa Clauses in the towns and they sat there and they would periodically say, “Ho, Ho, Ho, Merry Christmas.” Wonderful. We, we forgot to make it…. we’ve remembered to make them invulnerable….

Crystin: That was good. Nice.

Raph: We forgot to make it so that players couldn’t steal the clothing off. And, um, you know, Santa hats and Santa outfits were very quickly a hot commodity, which meant we were faced with two simultaneous problems, right? The first was you now log into the game and there were naked men in the game standing in front of you going, “Ho Ho Ho, Merry Christmas,” That was not so good.

And then what was perhaps just as bad is that we also had roving bands of homicidal Santa Clauses who would rush at you, kill you, and take your stuff chanting “Ho Ho, Ho” as their battle cry. So…

Crystin: That’s amazing!

Raph: well, and I think that’s the, that’s one of the beauties of live ops. Okay. Um, is, you know, sometimes this crazy shit happens and it’s actually, it can generate the best stories, the things that people remember that happened 20 years ago. People still tell these stories. And so, you know, to some degree you gotta roll with the punches. You gotta lean in on them sometimes, like, okay, why not?

Um, uh, one of my favorites is we allowed people to build furniture in Ultima Online. So, a guild decided to block all the entrances and exits to a city by piling up armoires and desks and chairs and they prevented people from getting out the main gate. And there was another gate that was really so players would walk out and get bopped on the back of the head and robbed the minute they came out.

Crystin: That’s classic. That’s actually like a medieval technique.

Raph: It is. It is. So our live ops solution was to lean in and the next patch note, we just quietly mentioned that access could now chop up furniture and…

Crystin: Yes, game play evolved and turned into drama!

Raph: Right. And, and I still favor that, you know, that sort of approach rather than the approach of having somebody just delete all the furniture from the database. Right. That’s not how memories prompt drama. It’s like yes and no, but the genesis of live was, well, it’s kind of like we’ve been in rehearsal, right, to shows live now. So yes, it is improv drama. That was literally where we were trying to draw the, the word from.

James: That’s awesome. Well thank you so much. Yeah. That’s a really, really good, good, good stories are great lesson seen.

Raph: Yeah, happy to.

James and Crystin: The LiveOps Academy podcast is brought to you by Microsoft Azure. PlayFab PlayFab is a complete live ops solution with everything you need to iterate, optimize, and grow your live game. Learn more and sign up for a free account at playfab.com that’s p l a y f a b.com. playfab dot com is also home to the live ops academy. Check out blogs, design guides, webinars, and more. All focused on learning about live ops in games. Visit playfab.com/live ops academy to access our backlog of great content and see what’s new every week. Thanks for tuning in.