Learn More about In-House Testing for SBS

star wars playtesting slam bolt scrappers fire hose games
Iterative design has been a big part of how we created and refined all the elements in Slam Bolt Scrappers. What is “iterative design” you ask? Think of it as an evolutionary process, where you continually kill off the weaker bits of the game in order to focus on the parts that people respond favorably to. The secret trick to pulling this off is continually finding people who are relatively unfamiliar with the game to play through it and give feedback on their experience. Gamers in the Boston area have treated us well – we’ve had a constant flow of people coming through our studio to tell us what they think of SBS. It’s been a great experience meeting all these local gamers and introducing them to the game, and it’s resulted in some major changes in SBS throughout its development.

Jared Stander from Dig Boston, recently sat down with Eitan, our Fire Chief, and grilled him about our experience for the article GEEKED: FIRE HOSE GAMES.  I wanted to highlight a few interesting parts about the “playtesting” that helped enable our iterative design process.

Jared: I know for the testing process you’ve been having people come into the studio for this ‘open beta’, why did you choose this method over a digital distribution model?
Eitan: Normally speaking, “open beta” is when you sit at your computer and download the beta. But the thing is that it’s a PS3 open beta, so we can’t easily give people downloadable stuff. So instead our open beta was, if you lived in the greater Boston area and you wanted to come and play the game, all you have to do is email us and say “I want to come test,” and that’s it. We hook you up, you come in, you test. You can sit on our couches, play out game, eat our snacks and drink our soda and you can just play the game. And, you know, we don’t get to quite as many people as a public open beta would on the PC, but this is what we can do and it works really well for us.

J: Are there any major changes you decided to make to the game based on your ‘open beta’ testing process?
E: Oh my God, yeah. So many. There’s so many changes. This isn’t terribly interesting, but it’s a good example. We move the tutorial. We have a tutorial level in the game that teaches you how to play. There’s a world map in our game, and it used to be that the tutorial was the first thing you saw on the world map. And we found that users were occasionally missing it because they wouldn’t go to the world map, they’d go straight to battle mode and then they’d start doing that. So after seeing that in testing, that people weren’t finding the tutorial always, like maybe half of them were missing it, we ended up taking the tutorial out of the world map and we moved it right to the main menu. So when you come into the game, the first thing you see is kind of a ‘how to play’ section which kind of teaches you how to play, before you even get to the world map or even get to battle mode. And that fixed that problem. So that’s not a terribly interesting thing, but it changed the dynamic of how our map looked in out game and how things were set up. And that’s just one example of a million things that we changed based on testing. Which is why people should come in and test for us. Because if they come in and play it, they’re actively helping make the game better.

J: I think it’s really cool having people come in and test in the studio as opposed to what a lot of these large game companies, like Bungie with Halo for example do. Where you’re in the game environment and they don’t really talk to you… They use this cold machine to measure all of your statistics to find performance problems, so I like your personal way of doing things.
E: Well, saying a cold machine, I actually don’t think that’s fair. I agree with some of what you said, but sometimes a cold machine is better and here’s why: If I’m sitting in a room with you and you’re playing my game, you’re going to like it more than you would otherwise because you’re sitting in here with me. Sweet, oh isn’t that fun! But if you’re just sitting with a machine, you have no incentive to like the game anymore. It’s amazing how psychology works that you’re going to like the game much more if the maker of the game is sitting there with you than if you had a machine sitting there. So actually, even though it sounds terrible, Bungie‘s getting more honest feedback that way, because no one’s sugar coating anything. It’s just a machine. Who cares if you piss it off? Versus, you’ll never tell me that you hate the game because you like me and you’ll feel bad about that. So in their defense, that actually makes sense. That being said though, indie studios are still way better and you’re totally right.

As the lead playtesting coordinator person, I’ve had a lot of interesting experiences getting feedback from gamers and helping the team focus on their concerns.  Questions? Leave em in the comments!

And don’t forget to check out Jared’s original article here.

Lastly, if you want to come playtest our game, drop us a line at test@firehosegames.com.