Rom's Rants

Wednesday, July 18, 2007

The Myth Of Testing Tools In Games

I'm a fan of test automation. I keep up on all of the latest and greatest automated testing tips and techniques from people like The Braidy Tester. I try to tie as much automated testing as possible into the applications that I write on a regular basis. When it comes to application testing, you won't find many supporters as die-hard as I am. However, I do recognize that test automation has its limits. In games testing, automation testing is for the most part only useful for verification purposes.

What does that mean? It means that you can use automated testing tools in games to verify that content is formatted as described and to a lesser extent verify that the content is "well formed," but unless your regular testers find a repeatable type of content failure and are able to train a tool to identify that particular type of content failure, you won't be able to find what is wrong with your content.

You can use automated tools to automate game UI testing and level load testing, but very little can be done to automate gameplay testing for 99% of the games on the market. You can use automated QA to generate the massive amounts of combinations for combination testing, but you still need a human to evaluate the results in most cases.

You can automate harnesses against backend servers to ensure that the proper errors are thrown and that the proper data is passed back and forth, but you still need to be testing the game itself against the server caused by humans.

While most applications can gain a real benefit from test automation and can even reduce their test headcount needs via automation, video game testing is almost the last place where flesh and blood cannot be replaced effectively at this time.

Unfortunately, many people are under the impression that automation testing for games is a lot further along than it really is. Look at Dave Perry's take on it. (I've already called him a God-damned idiot, what else can I do?) Back in March, I dug in a bit deeper against his assertions.

Long story short, investing in testing tools in games will help release a better product, but it will not replace the need for an effective tester to wield the tool.

Labels: Automation, Testing

posted by Michael Russell at 7/18/2007 09:35:00 AM | 0 Comments Links to this post | View blog reactions

Friday, November 17, 2006

Automating Games QA (part 3)

In this installment, we're going to be talking about combination testing.

Most games nowadays have some sort of customization system, be it your character, your "crib," your vehicle, etc. Testing the entire gamut of combinations by hand can actually get to the point where it is impossible to test within the time available for testing.

For example, let's say that you have a standard human avatar with a customizable shirt and pants. There are 10 different shirts available and 10 different pants available. That is 100 combinations right there. Let's add 10 hairstyles. That bumps it up to 1,000 combinations. Add 10 different fleshtones...10,000 combinations. Add a second gender...20,000 combinations. Add 5 different faces per gender...100,000 combinations. It adds up quickly.

Combination testing is designed to hit the two simplest types of bugs: single-value and two-value settings.

Now, if you look at the example above, while there are 100,000 combinations, there only a few individual settings: 10 shirts, 10 pants, 10 hairstyles and 5 faces per gender. That can mean either 40 settings if you assume everything else equal between the genders beside the faces, or 70 settings if everything is seperate between the genders. An automation script to individually cycle each of these single-value settings can quickly help eliminate bad items, and if the script screencaps each item, manual verification of item appearance will go fairly quickly.

One last thing: while handling the single-item tests, check the amount of memory that each item uses. A good additional test is to set all of your settings to their most memory-intensive setting and play the game that way to check for borderline out-of-memory conditions.

The second most-common type of bug is when two values are interdependent on each other. For example, you may have a hairstyle that clips through some geometry on a certain shirt in this example. Now even with automation, you're still going to have to manually verify the screenshots, so you want to minimize the number of shots you are looking at. This is where all-pairs testing comes in. This type of testing is also called pairwise testing, and there is a very in-depth example here.

This gets very easy if your combination lists are data-driven. Feed your lists into a tool like ALLPAIRS from James Bach and pass those lists into the game for testing. Pass the created list into your framework and have at it.

Automation testing is all about getting the grunt work done by the machine so you can focus on the non-automatable tasks. Always be on the lookout for tasks like this that you can automate.

The next games QA automation column will be on automating content testing. It's probably the hardest type of automation, and it will not be a good fit for most studios, which is why I left it for the end. Stay tuned.

Labels: Automation, Testing

posted by Michael Russell at 11/17/2006 10:58:00 AM | 0 Comments Links to this post | View blog reactions

Monday, November 13, 2006

Automating Games QA (Part 2)

Note: Prior to reading this post, please read the previous entry (link here) as well as this article on Gamasutra. Mr. Cooke only briefly touches on the test system architecture on page 4.

This post is meant to discuss game flow testing. The role of game flow testing is to verify that the appropriate content loads and that the proper branches are followed. This is generally more complex than simple UI automation testing and can require additional hookups as part of your game hookup.

The basic flow of a game flow automation script is to start the game from a known state (generally "New Game") and pass level completion/failure states to the game so that the game will progress to the next state, and so the game will record which state it is in at each step.

For example, let's say that you are working on a linear first-person shooter with no inter-level transitions. (Think Doom 1.) Your script starts a new game and verifies that you are in E1M1. Your script then passes in a command to tell the game that you have beaten E1M1. It then verifies that E1M2 loads. If you have a failure case or an alternate test, you'd handle that in a seperate script.

That's the simple answer in a nutshell. You aren't just loading each level, you're attempting to verify the links between the levels as well. If you have a data-driven level flow, you can often automate the creation of the scripts from that dataset.

One other nice part of this is that you're also checking for compounding issues. A lot of the time, issues won't manifest themselves in a level that is loaded anew because the issues are the result of memory fragmentation or memory leaks from a previous level. This is a nice, automated way of helping to bring those problems to the forefront.

Now, this does not remove the requirement that you still play through each path. This verifies that the links exist, not that the links can be triggered through normal gameplay. But if the link is broken in the script, you can save some time during testing by avoiding doing a known broken task.

In the next installment, we'll be going over combination testing, and the final installment will go over content testing theory and practice.

Labels: Automation, Testing

posted by Michael Russell at 11/13/2006 01:10:00 PM | 0 Comments Links to this post | View blog reactions

Wednesday, November 08, 2006

[Testing] Automating Games QA (Part 1)

In most software houses, automation has made major inroads over the last several years. Vista has over 700,000 lines of automation code that is run against every build. Companies like WorkSoft try to make creation of automated test cases easier for average test departments. Even unit tests run by developers are, at least in some way, considered automated tests.

However, automation testing has had a difficult time infiltrating game development houses for a couple of fairly hefty reasons. The first reason is that the number of bugs contained in the code are generally dwarfed by the number of bugs contained in the content. The second reason is that it is difficult to impossible to automate most games because most games have an element of randomness to them, and as such, it makes it difficult to determine success or failure on a test case in an automated fashion. Finally, there is rarely, if ever, a standardized way of querying a game about the state that it is in, or even passing input to a game to trigger a response.

This isn't to say that automation is impossible in a game scenario, but automation does require an additional level of developer interaction and even imagination that other scenarios simply do not have. There are generally four areas where automation testing can be efficiently used in game development: User Interface, Game Flow, Combination and Content.

User Interface automation testing is where you are going to see the biggest initial gain from a QA standpoint, and is an excellent place to push for an automation start in any company. The goal should be that anyone in your QA department should be able to write automation test cases without much training. I'm going to describe a simple framework that you can share with your development team as a starting point.

A game UI automation framework generally consists of four seperate components. The game hookup, the communication component, the use case library and the test cases themselves.

The "game hookup" is a piece of code inside the game itself that listens for commands and queries from the communication component. For example, on the Xbox, your game hookup may just be a background thread that sits and listens on the debug channel. On a PC game, it may listen on a named pipe or an IP address or some other similar item.

The "communication component" is generally going to be a COM component that sits on your PC, and is responsible for brokering communication between the use case library and the game hookup.

The "use case library" is a set of user actions in user-oriented-named subs written in VBScript. Each use case sends the appropriate commands to the communication component to execute a certain action, and requests information from the communication component to verify that an action executed correctly. This is usually jointly maintained by the developers and more technically oriented testers.

The "test cases" are the actual test cases that call the subs in the use case library to handle each individual test case. Let's hook all of these up and see how these would work and evolve over time.

You are working on an Xbox title. As a test to verify that UI automation is useful, the development team agrees to create the game hookup and communication component. The game hookup will only recognize a limited command set, "getscreen," "getcontrol," "getvalue," "nextcontrol," "prevcontrol," "cancel," and "activate." "Getscreen" returns the name of the currently active screen. "Getcontrol" returns the name of the currently active control. "Getvalue" returns the value of the current control, but can later be extended to let you query more data. "Nextcontrol" and "Prevcontrol" are simply tab-order style items, and act by sending the "up" or "down" input for the main controller. "Cancel" acts as the B button. "Activate" acts as the A button or START button. The communication component takes one argument when it is created: the name of the Xbox on the network that you want to control via automation. It has a sub for passing a command, a function for passing a query, a sub for sleeping for one tenth of a second, and a sub for restarting the console as a cold boot and launching into your game.

You decide that all of your test cases are going to be designed to start from power-up. As a result, your first use case will be get to main menu. The flow at this point in development is no videos, just the "press START" screen and then the main menu. You write your use case similar to this pseudocode...

Sub GetToMainMenu(Xbox1 As TestControls.XboxControl)
  Xbox1.Reboot
  For X = 1 to MaxTimeOut
    If Xbox1.Getscreen = "press_start" Then
      LogSuccess "At Start Screen"
      Xbox1.Activate
      For Y = 1 to MaxTimeOut
        If Xbox1.Getscreen = "main_menu" Then
          LogSuccess "At Main Menu"
        End If
      Next Y
      LogFailure "Could Not Get To Main Menu"
      Exit Sub
    Else
      Xbox1.Sleep
    End If
  Next X
  LogFailure "Could Not Get To Press START Screen"
End Sub

A sample test case that called this would be:

Sub TestSystemLinkMatchmaking
  Dim Xbox1 As New TestControls.XboxControl("TestXbox1")
  Dim Xbox2 As New TestControls.XboxControl("TestXbox2")
  GetToMainMenu Xbox1
  GetToMainMenu Xbox2
  ...

As part of your nightly build process, you then hook up a couple of these scripts to be executed after the build is done and deployed to some Xbox test kits. When you come in the next morning, check the logs.

So, let's say that TestSystemLinkMatchmaking reports a failure. It's easy to open the test case and manually try the steps inside to verify the failure. If it does fail, you can even tell the developer to just run the automated test case to trigger the failure. It saves the developer time to repro and debug the problem.

If your use cases are prolific enough, you can even write individually coded cases for regression testing and repros of manually found bugs.

It requires some effort to create a system like this, and changes to the UI require adjustments to the use case library. Major changes may even require changes to the test cases themselves. However, the time savings in comparison can be fairly hefty.

Obviously, this is an extremely simplified example, but hopefully it got you thinking.

In the next installment, we'll go over game flow testing.

Labels: Automation, Testing

posted by Michael Russell at 11/08/2006 12:02:00 PM | 1 Comments Links to this post | View blog reactions