Making Mods/Effective Testing

Effective Testing

Some Possible Scenarios

If either of the following scenarios strike a chord then this page should hopefully help you to fix the problem. Although, it's probable that you should be contributing to this page rather than reading it.

The developer

So, time has passed and people are interested in playing your mod. You've finished, or at the very least finalised all of the features in the first version of your mod, and it's ready for release. You upload it onto your mod web site with pleasure. Maybe you even dance around the room a little while it's uploading. In a few minutes people will be able to view all of your hard work and tell you how much it rocks.

The reality turns out to be most unpleasant. Your in-box is swamped with e-mails complaining about bugs, balance, and calls for support and changes to be made. All you can do is read in disbelief as all of your hard work is slated, shredded, and generally jumped on by hoards of ungrateful screaming barbarians.

Maybe you should have got some other people to look at your mod. Maybe you did, but their testing wasn't effective enough.

The tester

Wow, cool, you've been accepted as a beta tester on your favourite mod. You've been anticipating this one for a while and you're really excited that you'll get the chance to play it before anyone else. And give feedback. You install the mod, fire it up, and it fails to even start. "What sort of crappy half-arsed mod is this?", you cry before you remember it's a beta - which means it doesn't work. After some to-ing and fro-ing between yourself and the developer you eventually get the mod up and running and you start to test it.

While playing with the new hand held automatic limb dismember device you realise that the gun has stopped firing. It's obviously a bug. You send in your bug report to the developer: The gun stopped firing.. If you had a patient developer you might have got a request for more information. You may even have been removed from the beta-test team without any adequate explanation as to why.

What went wrong

In the developer's case, he simply hadn't put enough time or effort (if any) into testing his mod so it crashed and burned on release. In the tester's case, the bug report submitted contained almost no useful information. The feedback provided to the developer was almost useless in helping the developer find the problem.

Why do I need to test my mod

If you write perfect code all the time that always works first time, in the manner you expected, every single time then you can stop reading. The odds are however that every now and then small errors creep into your code. These may be simple typing errors, or they could be non-obvious logical flaws in the structure of your code. Remember, just because your code compiles doesn’t mean it will work in the way that you intended. Consider the code below.

for (I=0;I<10;I++);
  DoSomethingTenTimes();

for (I=0;I<10;I++)
  DoSomethingTenTimes();

Neither statement will produce a compiler error, but only the second one will actually perform the required operations.

Once you accept the fact that you cannot and will not write completely perfect code all of the time then it’s obvious that you need some way of finding the errors in the code. This process is called testing. The aim of the game is to put your code through the wringer, and hopefully find any problems before anyone else does.

How To Test A Mod

Doing it right

Testing a mod effectively is hard. Very hard. The testing process takes up a lot of time, and can also be a very tedious job. There are two types of testing.

Shake down testing – The easiest and lightest form of testing. If your mod is complex then this should be avoided.
Full testing – The only real way of ensuring your mod gets released with the minimum number of bugs.

The type of testing you choose to use will be influenced by the size and complexity of the mod. It will also have a big impact on the final quality of your mod. Shake down testing is great for small map components and simple mutators. Anything more complex than that and you should seriously consider using a full test approach. Otherwise you risk releasing a buggy pile of pap to the general public.

Shake down testing

This is the lightest form of testing. Anyone can start a mod up run around a bit, do some "stuff", and generally "have a look" for problems. This is the type of testing that most developers do. They start up their mod, check out the last thing they changed, and determine whether they are happy or not. They don’t bother checking to see if the change they just made broke something else. If they are sensible, the mod they are testing is sufficiently simple that there isn’t anything else to break.

If this testing technique is used then you should either not bother with a testing team at all, or, make your testing team as large as you possibly can.

If the mod is complex, then this type of testing can be more harmful than useful as it allows the developer to build up a false sense of security in the quality of their mod. This type of testing should not be used for complex mods.

Full testing

This type of testing is time consuming, laborious, and quite often dull. It is however, the most effective way or systematically removing problems and bugs from your mod.

Before you even start your mod for the first time, or while you are writing the code for it, create a list of things that could possibly go wrong. The list of things that may go wrong should be phrased as much as possible as questions that can be answered yes or no. This speeds the testing slightly as the tests become a simple tick list.

Rather than attempt to describe in detail how to come up with the list, here is an example taken from the list I’m building up for my single flag CTF mod. It’s only a small subset of a rather scary looking list.

When the game starts are both Flag Bases on the map visible?
Is there only one flag on the map?
Is the HUD showing that nether team has the flag and that it’s dropped soemwhere?
Is the flag picked up when touched?
Is the flag displayed properly when held (can I see myself carrying it in behind view)?
Is the flag respawned somewhere on the map if I fall into a pain zone?
Is the flag dropped if I use the teleporter?

Once you have your list of tests written out, you need to use it. Every time you do a build that could be a viable release you should run through all of your tests ticking each one off as you see the expected behaviour. If you get some unexpected behaviour then either your test is wrong, or, you have found a bug.

Once you have finished all of your test conditions it’s well worth entering a period of Shake down testing to try out new and strange things that "people would never do in real life". Every time you find a problem that isn’t covered by your existing test list add it. By adding tests to your initial list you are ensuring that a particular problem does not reoccur once removed.

Group testing

If the mod is a reasonable size then it’s worth trying to find some testers to help you. As the complexity of the mod increases the number of tests that need to be done grows hugely as the number of components that can be interacted with, and the number of combinations of components that can be interacted grows. The following are some points to consider when you have a group of testers

First thing you might want to think about is have one central point for bugs to be reported and/or documented (message board?). It does you no good if you have to wade through 30 reports of the same bug. You could expand on this to have someone lead or take charge of this part of the team. Having one source to keep track of the bugs and what has been or has not been fixed allows for the other members to have more time to fix the said bugs.
Delegate and/or assign certain parts of the mod testing to certain people. For example give a certain group of people (or a single person) an area they should look at for bugs and give another group a different area. For example have one group that looks at the weapons while the other group looks at the game play aspect of the mod. This allows for an efficient way to get more coverage and testing as well as it being a more effective way to user your resources. Your testers will also like is as they wont have to spend as much time testing/bug hunting. Some overlapping is a good idea but try to keep that to a minimum. If you did elect one member of the team to handle to be the central point for bug testing this then its should be their job to do the delegating.
Get confirmation from another member when a bug is reported. Sometimes one of your users could have accidentally corrupted a core file that will cause a bug to appear when it is more of a configuration issue on that users PC. For example say if they accidentally loaded a older version of a .u file and get some type of "weird results".
Make sure you have a reliable group. Having a group that does not test their assign areas will not help you at all. Make sure you find people that can dedicated the time that it is needed when your ready for testing. Nothing is more aggravating then posting an update and not seeing any feedback on it. If its at all possible you can try to have a set date for updates. This helps as then your beta testers will know when an update is out and when they need to reserve time for testing.

Don't forget about the network testing

One of the things that it's easy to forget, especially if it's a small mod, is to test your mod thoroughly over a network, preferably against a dedicated server. Network replication is one of the biggest pit fall a coder will have to face. Just because things work fine in a practice match does not mean that they will work in an on-line game. Usually they won't work in on-line games unless the coder has dealt with replication in their past work. The larger the mod the more chances for replication issues to crop up and usually these are the harder bugs to fix. You also can test what happens to your mod on dynamic conditions or when players leave under catastrophic circumstances (drop the network). For example what happens to the flag if the flag carrier leaves? Or what happens to your remote controlled weapon if the owner leaves? You can also check the network use of your mod as well. Finally don't forget to look at the log from the server when testing your mod with a dedicated server.

Reporting Faults

Finding a problem is not useful in itself. The developer is still unaware that the problem exists. How you communicate that problem to the developer is crucial to avoid wasting your time and his.

If you have a tick list of tests, and one of them failed then reporting the problem is easy. You can tell the developer which test failed, and supply him with your log file. Because the developer has an exact duplicate of your list he can see everything you did up to the point your problem occurred. By providing him with your log file he may be able to tell directly from the log what went wrong.

If this is a brand new test then the first thing you should so is update your tick list of tests. Then send the following information on to the developer.

The exact details of the problem.
What you did prior to the problem manifesting itself.
What you were trying to do.
What objects you had in your inventory – and your "player state".

The more detail you can supply to the developer about the bug the better. If you are testing a map then send some screenshots of the problem or feature that caused you the problem.

Always remember to send your log file along with your bug report. The log file is one of the most valuable tools the developer has when it comes to resolving problems.

Don’t be surprised (or offended) if the developer comes back with some more questions about your fault. They may not have been able to reproduce it, or, they might want additional clarification about the exact manifestation of the problem. If you can identify a problem with the way in which bugs are reported then change the way you (and the team) report bugs.

If you have a team of testers, consider using a standard format for error reporting across the whole team. That way new testers will get up to speed faster, and developers can find the information they are looking for easier.

Some common pitfalls

The change is only one line it can't possibly be wrong.: Those tiny one line fixes to your code will quite often cause you far more pain than a complete change to the class hierarchy. The reason for this is that the one-line code change doesn’t get the same level of consideration as a complete class hierarchy change. It’s very easy to fall into the trap of, "It’s only one line it won’t break anything and it will fix my problem.".
Changing code at the point of release.: If you ever change some code prior to building your release, and then fail to test it, your release will have bugs. In the best case you will have introduced a new bug (assuming the one you were trying to fix was fixed by your last minute change). At worst you will have completely broken your mod and it will no longer work.
Publishing an unattainable release date.: As a developer you are always optimistic about how much bug free code you can write in a given amount of time. Publishing a release date for your mod based upon your expectations of when the mod will be finished will out you and your team under extreme pressure. Have you allowed time for testing and bug fixing in your date? Do you have any spare time (contingency) added to cater for interruptions from real life? Bad things happen when release dates are missed. The developer and testers may get discouraged. People will start to lose interest in the mod and accuse it of being vapourware. Your team may get flamed to a crisp for not delivering on your published release date. Your "consumers" won’t care that your PC caught fire, or that a builder has severed your Internet connection so you couldn’t do anything. All they will see is that you failed to deliver, when you said you were going to.