Site Menu
Extra Info Menu
Cocoa Units Logo

Bug Testing

When testing for bugs, its nice to have a set of guidelines as to the kinds of possible bugs there are, and the kinds of standard tests there are for them. This is that set of guidelines.

From a black box point of view, bugs basically fall into four different categorys:

  • Well-formed inputs, but incorrect behaviour.
  • Valid inputs, but incorrect behaviour.
  • Malformed inputs, which are not properly caught by the function.
  • Invalid inputs, which are not properly caught by the function.

OK, but what do well-formed and valid mean? How can you have one without the other?

Well-formed means that you are passing in data in the arguments that is in the correct format for the function, but is out of range in some way. This is like passing a negative number to a function that only expects positive numbers. Or like passing a long string to a function that expects strings

Valid is even more restrictive than 'well-formed' is. It means that the data is both well-formed and with the expected range of input values.

These errors make up the sum total of what we can see and affect from outside the functions. In addition, it isn't easy to cover all possible inputs, because the range of valid inputs for any function is often very, very large.

These guidelines are meant to help you think about the kinds of tests that you can create. You don't need to create a test for each API that fits into each of these categories; for many APIs that isn't even possible. These possibilities are meant solely as suggestions.

If you think that there are other possible tests to perform, feel free to drop us an email we'll look into adding it in.

  • Valid Data Tests - This is the simplest, but one of the most important tests; checking to see if the API in question does the right thing when you pass in completely valid data. What do I mean by valid? If it conforms to Apple's documentation, then it is valid. If there is an error from valid data then either the documentation is wrong, or there is a bug, or both.
  • Well-formed but Invalid Data Tests - Well-formed, but out of range data is one of the most easily forgotten problems in programming. As examples here are some possible tests:
    • Data that is the right type, but out of range. This might be an integer that is too large or small, or anything else like it.
    • Buffer overflow or underflow errors. You might get this if a container class like NSArray or NSStringdoesn't specify the maximum or minimum number of items that it can hold; in that case, you can quasi-reasonably expect that it has no limit. I for one plan on testing every container that doesn't specify the maximum or minimum number of items that it can hold in its documentation by dumping as much or as little infomation into it as I can. If it breaks, its a bug!

      As a sidenote, buffer overflow and underflow errors are one of the most common security bugs that most programs have. Programmers often use libraries like the collection libraries to handle the task of automated buffer management. When the container fails to do a proper job of managing its internal buffer, then crackers may be able to use the bug as an exploit. For this reason, I encourage you to try to force buffer over and underflows where possible. The more of these that we catch, the fewer of them there will be for crackers to exploit, which means that we will have fewer cracked macs.
    • Lying to functions that take more than one argument. Certain well known functions like C's printf function can be lied to. The problem is that one argument is dependent on another argument being correct. For example in printf's case, you can create a function like the one below:

      printf("%d");

      and many compilers will happily accept it as being perfectly valid. The problem is that when you run this code, it will look for an argument on the stack that isn't there. At that point, you can't be sure what might happen next. If you're lucky, and the program has few privleges, then the OS will stomp on it, and thats the end of it. If instead it is a program that is running with root priveleges, then it might crash your machine, or worse, it might be tricked into running data as code. Cracker's have exploited this trick in the past, and will probably do so again in the future.
  • Malformed Data Tests - As its name suggests, this is just pure garbage. Most functions can't be tricked by this because the compiler will catch what you're doing, but functions that need to parse data streams can be tricked by this. For example XML Services is an API for a nonvalidating XML parser. Because it is parsing what is essentially a string into a more complex data structure, there is the possiblity that certain kinds of garbage input can be parsed as being correct XML. Hence, it is necessary to test it with everything from valid data to malformed data.

As it was said at the top of this page, please realize that we're human too. These suggestions are incomplete, and need more additions. If you can think of new classes of tests that should be performed, drop us an email.

Homepage Our Goals Guidelines Getting Started Links FAQ Contact Info Cocoa Units at Sourceforge