Software Testing: A Forgotten Art

The basic purpose of software testing is to ensure that a software system is free of significant defects and that the system possesses the required functionality to meet its intended objectives. There are a variety of types of testing that can be used to certify that a system is truly ready to be used by its target audience, of which the most important is what is known as system testing.

I'll provide a whirlwind overview of a typical system testing process as it applies to web sites and web-based applications. Then I'll describe several other types of testing that can be used in conjunction with system testing to ensure the creation of quality software systems.

System Testing

System testing needs to be organized into a systematic and comprehensive process in order to be effective. For this reason, system testing usually requires a detailed plan, a blueprint of how testing will be done. Such a blueprint is known as a System Test Plan, and is a formal document that defines in excruciating detail how system testing will be conducted.

The plan typically consists of a variety of test cases, or scenarios, which are specifically designed to exercise the functionality of the system in significant ways. A System Test Plan should have the following characteristics:

  • Testing of Typical Operations: Test cases must provide detailed coverage of the typical operations that will be performed by users. This ensures that testing will cover the majority of activities that users will engage in while using the system.

  • Exercising of Boundary Conditions: Some test cases should be deliberately designed to test software boundary conditions. For example, a test case may be created to test conditions such as the user executing a search with no criteria, leaving a field blank in a form, etc.

  • Repeatability: The test cases must be repeatable. All software changes should be subjected to system testing before being placed into production. This means that the System Test Plan will be executed numerous times, first as the initial system goes into production, and later after defect fixes and additional enhancements go into production.

System testing is an iterative process, requiring a number of cycles to ensure the defect-free status of a sizable and complex system. In a testing cycle, all of the test cases in the System Test Plan are executed against the software (preferably by people other than the original developers, who are too close to the system to be entirely objective).

Any defects revealed during testing should be carefully tracked. The defects should b assigned to individual development staff members for resolution. This means that a round of fixes generally follows a test cycle. A new edition of the system software incorporating the various defect fixes will then be created, and subsequently subjected to the next cycle of testing.

This process generally repeats until appropriate decision makers determine that the system should go into production or, if already in production for a limited number of users (for example, in Beta mode), that it should be made available to a larger pool of users. Alternately, system testing may continue until the clock runs out, and the system is pushed out the door, ready or not (Yes, this isn't what's supposed to happen...but it does. Trust me.).

Other Types of Testing

Organized system testing can be vital to the success of sizable development efforts, and typically results in the final certification that a system is ready to be placed into production. But system testing alone cannot ensure a successful and viable project outcome. There are several other types of testing that can also be useful, particularly for large systems.

  • Unit Testing: Developers typically perform unit testing upon completion of various software elements. In other words, developers should have done some ad hoc testing on their own software before submitting it to system testing.

    Warning: One of the often fatal mistakes for projects is to force developers to push software into system testing that is not ready.

  • Usability Testing: Usability testing consists of exposing the system software to focus groups of typical users. In effect, real users are asked to try out a Beta (or pre-Beta) version of the software. They are asked to perform typical operations using the system. After using the system, users are then questioned to elicit feedback concerning ease of use, opinions on the site organization, usefulness of help information, etc.

    Warning: So what if the software works correctly? If the users can't figure out how to use it, the project's a bust. Test your system with real users. Test early, and test often.

  • Link Testing: Any web-based application will have links to other web-based resources, whether part of the same site or on other web sites. These links need to be tested to ensure that none of the links are "dead", i.e. - to ensure that they point to an accessible destination.

    Many links may be exercised in the course of a normal test cycle of the overall System Test Plan. However, link testing is not actually the focus of system testing. Therefore, link testing is an organized and comprehensive effort to test all links accessible through the system. Link testing is typically automated, either using commercial testing software or custom scripts.

    Warning: Dead links make web sites seem unprofessional and out-of-date.

  • Editorial Review: System testing typically focuses on software functionality. An editorial review focuses on content. Some editorial issues may surface during system testing, i.e. - a tester may report typos in the text of a web page. An editorial review is a concerted effort by someone with writing ability (which does not usually include developers) to ensure that the text elements of the system are free of typos, display excellent grammar, flow well and exhibit the proper tone for the system.

    Warning: Typos on a web site are as bad, if not worse, than dead links. At least with dead links, there's a chance that the user won't click on it and therefore won't notice it.

  • Data Quality Assurance: Many systems have important data components, such as information publishing systems or systems that dynamically provide access to information in a database. Any time that a large amount of content is manipulated in order to make it available through an information publishing system, there will be significant data issues.

    An organized and concerted effort will be required to ensure that all content is tested thoroughly to verify that it is free of typos, possesses all required data elements, is organized properly for indexing and has the appropriate structure and formatting.

    Warning: Many clients often have unrealistic assumptions about the ease with which large amounts of information can be published to the web. For a large-scale system, Data Quality Assurance can be a considerably more time-consuming and grueling process than system testing.

  • Stress Testing: Stress testing is an organized effort to overload the capabilities of the system. Stress testing addresses issues such as the number of concurrent users that can be handled by the system, the number of searches that can be executed in a set period of time, the performance of the system with an inordinately large database, etc.


Sometimes it seems like the art of testing has been all but forgotten by many companies, especially dot-coms. How many web sites are there that could have used a little more usability testing for their shopping cart features? How many web sites got pushed out the door with no organized testing at all?

A solid development process encourages developers to engage in unit testing, before they roll onto the next task on their list. Such a process follows this up with organized system testing, supplemented by other types of testing as needed.


No comments yet. Be the first.

Leave a Comment

Comments are moderated and will not appear on the site until reviewed.

(not displayed)