TestRunner is a portable, easy to use, automated software testing utilty. Designed with command line utilities in mind, TestRunner supports exit code monitoring, output file consistency checking, and time-based performance testing. Installation and setup is simple and can be completed in minutes. Adding tests only requires that a new test directory be created with the necessary configuration files for the application. On demand expected results generation takes care of the hard work, completing test creation automatically.
Featuring built-in (subversion) repository operations, TestRunner streamlines collaborative development. Master and slave repository modes, in addition to the default local mode, facilitate integration with automated builder systems, such as BuildBot and Jenkins.
Benefits of Software Testing
Software testing allows developers to continually monitor application functionality, while moving forward with development. When integrated with an automated builder system, testing can quickly catch functional and performance regressions, providing instant feedback and ensuring continual quality improvement.
The TestRunner Approach
Many approaches to software testing exist, each with its own strengths and weaknesses. TestRunner has been designed to test a complete application from the user’s perspective, running the application with specific configuration settings and verifying that expected output has been generated.
- Ensures consistent results for a given configuration
- Upholds end user expectations
- Verifies successful component integration
- Single tests can supply large code coverage
- Completely implementation agnositic
- Lightweight, requiring minimal setup and maintenance
- Large code coverage of tests may make it difficult to identify cause of failure
- Can only cover code exposed through user interfaces
- Requires completed functionality for automated expected results
- Currently has no capability to test interactive (especially graphical) user interfaces
- Output File Consistency
- Monitors application output for line-by-line variance. Intelligently ignores blank lines and #-delimted comment lines.
- Exit Code Handling
- Allows tests to verify error handling within the tested application.
- Performance Testing
- Track application performance, identifying both improvements and regressions.
- Automatic Expected Results
- Generates new expected results on demand, simplifying test creation.
- Built-in Repository Operations
- Performs automatic maintenance of Git or Subversion managed source code, adding new expected results to the source code repository.
- Automated Builder Integration
- Supports Slave mode for easy integration with automated builder systems.
- Parallel Test Execution
- Effectively utilize multi-core systems with concurrent execution of consistency tests.
- Python 2.6.x or later
Projects using TestRunner
License, Feedback and Suggestions
TestRunner has been licensed under a BSD-style open source license that is OSI-approved. You are welcome to use the code however you see fit, following the simple conditions of the license. As stated in the license, you are not required to release any modifications to TestRunner. However, if you make improvements that you think should be included in future releases, please feel free to send them to me [contact].
Along similar lines, general feedback and suggestions are strongly encouraged. If you decide to utilize TestRunner in your project (both unmodified and modified versions), I would love hear about it. I will include a link to your project above, if desired.
Thanks for considering TestRunner!
Last updated: 2011-05-06