Performance Regression Unit Testing: A Case Study

Abstract

Including performance tests as a part of unit testing is technically more difficult than including functional tests -- besides the usual challenges of performance measurement, specifying and testing the correctness conditions is also more complex. In earlier work, we have proposed a formalism for expressing these conditions, the Stochastic Performance Logic.

In this paper, we evaluate our formalism in the context of performance unit testing of JDOM, an open source project for working with XML data. We focus on the ability to capture and test developer assumptions and on the practical behavior of the built in hypothesis testing when the formal assumptions of the tests are not met.

Keywords

• Stochastic Performance Logic
• regression testing
• performance testing
• unit testing
• performance evaluation