Mutation Testing Repository

  • Fittest_Banner
  • Empirical Studies

    Far more new real programs than toy programs have been studied...

    Learn more

Mutation Testing Publication Trend

To understand the general trend for the Mutation Testing research area, we analysed the number of publications by year from 1977 to 2008 (the publication data in 2009 is not complete). This results are depicted in the figure below. In the figure below, there are three apparent outliers in years 2001, 2006 and 2007. The reason for this, is that there were three Mutation Testing workshops held in 2000 (with proceedings published in 2001), 2006 and 2007. The reader will also notice that 1986 is unique; there were no publications found. An interesting anecdote was provided by Offutt: "1986 was when we were maximally devoted to programming Mothra. I spent 70+ hour per week banging my head against vi, gcc, and dbx."

We performed a regression analysis on these data, and found there is a strong positive correlation between year and the number of publications (r = 0.8005). In order to predict the trend of publications in the future, we have tried to find a trend line for this data using several common regression models: Linear, Logarithmic, Polynomial, Power, Exponential and Moving average. The dashed line in the figure is the best fit line we found. It uses a quadratic model, which achieves the highest coefficient of determination (R^2 = 0.723). From this analysis is clear that Mutation Testing remains an active research area with growing interest.

In order to take a closer look at the growing trend of the research work on Mutation Testing, we have classified this work into theoretical work and practical work. The theoretical category includes the publications on the hypotheses supporting Mutation Testing, optimization techniques, such as techniques for reducing computational cost and techniques for the detection of equivalent mutants and surveys. The practical category includes publications on applications of Mutation Testing, development work on Mutation Testing tools and related empirical studies.

The goal of this separation of papers into theoretical and practical work is to allow us to analyse the temporal relationship between the development of theoretical and practical research effort by the community. The figure below shows the overall cumulative result. It is clear to see that both theatrical and practical work is increasing by year, and in 2006, the total number of practical publications surpasses the number of theoretical publications for the first time.

To take a closer look at this relationship, the figure below shows the number of publications by year. From 1977 to 2000, there were fewer practical publications than theoretical. From 2000 to 2008, most of the research work appears to shift to the application area. This provides some evidence to suggest that the field is starting to move from foundational theory to practical application, possibly a sign of increasing maturity.

Empirical Evaluation

To provide an overview of the trend of empirical studies on Mutation Testing to attack more challenging program, we calculated the size of the largest subject program for each year. A cumulative result is shown in the figure below. Clearly the definition of ‘program size’ can be problematic, so the figure is merely intended to be used as a rough indicator. There is evidence to indicate that It is clear to see that the size of the subject programs that can be handle by Mutation Testing is increasing. However, caution is required. We found that although some empirical experiments were reported to handle large programs, some studies only applied a few mutation operators.

We also counted the number of newly introduced subject programs for each year. The result are shown in the figure below. The dashed line in the figure is the cumul ative view of the results. The number of newly used subject programs is in gradually increasing, which suggest a growth in practical work as well.

In the empirical studies, it may be more indicative to use a real world program rather than laboratory program. To understand the relationship between the use of laboratory programs and real world programs in mutation experiments, we have counted each type by year. The results are shown in the Figure below. In this study, we consider a real world program to be either an open source or an industry program. In the figure, the cumulative view shows that the number of real world programs started increasing in 1992, while the number of laboratory programs had already started increasing by 1988. The figure also shows the number of laboratory and real programs introduced into studies each year as bars. This clearly indicates that, while there are correctly more laboratory programs overall. Since 2002, far more new real programs than laboratory programs have been introduced. This finding provides some evidence to support the claim that the development of Mutation Testing is maturing.

Tools Support Mutation Testing

In the figure below, the dashed line shows a cumulative view of this development work. We can see that the tool development trend is rapidly increasing since year 2000, indicating that research work on Mutation Testing remains active and increasingly practical.