DrupalCon Barcelona 2015: SmarTest: Proposal for accelerating the detection of faults in Drupal
This session is based on a paper (http://www.isa.us.es/publications/type/article-journal/9998/variability-testing-wild-drupal-case-study) that has been recently published in a relevant software engineering journal (Software and System Modeling Journal). I am a researcher at the University of Seville. My Ph.D. is focused on the automated testing of variability-intensive systems (VISs) like Drupal CMS.
In the paper previously mentioned, we presented a real problem with VISs, the testing process. Testing a VIS such as a Drupal system is really challenging due to the potentially huge number of configurations under test. This makes exhaustive testing of a VIS infeasible, that is, testing every single configuration is too expensive in general. In order to address this problem, a variety of testing techniques has been reported to reduce the cost of testing in the presence of variability. In particular, we propose new testing techniques to accelerate the detection of faults in Drupal and other similar systems, providing faster feedback to the testers and reducing debugging efforts.
Based on the previous study, we are developing a Drupal module that extends the functionality of SimpleTest core module. Our module, SmarTest, integrates our testing techniques and provides the Drupal developer with a dashboard with statistics about Drupal in real time. This information allows us to guide the testing in our system through faults propensity data in different parts of our code. This is based on a correlation study that relates number of commits made in a module, faults detected in last tests executions, cyclomatic complexity, etc, with the faults propensity of the module. Also, in this session, we would make a demo of the SmarTest module to show the possibilities of our tool to the Drupal community and, of course, to get feedback from Drupal professionals to improve SmarTest module.
Now, we are submitting the SmarTest module as a Drupal sandbox project.
In the paper previously mentioned, we presented a real problem with VISs, the testing process. Testing a VIS such as a Drupal system is really challenging due to the potentially huge number of configurations under test. This makes exhaustive testing of a VIS infeasible, that is, testing every single configuration is too expensive in general. In order to address this problem, a variety of testing techniques has been reported to reduce the cost of testing in the presence of variability. In particular, we propose new testing techniques to accelerate the detection of faults in Drupal and other similar systems, providing faster feedback to the testers and reducing debugging efforts.
Based on the previous study, we are developing a Drupal module that extends the functionality of SimpleTest core module. Our module, SmarTest, integrates our testing techniques and provides the Drupal developer with a dashboard with statistics about Drupal in real time. This information allows us to guide the testing in our system through faults propensity data in different parts of our code. This is based on a correlation study that relates number of commits made in a module, faults detected in last tests executions, cyclomatic complexity, etc, with the faults propensity of the module. Also, in this session, we would make a demo of the SmarTest module to show the possibilities of our tool to the Drupal community and, of course, to get feedback from Drupal professionals to improve SmarTest module.
Now, we are submitting the SmarTest module as a Drupal sandbox project.