Should we reject papers with bogus artifacts?
Eric | May 19, 2016Yesterday I blogged about the accepted artifacts at ISSTA. I think it is worthwhile noting that out of the ten papers that got accepted and which had artifacts submitted there seven for which the artifacts checked out. That is a good thing!
What worries me, however, are the three papers for which Artifact evaluation failed. For those three papers, we were largely unable to reproduce their results, and yet the papers made it into the program. Moreover, for two of those three, the fact that they failed artifact evaluation was already known before the PC meeting, i.e., there would have been a chance to reject them.
The reason for why the PC did not is that only positive reviews were taken into account this time, in order not to discourage people from submitting artifacts in the future. We as a community should really think about whether we cannot find a way to make artifact evaluation the default so that people have no other chance than to submit all the evidence they have to back up their claims.
Cross-posted from Secure Software Engineering
