ACM SIGMOD City, Country, Year
sigmod pods logo

SIGMOD Conference Experimental Repeatability Requirements

SIGMOD 2008 was the first database conference that proposed testing the code associated to conference submissions against the data sets used by the authors, to test the repeatability of the experiments presented in the submitted papers. A detailed report on this initiative has been published in ACM SIGMOD Record, 37(1):39-45, March 2008.

The experience has been continued in a slightly modified form in conjunction with the SIGMOD 2009 conference. A report on this effort will soon appear in the SIGMOD Record.

The repeatability and workability evaluation in conjunction with SIGMOD 2010 will continue along the lines of the 2009 edition, with some improvements related to the procedure.

The goal

On a voluntary basis, authors of accepted SIGMOD 2010 papers can provide their code/binaries, experimental setups and data to be tested for

by a repeatability/workability committee (to be announced), under the responsibility of the repeatability/workability editors-in-chief (RWE in short). The RWE are Ioana Manolescu and Stefan Manegold.

The procedure

Authors of accepted papers will be contacted by e-mail as soon as acceptance is determined and invited to submit

Submissions will be handled using a Repeatability/Workability Conference Management Tool (RWCMT, in short).

Past experience has demonstrated that repeatability and workability can greatly benefit from the availability of authors to interact with the repeatability reviewer, and help solve minor issues related to the installation, configuration, and usage of the code. Thus, the electronic tool will enable repeated interactions between the reviewers and the authors in the style of a message board. Thus, the 2010 process will involve such interaction, too.

For reference, the instructions sent to the authors in 2009 can be found here.

The RWE will designate for each submission:

The first and second reviewer will interact until they are both satisfied with the terms of the report. They will both sign the report. If there is disagreement that the reviewers cannot work out, the RWE have the final say. They may propose alternative wording for the report, more tests, and/or endorse responsibility together with one reviewer, if the other cannot agree with the chosen wording (and thus is unwilling to sign it).

During the evaluation, the first reviewer and the authors interact via the RWCMT. The second reviewer and the RWE may also participate to the discussion. The recommendation is that the first reviewer is left alone with the authors during the first 2/3rds of the reviewing period, to avoid confusion. The RWCMT documents all interaction between the reviewers and the authors.

The identity of the reviewers is hidden during the evaluation process, but obviously will be revealed afterwards when the reviewers sign their report.

The output

The final RW report will include

The final RW report may be published in the ACM PubZone, assuming the authors agree to this.

Code archiving

Participating in the repeatability/workability evaluation does not imply that the code will be archived for everyone to use subsequently. Pending authors' agreement, the code could be uploaded in the SIGMOD PubZone.