Reproducing results from the AStat paper

We provide a single script for reproducing all results and figures from our AStat paper.

An easy way to install and run the script is using pipenv. First, run the command pipenv install 'gues[repro_astat]==0.3.0' in a terminal, in a directory where you want the results to be saved. This will install the version of the gues package used to run the experiments as well as compatibale versions of all dependencies for plotting and analysis.

The entire script (which is parallelized to take about 30 hours and needs 32GB RAM, and includes generating lots of synthetic data to run GrUES on) can then be run with pipenv run reproduce_astat. The script can alternatively be given arguments to only reproduce certain plots or to download the raw experimental results from which plots can be generated (taking just a few minutes, depending on download speeds, and requiring much less RAM). For example, pipenv run reproduce_astat 'download' 'tri_n,uni' will dowload the experimental results and generate the plots for the experiment with triangle prior \(\Delta_s^n\) shown in Figure 10 and Table 2, as well as those with the uniform prior shown in Figure 11 of the paper.

More generally, the first argument to the script can be 'fresh', which generates the synthetic data and runs GrUES and produces the plots, or 'download' which downloads the experimental results and produces the plots, or 'redo_plots' which generates plots from existing local results (which need to have been previously generated or downloaded). The second argument can be 'all' to recreate all figures from the paper, or a comma separated list with some subset of 'hist,tri_n,uni,n_8,n_10,complexity', which respectively correspond to Section 6.2.1, First–Fourth experimental settings in Section 6.2.2, and Section 6.2.3 from the paper. Running the script without arguments is equivalent to pipenv run reproduce_astat 'fresh' 'all'.