The RELM testing region and collection region are precisely defined by the following lists of cells:
These files list the center point of each cell, and the cells are 0.1° by 0.1°. One can obtain a very good (and usually exact) approximation of the earthquakes in the RELM testing region by using the search string "polygon=43,-125.2,43,-119,39.4,-119,35.7,-114,34.3,-113.1,32.9,-113.5,32.2,-113.6,31.7,-114.5,31.5,-117.1,31.9,-117.9,32.8,-118.4,33.7,-121,34.2,-121.6,37.7,-123.8,40.2,-125.4,40.5,-125.4,43,-125.2" on the ANSS catalog webpage (http://www.quake.geo.berkeley.edu/anss/catalog-search.html). The corresponding collection region search string is "polygon=43.5,-125.7,43.5,-118.5,39.7,-118.5,36.1,-113.6,34.6,-112.6,34.3,-112.6,32.7,-113.1,31.8,-113.2,31.2,-114.5,31.0,-117.1,31.1,-117.4,31.5,-118.3,32.4,-118.8,33.3,-121.3,34.0,-122.0,37.5,-124.3,40.0,-125.9,40.5,-125.9,43.0,-125.7,43.5,-125.7".
We here present further quantitative details of the RELM experiment analysis. Tables S1 (mainshock) and S2 (mainshock+aftershock) indicate each forecast's absolute rate in the bins where target earthquakes occurred and elsewhere (i.e., the bins where no target earthquakes occurred). Tables S3 (mainshock) and S4 (mainshock+aftershock) include the p-values associated with each consistency test. Tables S5 (mainshock) and S6 (mainshock+aftershock) include the rate-corrected average information gains with their associated uncertainty and the p-values from the T- and W-tests.
Table S1. For mainshock forecasts, rates where earthquakes occurred. Red cells indicate that the worst forecast for the target earthquake(s) in this bin (i.e., had the lowest rate), and green cells indicate the best forecast. Earthquake numbers match those used in Table 1 from the article
Table S2. Same as Table S2 for mainshock+aftershock forecasts
Table S3. For mainshock forecasts, p-values associated with each consistency test, based on 1,000,000 simulations. N-test results are equivalent to δ1 and δ2 from Zechar, Gerstenberger, and Rhoades (2010)
Table S4. Same as Table S3 for mainshock+aftershock forecasts
Table S5. For mainshock forecasts, results of comparison tests for each pair of forecasts. In each cell, the first line is the rate-corrected average information gain for the models listed in the corresponding row and column (positive value favors the row model, negative value favors the column model), following Rhoades et al. (2011). The second line in each cell contains the p-values from T- and W-tests.
Table S6. Same as Table S5 for mainshock+aftershock forecasts
We here present the results of the experiment with the uncorrected forecasts submitted by Ebel et al. Figure S1 shows the space-magnitude component of each corrected forecast. In Figure S2 we present the results of the RELM consistency tests: the N-, L-, S-, and M-tests. In Figure S3 we present the results of the comparison tests.
Figure S1. Map-view of Ebel et al's uncorrected mainshock and corrected mainshock+aftershock forecast; the color of each latitude-longitude cell represents the forecast rate obtained by summing over all magnitude bins within the spatial cell. Overlaid square rings represent the observed target earthquakes, the color represents the earthquake occurrence time.
Figure S2. Results of consistency tests for Ebel et al.'s uncorrected mainshock and mainshock+aftershock forecasts. Red points indicate that the forecast failed the test and green points indicate that the forecast passed the test. Horizontal black lines delimit the 95% confidence “pass” region. a) Results of the two-sided N-test, where the the observed number of target earthquakes is indicated by the filled circle. b-d) Results of the one-sided L (S,M)-test; circles represent the observed space-rate-magnitude (space, magnitude) log-likelihood for each forecast and the horizontal black lines delineate the top 95% of log-likelihoods from simulated catalogs that are consistent with the forecast.
Figure S3. Results of comparison tests for Ebel et al.'s uncorrected mainshock and mainshock+aftershock forecasts. For each pairwise comparison, the arrow points to the favored forecast (i.e., the forecast with a positive rate-corrected average information gain per earthquake). Green arrows indicate that the difference between the forecasts is statistically significant with at least 95% confidence in both the T-test and the W-test. The number in each cell indicates the number of target earthquakes used in the comparison (i.e., the number of target earthquakes that occurred in regions where both forecasts were unmasked).
After the publications of Schorlemmer, Zechar et al. (2010) and Zechar, Gerstenberger, and Rhoades (2010) indicated problems with the Holliday et al. forecast, the authors of the Holliday et al. forecast claimed that their forecast had been misinterpreted--that the forecast was intended as a mainshock+aftershock forecast and the masking should be ignored (D. Turcotte, written communication 2012). (This claim happened despite the fact that Schorlemmer et al., during the drafting of their article, solicited feedback from the participating modelers and received no such complaint from Holliday et al.) To address this contention, in the main article we present the results of the experiment with the Holliday et al. forecast treated as a mainshock+aftershock forecast ignoring masking. Here, we present the results honoring the masking. In Figure S4 we present the results of the RELM consistency tests: the N-, L-, S-, and M-tests. In Figure S5 we present the results of the comparison tests.
Figure S4. Results of consistency tests for Holliday et al.'s forecast with masking. Red points indicate that the forecast failed the test and green points indicate that the forecast passed the test. Horizontal black lines delimit the 95% confidence “pass” region. a) Results of the two-sided N-test, where the the observed number of target earthquakes is indicated by the filled circle. b-d) Results of the one-sided L (S,M)-test; circles represent the observed space-rate-magnitude (space, magnitude) log-likelihood for each forecast and the horizontal black lines delineate the top 95% of log-likelihoods from simulated catalogs that are consistent with the forecast.
Figure S5. Results of comparison tests for Holliday et al.'s forecast with masking. For each pairwise comparison, the arrow points to the favored forecast (i.e., the forecast with a positive rate-corrected average information gain per earthquake). Green arrows indicate that the difference between the forecasts is statistically significant with at least 95% confidence in both the T-test and the W-test. The number in each cell indicates the number of target earthquakes used in the comparison (i.e., the number of target earthquakes that occurred in regions where both forecasts were unmasked).
Zechar, J. D., M. C. Gerstenberger, and D. A. Rhoades (2010). Likelihood-based tests for evaluating space-rate-magnitude earthquake forecasts, Bull. Seismol. Soc. Am. 100, 1184–1195, doi:10.1785/0120090192.
Schorlemmer, D., J. D. Zechar, M. J. Werner, E. H. Field, D. D. Jackson, and T. H. Jordan (2010). First results of the Regional Earthquake Likelihood Models experiment, Pure Appl. Geophys. 167, 859–876, doi: 10.1007/s00024-010-0081-5.
Rhoades, D. A., D. Schorlemmer, M. C. Gerstenberger, A. Christophersen, J. D. Zechar, and M. Imoto (2011). Efficient testing of earthquake forecasting models, Acta Geophys. 59, 728–747, doi: 10.2478/ s11600-011-0013-5.
[ Back ]
