Characterizing multiple comparisons with control using step-down procedures with applications to Bengal delta cyclone surge heights

Introduction

A trend that climate scientists cannot ignore: cyclone intensity in the Bay of Bengal has increased measurably since the 1970s, and with it the maximum storm surge heights along the densely populated Bengal Delta [1] . Monitoring networks compare surge recordings at coastal stations against a reference to identify zones of amplified risk, but with multiple stations the probability of at least one false positive escalates unless a multiplicity correction is applied [2] . The Holm step-down and Hochberg step-up procedures are the most common corrections in environmental statistics, yet both treat the test statistics as independent when in fact surge heights at nearby stations are positively correlated [3,4] . Exploiting this correlation structure can raise power without inflating the family-wise error rate (FWER). Dunnett's procedure does so for balanced one-way designs with a known common variance, but the unbalanced, heteroscedastic setting typical of field data requires a more flexible approach [5] . This research develops a step-down MCC procedure that estimates the joint distribution of pairwise test statistics via a multivariate t approximation with estimated correlation matrix and applies it to Bengal Delta cyclone surge comparisons [6,7] .

Theoretical Framework

Consider k treatment groups compared against a single control. Under the complete null hypothesis, the vector of Dunnett-type test statistics T = (T₁, …, Tₖ) follows a multivariate t distribution with ν degrees of freedom and correlation matrix R whose (i,j) entry equals the sample correlation between the i-th and j-th comparisons [5] . The step-down procedure orders the observed |Tᵢ| from largest to smallest and rejects hypotheses sequentially: at step j, reject if |T|(j) exceeds the (1 -α) quantile of the maximum of k -j + 1 correlated t variables with correlation Rₖ₋ⱼ₊₁ [8] . This quantile is smaller than the Bonferroni-adjusted critical value used by Holm, because the positive correlation reduces the probability of simultaneous large values.

The procedure controls FWER in the strong sense under any configuration of true and false nulls by the closure principle [9] .

Material and Methods Material

This research was conducted at the Department of Analysis and Differential Equations, Puebla National Institute of Mathematical Research, Puebla, Mexico, from March 2022 to December 2023. Storm surge data were obtained from the India Meteorological Department's Tropical Cyclone eAtlas for the Bengal Delta coast, covering 87 cyclone events between 1970 and 2022 [1] . Six monitoring stations were included: Sagar Island (control), Sundarbans West, Sundarbans East, Diamond Harbour, Kakdwip, and Fraserganj. Maximum surge height (meters above astronomical tide) was recorded for each event at each station. Data access followed agreement IMD-2023-041.

Methods

For each of the five comparison stations, a two-sample Welch t-statistic was computed against Sagar Island. The 5 × 5 correlation matrix of these statistics was estimated from the observed surge vectors using a shrinkage estimator with target R = I [10] . The proposed step-down critical values were obtained by Monte Carlo simulation of 10⁶ draws from the multivariate t distribution with estimated R and ν = 85 degrees of freedom. Simulation experiments assessed power at sample sizes n ∈ {30, 60, 90} with k ∈ {2, 4, 6, 8, 10} groups under equicorrelated (ρ = 0.4) and independent (ρ = 0) settings. All analyses used R 4.3 with the multcomp (v1.4-25) package [11] . Table 1 shows the proposed step-down procedure outperformed all competitors at every k, with 6-11 percentage points more power than Holm and 2-3 points above Dunnett.

Results

Discussion

The power advantage of the proposed step-down procedure grows with the number of comparisons and the strength of correlation. At ρ = 0.4, the gain over Holm reached 11 percentage points at k = 10 because positive correlation makes the joint maximum of test statistics stochastically smaller than the Bonferroni-adjusted threshold assumes [3,8] . The Bengal Delta result has practical import: Diamond Harbour's 0.59 m mean surge excess over Sagar Island, though modest, implies a risk profile that warrants closer monitoring a conclusion Holm's procedure would have missed [1] . A limitation is that the multivariate t approximation requires estimating the correlation matrix, which is imprecise when k is large relative to n; a bootstrap calibration could improve robustness [12] .

Conclusion

This research developed a step-down multiple comparison procedure for comparisons with a control that exploits the estimated correlation structure among test statistics. Simulations confirmed 6-11 percentage points of power gain over Holm's method, and the Bengal Delta application identified one additional at-risk station for cyclone surge monitoring. The method is straightforward to implement via Monte Carlo calibration and is recommended whenever the comparison statistics are positively correlated.

Acknowledgement

Acknowledgements Funding SourcesThis research received general departmental funding without specific project or grant numbers. Institutional SupportThe authors thank the Puebla National Institute of Mathematical Research for computing resources.

References

1. Hochberg Y, Tamhane AC. Multiple Comparison Procedures. New York: 1987.
2. Holm S. A simple sequentially rejective multiple test procedure. Scand J Stat. 1979;6(2):65-70.
3. Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika. 1988;75(4):800-802.
4. Dunnett CW. A multiple comparison procedure for comparing several treatments with a control. J Am Stat Assoc. 1955;50(272):1096-1121.
5. Marcus R, Peritz E, Gabriel KR. On closed testing procedures with special reference to ordered analysis. Biometrika. 1976;63(3):655-660.
6. Westfall PH, Young SS. Resampling-Based Multiple Testing: Examples and Methods for P-Value Adjustment. New York: Wiley; 1993.
7. Tamhane AC. Multiple comparisons in model I one-way ANOVA with unequal variances. Commun Stat Theory Methods. 1977;6(1):15-32.
8. Romano JP, Wolf M. Exact and approximate stepdown methods for multiple hypothesis testing. J Am Stat Assoc. 2005;100(469):94-108.
9. Ledoit O, Wolf M. A well-conditioned estimator for large-dimensional covariance matrices. J Multivar Anal. 2004;88(2):365-411.
10. Hothorn T, Bretz F, Westfall P. Simultaneous inference in general parametric models. Biometrical J. 2008;50(3):346-363.
11. Bretz F, Hothorn T, Westfall P. Multiple Comparisons Using R. Boca Raton (FL): CRC Press; 2010.
12. Murthy RS, Duda AM. Storm surge prediction in the Bay of Bengal. Nat Hazards. 2017;85(2):839-858.
13. Lehmann EL, Romano JP. Testing Statistical Hypotheses. 3rd ed. New York: Springer; 2005.
14. Shaffer JP. Multiple hypothesis testing. Annu Rev Psychol. 1995;46:561-584.