A preregistered test that outcome-permuted data also pass: mathematical coupling in a pan-cancer analysis of proliferation adjustment in prognostic gene expression studies

Source abstract

Abstract Background Preregistration fixes an analysis before the data are examined, but not whether the registered test can fail. I report a case from prognostic gene expression research, where, in a survey of hepatocellular carcinoma studies, no survival model was adjusted for tumor proliferation. Methods For every expressed gene, two Cox models for overall survival were fitted, without and with a standardized proliferation score. A registered plan predicted that b, the slope across genes of the log shift in hazard ratio on correlation with the score, would track L, the log hazard ratio of the score, across cohorts. The test used 20 public cohorts (4,986 patients, 14 cancer types). Post hoc, I permuted survival within all twenty cohorts and refitted the registered test 100 times; separately, in the observed data, I counted genes passing Benjamini-Hochberg control under three covariate sets. Results All three registered hypotheses were confirmed: coefficient − 0.84 (95% CI − 0.90 to − 0.78), intercept − 0.009 (− 0.030 to 0.012), sign rule met in 20 of 20 cohorts. In 100 null replicates, in which nothing is prognostic, all three criteria were met in 92 (95% CI 85 to 96), below the 95% fixed in advance, but too often for the confirmation to support confounding. The observed coefficient lay at the upper edge of the null distribution. In 11 observed cohorts, 22,315 genes passing with age and sex fell to 15,988 when stage was added and to 7,638 when the score was added; stage alone removed most genes in some cohorts. Conclusions A preregistered hypothesis can be confirmed largely by construction. Running the registered pipeline on outcome-permuted data, and registering the values its criteria take, would have exposed this before any result was seen. Lists of prognostic genes depend on the covariate set, which should be stated and varied.