Multiple Outcomes and Attrition Bounds Holm · sharpened q-values · Lee and Manski bounds

Two questions a reader asks of any trial report. With this many outcomes tested, which results would still stand after allowing for the number of tests? And if more people dropped out of one arm than the other, how far could that alone move the estimate?

Outcomes

The names and p-values above are invented. Enter every outcome that was tested, including the ones that did not reach significance, or the adjustment is too kind.

Adjusted p-values

Method

Bonferroni multiplies each p-value by the number of tests. Holm (1979) sorts them and multiplies the smallest by M, the next by M − 1 and so on, taking a running maximum; both control the chance of even one false rejection. Benjamini and Hochberg (1995) control the expected share of rejections that are false: the adjusted value of the p-value ranked r is the smallest p·M/r at that rank or above. The sharpened q-value (Anderson 2008) uses the two-stage procedure of Benjamini, Krieger and Yekutieli (2006), which first estimates how many hypotheses are true nulls, and reports the smallest level on a grid of 0.001 at which the outcome is rejected. A sharpened q-value can be smaller than the p-value it comes from.

Data

The example is simulated: 600 people in each arm, a true effect of 0.20, and treated people with low outcomes less likely to drop out. The file is read in this page and is not uploaded.

Bounds on the effect

Load data to begin.

Method

The Lee (2009) bounds assume that assignment to treatment only ever moves people in one direction, into response or out of it. The arm with the higher response rate then contains extra respondents, a share p of its respondents equal to the difference in response rates over the higher rate. Trimming that share from the top of its outcomes gives one bound and from the bottom the other. When the trimmed number is not whole, the observation at the cut is given a fractional weight. The Manski (1990) bounds assume nothing about who is lost: every missing outcome is set to the lowest possible value in one arm and the highest in the other, and then the reverse. The interval beside each pair is a percentile bootstrap, resampling people within each arm with their missing outcomes, from the 2.5th percentile of the lower bound to the 97.5th of the upper.

Need help setting out your story, what sets your work apart, the context, the return on investment and the projections? Pinpoint Ventures can help. Write to hello@pinpointventures.in or message on WhatsApp.