Extreme original data yield extreme decline effects

Abstract

Our meta-analysis showing a dramatic decline effect in ocean acidification (OA) studies on fish behavior [1] was criticized and reanalyzed by Dr. Munday [2]. After applying changes to the dataset in a seemingly biased direction (see Supp. Data 1 at https://osf.io/7spzx/), Munday found a slightly less “extreme” decline effect than the original. Nonetheless, Munday’s reanalyzed decline effect remains one of the strongest examples of this phenomenon in ecology [3], despite the claim that his “reanalysis shows there is not an extreme decline effect in fish ocean acidification studies.”

Why is the decline effect still present in the reanalysis, and why did early studies have such inflated effect sizes? The reason is that many early papers by Munday and colleagues included data that are extreme in and of themselves, and likely nonbiological in origin [4].

Alongside specific comments and revisions to our dataset based on subjective criteria (some we accept, many we do not; Supp. Data 1, Supp. Table 1 at https://osf.io/7spzx/), Munday critiques how we assigned values to means of zero when computing effect sizes. Since biological data rarely have means and variances of precisely zero, treating means of zero in meta-analyses has not been extensively explored. We agree with Munday that a discussion about how to handle mean values of zero in calculations of effect size is warranted, which we elaborate upon in Box 1. Munday’s critique is that our use of small fractional means for percentage and proportional data where the original means and variances are precisely zero “artificially inflates” effect size estimates. But context matters—the “artificially inflated” effect sizes in early studies are derived from initial data that are highly unlikely to begin with. For example, in a paper by Dr. Dixson and colleagues [5], choice-flume experiments measuring time spent in predator chemical cues yielded means for control and OA-exposed fish of precisely 0% and 100%, respectively, each with variances of exactly 0%.