Sensitivity Analysis (Power Analysis)

[DominiqueMakowski]

To follow up on the discussion in #2

Although the end goal is to use a Bayesian approach, which is fundamentally at odds with what people typically mean by power analysis (see this thread), I agree that it is useful in practice to have some ballpark estimate on how many participants we should collect to expect seeing something.

I quickly ran a sensitivity analysis targetting the effect of AI-generated cue on Arousal (our primary response), that is:

• For various hypothetical sample sizes (n), we draw a random new sample with replacement from our (real) data
• We do that several times (i iterations)
• For each iteration, we fit a model¹ (in this case, a linear mixed model that tests the effect of fiction for males only (as we don't have much women for now to do anything meaningful) for erotically relevant stimuli (i.e., images of girls for hetero men)
• We extract the p-value of our effect of interest
• We plot the p-values of all iterations as a function of sample size

Conclusion:

• Basically, n > 100 seems warranted²

Notes:
¹ The model used in this example is a "simple" (i.e., without covariates) frequentist linear mixed model (which likely different that what we will use in the end). This means that this sensitivity analysis is based on different statistics, so it's a bit like potatoes and apples... But hopefully it can be useful nonetheless to gain a (very) rough idea.

To expand on the above note, the main thing to get more reliable estimates is time, as computing the model for many iterations takes quite some time. But even if we had infinite time, and we could fit the exact frequentist equivalent of the Bayesian model, the sensitivity analysis would still be based on a different philosophical approach than Bayesian stats.

² This analsyis is based on the first batch of data, which comes mostly from reddit, which I believe is fairly low quality and noisy data. So things might look better with better stats models and better data (e.g., from laboratory conditions?)

All that to say, this kind of stuff should be taken with a pinch of salt ☺️