Multiverse/Manylabs Analysis?

[DominiqueMakowski]

The more I think about it the more I find the idea appealing to try out a manylabs/multiverse approach, which would entail the following:

• Starting with the same (cleaned) data, teams run analyses separately (using whatever methods & software you prefer)
• We then qualitatively compare our findings and conclusions, and see how much things vary depending on how we analyze it etc.
• For the paper itself, for clarity and conciseness, we can present the "best" pipeline by taking the bits and pieces that we prefer, but we can mention that on top of the analysis presented in the paper, we also approached the data using different methods which helped us gain confidence in the fact that our results were robust (or not) to various analytical plans (and we can link to a public discussion where we discuss the differences and similarities).
• This thorough discussion and sharing of methodological ideas would allow us to develop a more principled analytic plan that we could use for preregistration (or even registered reports) for follow-up studies

This kind of approach is rather on the bleeding edge of openness and of "new ways" of dealing with replicability issues, so I'd understand if you'd prefer not to be bothered and just do one analysis, but I thought I'd mention that here in case you're interested

What do you think?

@MarcoViola88 @marcosperduti @Antonio8424 @AleAnsani @marmarini

(also, in case you don't receive notifications from these discussions, here's something on power analysis)

[guidocor]

Before making a decision. Do you have any published example of a similar analysis? I find it extremely interesant and good the approach.

[marcosperduti]

It is an interesting approach, but I have some doubts:

1. It seems to me a little bit artificial. I mean, we are collecting data together, you have already a pipeline...but at the end, we'll all made different analyses. We can do it for our fun, but I don't know if it would be convincing for a reviewer.
2. why we want to compare some preferred models with suboptimal models? For example, @DominiqueMakowski is using Bayesian mixed models, and @marcosperduti is using repeated measure ANOVA (in the frequentist framework). In the best scenario, we arrive at the same conclusion, and we present in the paper the best model: the Bayesian mixed models (so having done te two analyses is probably a waste of time). In the worst scenario, we arrive at different conclusions, but we know that the repeated measure ANOVA is a suboptimal approach. Thus we trust more the Bayesian mixed models and report it in the paper (same final of the first scenario).

Why we don't just add the lab (or even the way we collected the data: prolific, university website etc.) that collected the data in the model ? This would reflect the nested nature of our data (conditions are nested in participants that are nested in the country - lab- in which they are recruited). Moreover, this approach would allow to statistically test the possible impact of these variables on our results.

[DominiqueMakowski]

Fair points

Adding the lab as a factor

We can do that, but this requires to have enough data points in all groups

[marcosperduti]

How many data points?

[DominiqueMakowski]