Open Science and Replication Research - Session III

Question 1 of 22

What is the epistemological justification for replication within critical rationalism?

Knowledge progresses by attempting to falsify, rather than confirm, hypotheses—replication plays a central role in this process.
Truth is best pursued through accumulating positive evidence.
Scientific claims gain legitimacy through singular, high-powered studies.
Inductive generalization is sufficient for theory-building, making replication optional.

Question 2 of 22

Which of the following accurately distinguishes constructive from exact replications based on the workshop’s definitions?

Constructive replications deliberately introduce one focused design change; exact replications aim to match the original as closely as possible.
Constructive replications attempt to correct errors in the original data; exact replications do not.
Exact replications vary the causal model while constructive replications retain it.
Only constructive replications require access to the original dataset.

Question 3 of 22

Why is the Region of Practical Equivalence (ROPE) a meaningful addition to the interpretation of replication studies?

It focuses on whether the replication's CI lies entirely within a negligible effect range, refining interpretations of non-significant results.
It provides a frequentist alternative to p-values.
It allows researchers to quantify effect heterogeneity across subgroups.
It adjusts for sampling bias in Bayesian posterior estimates.

Question 4 of 22

In the context of replication studies, which of the following most accurately challenges the use of p-value thresholds (e.g., p < 0.05) as the sole basis for inference?

Statistical significance thresholds promote dichotomous thinking.
Larger samples eliminate all Type I errors.
Statistical significance (p-values) are not affected by sample size.
P-values are not applicable to replication models with control variables.

Question 5 of 22

Which of the following studies would you recommend for a replication project, assuming the criteria citations, transparency, author cooperation, and access to the original data are equally important?

Moderately cited, very transparent, cooperative original authors, access to original data.
Highly-cited, moderately transparent, unresponsive original authors, no access to original data.
Highly cited, moderately transparent, cooperating original authors, no access to original data.
Highly cited, very transparent, unresponsive original authors, no access to original data.

Question 6 of 22

An original study (n= 400) reported an effect of b = 0.15 [0.01, 0.28] and the exact replication (n = 400) reported an effect of b = 0.13 [-0.01, 0.26]. Which of the following do you consider the correct interpretation of this replication result?

The lack of statistical significance in the replication alone is insufficient to invalidate the original study’s findings. Instead, additional evaluations considering the overlap of the confidence intervals and effect size difference are considered best practices.
The replication results show conclusively that the original study reported false-positive findings and should be ignored.
Because the replication study produced no statistically significant results, it should be ignored as it cannot provide any additional information about the validity of the original study’s findings.
The smaller effect size in the replication clearly indicates that the original study likely engaged in p-hacking and considering the similarity in sample size, we can be rule out sampling error out as an explanation.

Question 7 of 22

What is the key advantage of first reproducing the results of the original study using the original study’s data?

The reproduction of the original study's results ensures that the analyses during the replication are identical to the analyses performed during the original study.
Original authors are more likely to support a replication effort if they know it involves a reproduction of their results.
Most journals will desk reject studies that do not engage in such reproductions, and consequently, obtaining the original study’s data has become a necessity for engaging in publishable replication research.
There are no advantages to conducting a reproduction first, and replication researchers should not engage in such efforts.

Question 8 of 22

Suppose a replication study yields a statistically insignificant result, and the 90% CI falls entirely within the original study’s ROPE. What is the most appropriate inference?

The null effect is supported; the original may have overstated the effect.
The replication fails and confirms the original effect.
The replication is flawed due to an underpowered design.
The replication should be disregarded due to low sample size.

Question 9 of 22

What is a key limitation of relying solely on lack of confidence interval overlap for interpreting replication results?

It simplifies replication interpretation to a binary conclusion.
It requires Bayesian assumptions to be valid.
It ignores publication bias.
It is robust only when meta-analytic weights are applied.

Question 10 of 22

How does Hierarchical Bayesian Meta-Analysis improve upon comparing individual replication results?

It pools data while accounting for systematic variation across studies.
It eliminates the need for replication data.
It focuses on the statistically significant effects of the compared studies.
It reduces variance by adjusting all effects toward the mean.

Question 11 of 22

Which of the following is a reason replication studies face resistance in publication, despite their scientific importance?

Replications are seen as lacking originality and offering only incremental contributions.
Editors prioritize methodological transparency.
Replications often produce novel theoretical constructs.
Replication studies are mostly qualitative and thus less generalizable.

Question 12 of 22

Based on the discussed replication framework and ARIM’s approach, what is a main educational benefit of replication studies for doctoral students?

Learning general research skills like design, data collection, and publication through a high-stakes but scaffolded process.
Gaining experience with Bayesian software and analyses.
Creativity in identifying worthy research topics and phenomena.
Practicing ethical reviews of original study authors.

Question 13 of 22

Why might a meta-analysis of multiple replication studies be preferred over relying on a single, well-powered replication?

It improves generalizability by aggregating across variations in design and context.
It eliminates researcher bias in the original study.
It provides a narrative explanation for effect sizes.
It allows authors to avoid dealing with conflicting results.

Question 14 of 22

Which of the following most accurately explains why p-hacking undermines the credibility of original findings?

It typically overestimates effect sizes by capitalizing on random noise.
It limits the use of Bayesian analyses.
It prevents the use of confidence intervals.
It reflects legitimate exploratory testing practices.

Question 15 of 22

If a replication study reproduces the effect direction but not its statistical significance, what is the most defensible conclusion?

The effect may still exist, but replication design limitations or sampling variability prevent strong conclusions.
The theory is invalid and should be rejected.
The original study was p-hacked.
The replication must be flawed because the effect direction was the same.

Question 16 of 22

Which of the following best illustrates a constructive replication according to replication typology discussed in the course?

Replicating a study collecting data from a decisively different population while retaining original design and methods.
Replicating a study at a later point in time with no changes in design, execution, and analytical methods.
Using new analysis approaches to reproduce the original effect in the original data.
Sharing publicly the replication raw data to enable future constructive replications.

Question 17 of 22

What makes hierarchical Bayesian regression a powerful tool in replication analysis?

It integrates evidence from multiple studies while modeling between-study heterogeneity.
It avoids the need for confidence intervals altogether.
It allows researchers to omit original study data.
It only evaluates whether replication results match original p-values.

Question 18 of 22

What is a practical implication of the “cliff effects” problem associated with p-values?

They can cause small data differences around p = .05 lead to large swings in significance interpretation.
They increase interpretability of small sample effects.
They reduce the need for replication.
They increase the reliability of p-value interpretation.

Question 19 of 22

When is pre-registration considered a best practice in replication studies?

Before executing constructive replications.
When replications expect power problems.
When transparency increases the risk of post hoc rationalization.
To speed up the execution of the entire replication project.

Question 20 of 22

How does Bayesian model updating work when analyzing in replication data?

It uses replication data to update prior beliefs base on the original study and refine effect distribution estimates.
It reduces the need for theoretical frameworks.
It assumes that the null hypothesis is true to estimate effect size and effect uncertainty.
It compares the replication data to normal distributions to estimate updated effect .

Question 21 of 22

According to the course, which of the following is least likely to be a barrier to the broader adoption of replication studies in management research?

Methodological arguments questioning the value of replication for developing evidence-based theories and theoretical.
Lack of clear publication outlets.
Underdeveloped replication methodologies.
Insufficient training in conducting replication work.

Question 22 of 22

What distinguishes incremental theory-building replications from purely validation-focused replications?

They aim to explore moderators, boundary conditions, and mediators that extend the theory.
They are more likely to challenge the original authors directly.
They are uninterested in the effect direction or magnitude.
They rely solely on confidence intervals more than effect size comparisons.

Name	You must fill out this field.
Email	You must fill out this field.

Open Science and Replication Research – Session III

Quiz Summary

Information

Results

Results

Categories

1. Question

2. Question

3. Question

4. Question

5. Question

6. Question

7. Question

8. Question

9. Question

10. Question

11. Question

12. Question

13. Question

14. Question

15. Question

16. Question

17. Question

18. Question

19. Question

20. Question

21. Question

22. Question