Webcast Lecture Series

Evaluating Alternative Data-Generating Mechanisms for Statistical Theories: A Tutorial With Examples From Organizational Research

Is it possible that the results that an organizational researcher gets from a particular statistical model in reality come from an entirely different true model? Researchers frequently treat results from statistical models as facts. However, alternative data-generating mechanisms are a possibility for most, if not all, nonexperimental designs. If the statistical model does not capture the essence of the underlying process, coefficients and estimates do not provide meaningful insights. The presentation discusses four recently proposed criteria and related checks (Lang & Blues, 2024) that are effective in ruling out the possibility of alternative data-generating mechanisms for statistical theories: (1) Theory/subject matter insights on the mechanism, (2) parameter recovery in simulated data, (3) fit to real data, and (4) context. None of these criteria delivers a definitive answer in isolation. However, researchers can typically receive reasonable answers by having a balanced look at the overall picture. The presentation then goes on to illustrate these criteria for several classical examples and popular statistical models in organizational research. The discussion focuses on best practice recommendations.