What is the Summer Watch Series?
The Summer Watch Series concept originated in June 2021 at the University of Minnesota and was proposed to CARMA by Elizabeth Campbell, Betty Zhou, and Chris Winchester. Their goal was to informally convene groups of those interested in research methods to learn more about topics of interest and build connections.
Based on their success, CARMA will expand this Series in August 2025 to include additional sessions and involve more schools. The CARMA Methods Review Series – Summer 2025 will be open to participants from CARMA’s Institutional Membership and Affiliate Programs, as well as non-members who register for CARMA’s Limited Access Pass.
The Methods Review Watch Series will take place on four dates during Summer 2025, with both morning and afternoon sessions, offering live viewing of eight CARMA Webcast Lectures. Scheduled dates are August 5, 7, 12, and 14, selected to avoid conflicts with other CARMA summer programs, RMD events, and the AOM Conference.
Lectures have been selected from the CARMA Video Library to support those seeking a review of key research methods topics—particularly helpful for those preparing for comprehensive exams. For each of the eight sessions, background readings will be provided, the lecture will be streamed live by CARMA, and a Q&A session with a Discussant will follow.
A Methods Review Watch Series Discussion Board will also be available to encourage interaction among participants throughout the Series.
Summer 2025 Watch Series Schedule
Experience sampling (ESM) is a well-known and widely used study design aimed primarily at examining within-individual covariation of transient phenomena utilizing repeated measures. ESM studies are increasingly popular among researchers, however there are a number of important considerations and nuances associated with these methods. While many papers have been published and talks given on advanced issues pertaining to ESM studies, what remains largely undiscussed are some of the basic operational and tactical decisions that must be made when designing and running an ESM study. Thus, the purpose of this talk is to walk through some of the basics of these studies and demystify the processes in service of helping scholars to collect data that captures their focal phenomenon with a high degree of accuracy and validity.
In this presentation, Dr. Justin DeSimone explores the concept of “dirty data” in survey research—responses that are construct-irrelevant due to low participant effort. He provides a conceptual and practical framework for understanding, detecting, and managing such data, distinguishing it from related constructs like response sets and outliers. The talk outlines two primary types of dirty data: random responding and straightlining, discussing their different effects on psychometric properties like reliability, validity, and factor structures. DeSimone also presents empirical evidence demonstrating how even a small proportion of dirty data can significantly bias results.
The presentation offers a range of preventative strategies to discourage dirty data, including motivational messaging, reward-based incentives, and adaptive “living surveys.” It then reviews detection techniques, such as instructed response items, bogus items, response time analysis, and consistency indices (e.g., long-string and psychometric synonym/antonym methods), emphasizing the importance of using multiple and appropriate methods. Dr. DeSimone further cautions against post hoc data screening and underscores the need for transparency, ethical rigor, and preplanned protocols in survey design.
Chair: Zhiyan Wu, Erasmus University
This presentation challenges the prevailing norms in strategy and management research that emphasize either airtight identification or novel theoretical contributions in isolation. Instead, it proposes a shift in focus toward building causal understanding through a cumulative body of research. Drawing from empirical examples and methodological critiques, J. Myles Shaver argues that credible causal claims—essential for informing managerial decision-making—require sustained, collective effort rather than being resolved in a single study. He outlines how current research norms often hinder progress and offers practical guidelines for designing studies that advance causal identification incrementally. Emphasizing “research design” over “data and methods,” this talk calls for changes in editorial standards and a broader acceptance of studies that reaffirm or refine existing knowledge. Ultimately, it’s a call to rethink how we generate useful, credible insights about the complex organizations we study.
This presentation examines how artificial intelligence (AI) can be strategically leveraged to support and enhance the research process in academic and applied settings. It highlights innovative applications of AI in graduate education, where students engage in structured self-learning through a combination of traditional materials and AI-based tools like ChatGPT. These tools not only facilitate conceptual understanding in complex subjects such as structural equation modeling but also promote active learning and engagement. Beyond education, the presentation explores how AI can aid in the research pipeline by automating routine tasks such as extracting and comparing citations, recommending publication outlets, and mapping journal relevance. Additionally, AI is shown to play a powerful role in theoretical development. By analyzing large-scale unstructured data—such as online discourse and official documents—researchers use techniques like Structural Topic Modeling and Sentence-BERT to surface patterns that inform novel theoretical insights, especially in complex social contexts. Finally, the application of interpretable machine learning and multi-objective optimization is presented as a means to investigate multifaceted organizational goals. Tools such as permutation feature importance, partial dependence plots, and Pareto-optimal weighting allow for nuanced exploration of how different predictors influence outcomes like job performance, diversity, retention, and well-being. Together, these approaches illustrate the diverse and expanding role of AI in advancing both the efficiency and depth of academic research.
The omission of relevant explanatory variables in a regression model generally causes its estimators to be biased. This issue is referred to as omitted variable bias (OVB) and is recognized as one of the primary sources of endogeneity. In turn, the concern of OVB is often a key motivating reason for adopting instrumental variable techniques. These techniques typically involve a two-step procedure that constructs a version of the independent variable that does not feature variance due to the omitted variable. While these techniques can help alleviate the OVB concern, they also have critical assumptions that must be met regarding the instrumental variables employed (i.e., relevance and exogeneity). Even when these assumptions are met, instrumental variable techniques are often less efficient than ordinary least square regression. Recently, the impact threshold of a confounding variable (ITCV) has been introduced in organization research. The ITCV can be used to understand whether a statistical inference is changed because of the potential for an omitted variable. In this talk, the issue of OVB will be formally defined, and both instrumental variable techniques and the ITCV will be discussed as ways to help alleviate this concern.
This presentation provides an overview of multilevel modeling approaches for analyzing longitudinal and experience sampling method (ESM) data. It outlines key distinctions between modeling data collected within individuals over time and data nested within groups, highlighting the importance of accounting for temporal ordering and dependencies. The discussion includes considerations for determining when to treat model parameters as fixed or random effects, emphasizing the impact these decisions have on statistical and theoretical interpretations. Attention is given to the foundational role of theory, construct definition, and research design in shaping effective multilevel analyses. The value of modeling dependence is underscored as a means of generating deeper insights, refining existing theories, and enhancing the quality of scientific conclusions. Ultimately, the presentation advocates for careful alignment throughout the modeling process to ensure meaningful and accurate results.
Chair: Aurelius Sindila, Hong Kong Polytechnic University
Meta-analysis is a useful method to summarize and critically assess the state of knowledge on a research topic. It can also help researchers investigate boundary conditions of theoretical relationships and further test, refine, and build theories. This webinar focuses on the publication process for meta-analyses from an editor’s perspective and ways to maximize the contributions a meta-analysis makes to the field. Dr. Su will discuss characteristics of high-impact meta-analyses that have been published in top-tier journals, share common pitfalls in meta-analysis submissions and reasons for rejection, and highlight recent trends in published meta-analyses.
Chair: Dr. Borbala Csillag, Oregon State
Facets of the research process are often taught and discussed in isolation. I will offer an integrative perspective. I will argue that research should originate from an important question that matters to scholars and/or practitioners – that drives impact. Then, the facets of the research process including sampling, research design, measurement, and analysis need to be integrated and aligned with the research question for optimal value. These facets drive inferences concerning external, internal, construct, and statistical conclusion, respectively, as elements of a unified validity framework. I will then differentiate and illustrate different types of multi-methods and mixed-methods designs suitable for different purposes.
