Session 1 Panelists:
1. Andrew Gordon, Prolific
About the Panelist:
Andrew Gordon, PhD, is a Staff Research Scientist at Prolific, where he leads the Research Sciences team working at the intersection of data quality, online research methodology, and public opinion. His work focuses on establishing new standards for how online research is designed, executed, and validated. With a PhD in Cognitive Neuroscience from the University of Bristol and postdoctoral experience at UC Davis, Andrew brings a deep background in human behavior and decision-making that informs his current focus on data integrity and methodological rigour in online research settings. His research has been published in journals including Communications of the ACM, Behavior Research Methods, and NeuroImage, and his public opinion work has been covered by The Guardian and The Independent. He is passionate about meta-science and committed to building bridges between academic research and industry.
Presentation Overview:
Despite widespread concern that AI agents are infiltrating online survey samples, the evidence suggests the real threat lies elsewhere. In a pre-registered study recruiting 5,200 respondents across 10 platforms spanning direct panels, hybrid networks, and marketplace aggregators, automated agent detections were almost entirely confined to Amazon Mechanical Turk, and even there, response profiles were more consistent with traditional scripted bots than sophisticated LLM-based agents. The more consequential finding was substantial and systematic variation in human data quality across platform types: direct panels outperformed hybrid platforms, which outperformed marketplace aggregators, across nearly all behavioral measures. This hierarchy was several times larger in magnitude than any AI-related effect, and direct panels proved the most economical option once quality thresholds were applied. The field’s most pressing data quality challenge is not a novel one, and the solutions are closer to hand than the current focus on AI agent detection implies.
2. James Martherus, Morning Consult
About the Panelist:
James Martherus is a Senior Research Scientist at Morning Consult, where he is responsible for data quality, survey weighting, and advanced analytics. He earned his doctorate in Political Science from Vanderbilt University, and his work has been published in Science, Political Behavior, Survey Practice, and more.
Presentation Overview:
AI agents capable of autonomously browsing the web and completing online surveys pose a growing threat to survey data quality. In two related studies, we examine this threat and evaluate strategies for detecting AI-generated responses. The first study (Survey Practice, 2025) uses OpenAI’s Operator to document the strengths and weaknesses of a commercially available agent. We find that several types of survey questions and behavioral patterns reliably identify AI-assisted responses. The second study (under review) extends this work by embedding a JavaScript behavioral monitoring suite in a large-scale omnibus survey, collecting data from human respondents alongside sessions run by Claude and ChatGPT, and evaluating whether interaction patterns like mouse movement, click timing, scrolling, and keystrokes can distinguish agents from humans. Together, the two papers offer a two-layer detection framework.
3. Adrienne Sudberry, Longwood University
About the Panelist:
Dr. Adrienne Sudbury grew up in East Tennessee and earned her Ph.D. in Economics at The University of Tennessee. After graduation, she accepted a position at Longwood University as an Assistant Professor of Economics. She teaches both undergraduate and graduate courses in microeconomics, macroeconomics, and experimental economics. Her research interests include behavioral and public economics with a focus on philanthropy, crowdfunding, and experiments. She currently lives in Virginia with her husband, two children, and two dogs.
Presentation Overview:
Researchers have increasingly pivoted from paper and phone to online survey delivery. However, bots and inattentive participants can lower data quality dramatically compared with in-person delivery. In this paper, we demonstrate to the reader the effectiveness of measures used to screen for bots and inattentive participants. Using our own survey that incorporates current best practices, we illustrate that these measures aided us in detecting both bots and inattentive participants; however, our attention checks eliminated more inattentive respondents than our bot detection questions. Thus, we show that including checks for bots and reader attention is crucial to implementing online surveys and generating high-quality data.
Session 2 Panelists:
1. Thomas Shaw, Virginia Tech University
About the Panelist:
TJ Shaw, M.S. is a doctoral candidate in clinical psychology at Virginia Tech. His research focuses on better understanding the etiology and course of posttraumatic stress disorder and its many comorbidities using psychophysiological data and computational methods. Additionally, he has participated and led studies that aim to better understand the scope of bot infiltrations in research, and how to effectively respond to these intrusions.
Presentation Overview:
“I, ROBOT: Understanding the Prevalence and Impact of Bots in Research” will provide an introductory assessment of research examining the rise of malicious bots infiltrating research studies. Using a case study approach, this talk will provide basic information on the prevalence of bots in research and where they are most commonly found. Additionally, the scope of this problem for researchers will be discussed, along with suggestions for protecting one’s research from bots and educating colleagues and editors about the dangers of bots in research.
2. Gargi Sawhney, Auburn University
About the Panelist:
Gargi Sawhney is an Associate Professor of Industrial-Organizational Psychology at Auburn University and Chief Researcher at ResponsePie. Her research examines occupational stress, employee well-being, research methods, and data quality in online research. For more than three years, she has studied survey fraud, including developing own AI agents that autonomously complete surveys and creating as well as evaluating tools designed to identify AI-assisted, automated, and fraudulent responding. Her research has appeared in journals including the Journal of Vocational Behavior, Human Resource Management, and the Journal of Occupational and Organizational Psychology, among others. She also serves on the editorial boards of Stress and Health and the International Journal of Stress Management.
Presentation Overview:
AI poses a growing challenge to online research through fully autonomous agents as well as assistive tools, including browser extensions, chatbots, and automated form-filling technologies. This presentation examines how modern AI agents perform on commonly used in-survey checks and whether behavioral, device, and technical paradata distinguish them from human participants. The findings illustrate why protecting online data integrity requires a layered approach rather than reliance on any single method.
3. Amber Thompson, University of Utah
About the Panelist:
Amber D. Thompson is a sociologist and Health Service Research fellow studying caregiver and family support, non-medical support, and inequalities/gaps around end-of-life care. Current research interests center around her broad interdisciplinary interests in the process and organization of healthcare in America. She is most interested in how families manage and are affected by chronic/terminal disease care transitions. Her work has been published in journals such as Journal of Aging and Health, Qualitative Health Research, Quality & Quantity, and Innovations in Aging.
Presentation Overview:
This presentation provides a detailed description of how scammers bots and insincere respondents infiltrated a cross-sectional descriptive study on end-of-life doulas, a specific and targeted hard to reach population. The study used an online survey in which 85% of the eligible responses were deemed insincere or bot-generated and dropped from the analytic sample. This study, which utilize the most common remote data collection approaches in social science research, provides evidence of a growing threat to the reliability of data collected using remote methods. Findings demonstrate that insincere and bot responses are: (1) becoming bold in their attempts to infiltrate research studies, (2) more sophisticated in their strategy to “pass” as sincere respondents, and (3) less unlikely to be easily detected by built-in platform security measures. We will discuss how these insincere responses were identified and suspected false data was handled in the data cleaning process. It is highly recommended that researchers not only consider the possibility of insincere responses during the design phase, but expect that this will occur and be ready to address this issue throughout all study phases. On-line methodologies and remote data collection are valuable tools for researchers and should not be abandoned due to the inherent threats to their integrity. Rather, researchers must continually evolve on-line methods in step with technological capabilities and known best practices.