arXiv subjects

N. G. Holmes

Publications and source records attributed to N. G. Holmes.

At least 19 recordsLinked to original sources

arXiv2026

A Framework for Deductive Semantic Content Analysis at Scale in Science Education Using Text Embeddings

Qualitative content analysis of open-ended survey responses is a commonly used research method in science education. However, traditional coding approaches are often time-consuming and prone to inconsistency, especially when applied to large datasets. Existing solutions from Natural Language Processing such as supervised classifiers, topic modeling techniques, and generative large language models have limited applicability in analysis of open-ended survey responses, since they demand extensive labeled data, disrupt established qualitative workflows, and/or yield variable results. In this paper, we introduce a text embedding-based classification framework called Deductive Semantic Content Analysis (DeSCA) that requires only a handful of examples per category to run, is transparent and replicable, and fits well with standard qualitative workflows. When benchmarked against human analysis of a physics education survey consisting of 2899 open-ended responses, the method described by our framework achieves high agreement with expert human coders across ten embeddings models on a simulated exhaustive coding task, using approximately 1-2% of the total dataset for training. The method achieves lower agreement on a complete selective coding task; this performance, however, improves with fine-tuning of the text embedding model, which can be done with a small amount of additional data. We unpack these results in terms of the theoretical assumptions of text embeddings, and further demonstrate how embeddings can be used to audit previously-analyzed datasets for coding consistency. These findings demonstrate that text embedding-assisted coding can flexibly scale to thousands of responses without sacrificing interpretability, opening avenues for deductive qualitative analysis at scale.

cs.CL↗

arXiv2026

CURE-like, not cure-all: Varying broad relevance in experimentation labs produces similar student outcomes

Physics labs that engage students in practices authentic to experimental physics (experimentation-based labs) are being implemented to modernize the undergraduate physics curriculum and broaden participation in physics. Accordingly, prior research has positioned Course-Based Undergraduate Research Experiences (CUREs) as a means to extend the benefits of authentic undergraduate research experiences to more students. However, CUREs are resource-intensive and difficult to implement; a continuous stream of novel research projects adaptable for undergraduate courses is rare. Further, little is known about which specific components of a CURE are crucial to improving student outcomes and which components could be scaled back to improve feasibility for a wider range of class settings. In this study, we aim to isolate the component of broad relevance by running two experimentation-based labs in parallel: one "CURE-like" that increases broad relevance through the use of muon detectors, and one that uses equipment typical to an introductory physics lab and not relevant beyond the classroom. We measure student outcomes for both experimental critical thinking skills and attitudes towards physics labs. We use hierarchical linear modeling to compare student outcomes between the two labs. We find that both experimentation-based labs produce similar student outcomes. Our results suggest that increased levels of broad relevance may not inherently improve gains in student learning or attitudes. Future work should further investigate which components of different experimentation-based lab formats are associated with gains in student outcomes. Although this study did not implement a full CURE, our findings align with a growing body of evidence challenging the idea that CUREs are uniquely positioned to achieve superior student outcomes over other well-designed experimentation-based labs.

physics.ed-ph↗

arXiv2026

Predictive Modeling for High Impact Active Learning Classrooms

Over the past several decades, a large body of research has shown that undergraduate science students learn more and more equitably in active learning classrooms; however, the term "active learning" lacks definition and little research has examined which types and combinations of active learning strategies are most effective. In this study, we use a dataset representing over 10,000 students and 24 institutions to create a predictive model that maps classroom time spent on different activities to student conceptual learning. We find that four variables -- classroom time spent on lecture, group worksheets, clicker questions, and student questions -- are sufficient to reliably predict student learning, as measured by concept inventory scores. We identify one type of class that consistently demonstrates exceptional student learning gains (effect sizes greater than 2): those that spend 10-20% of class time on group worksheets, 20-40% of class time on group clicker questions, and average two or more student questions per hour of class time. We also find that classes which do not utilize group worksheets consistently have learning outcomes comparable to fully lecture classes. These results provide testable recommendations for future controlled studies to investigate effective active learning implementation in undergraduate physics courses.

physics.ed-ph↗

arXiv2026

Lessons from pendulums: A design comparison of three lab activities

We present three versions of a pendulum lab activity to explore how theoretical commitments, motivations, and aspirations reflect in curriculum design. In earlier work, Boudreaux & Elby (2020) discussed how different theoretical perspectives led them to different designs of tutorials. Here, we discuss our finding that even starting from the same theoretical perspective and having similar goals for students can lead to differences of design. We give three interacting reasons why our labs diverge: We have different expectations of students at our respective institutions; we have different ancillary goals; and we draw different implications from our shared theoretical commitments. Our account demonstrates the complexity of the relationship between theory, goals, and curriculum design. In this way, it adds to prior arguments for the importance of designers' articulating the reasoning for their choices as well as for the possible value of instructors' responsive adaptations to curricula.

physics.ed-ph↗

arXiv2025

Share, Rotate, Split: The Effects of Group Work Role Distributions on Student Outcomes

Education literature recommends many different strategies for structuring student group work in labs. Many of these strategies, however, have not been sufficiently evaluated for their effects on student outcomes. One prior study suggested that sharing roles, rather than splitting roles, in lab groups can boost students' physics interest and self-efficacy. Here, we expand upon this literature by evaluating the effects of a broader range of role distributions across several student outcomes from a large sample at two different institutions. We developed a survey item to probe the ways students distribute their roles in lab groups. The item asks for the percent of time in lab they spent working together on lab roles (sharing), working alone on roles but rotating each session (rotating), and working alone in the same role throughout the semester (splitting). We employed hierarchical linear modeling to measure the effects of these role distributions on student critical thinking, self-efficacy, perceived agency, belonging, and sense of recognition based on survey items specific to physics lab contexts. We found that role distributions did not differentially impact student critical thinking. We also found that sharing roles tended to have a positive impact on student attitudes; splitting had a negative effect on attitudes; and rotating fell in between. Statistical significance varied across these attitudinal outcomes. Our findings invite further research and controlled studies to better understand the apparent benefits of sharing, rotating, and splitting roles in introductory physics labs.

physics.ed-ph↗

arXiv2025

Comparing large language models for supervised analysis of students' lab notes

Recent advancements in large language models (LLMs) hold significant promise in improving physics education research that uses machine learning. In this study, we compare the application of various models to perform large-scale analysis of written text grounded in a physics education research classification problem: identifying skills in students' typed lab notes through sentence-level labeling. Specifically, we use training data to fine-tune two different LLMs, BERT and LLaMA, and compare the performance of these models to both a traditional bag of words approach and a few-shot LLM (without fine-tuning).} We evaluate the models based on their resource use, performance metrics, and research outcomes when identifying skills in lab notes. We find that higher-resource models often, but not necessarily, perform better than lower-resource models. We also find that all models estimate similar trends in research outcomes, although the absolute values of the estimated measurements are not always within uncertainties of each other. We use the results to discuss relevant considerations for education researchers seeking to select a model type to use as a classifier.

physics.ed-ph↗

arXiv2024

Bias in physics peer recognition does not explain gaps in perceived recognition

Gaining recognition as a physics person by peers is an important contributor to undergraduate students' physics identity and their success in physics courses. Previous research has separately demonstrated that women perceive less recognition from peers than men in their physics courses (perceived recognition) and that women receive fewer nominations from their peers as strong in their physics course than men (received recognition). The relationship between perceived and received peer recognition for men and women, however, is not well understood. Here we test three plausible models for this relationship. We conduct a large-scale, quantitative study of over 1,700 students enrolled in introductory physics courses at eight institutions in the United States. We directly compare student gender, perceived recognition, and received recognition, controlling for other student demographics and course-level variability. Results show with high precision that, for students receiving the same amount of recognition, and having the same race or ethnicity, academic year, and major, women report significantly lower perceived recognition than men. These findings offer important implications for testable instructional interventions.

physics.ed-ph↗

arXiv2024

Dynamics of Productive Confirmation Framing in an Introductory Lab

In introductory physics laboratory instruction, students often expect to confirm or demonstrate textbook physics concepts (Wilcox & Lewandowski, 2017; Hu & Zwickl, 2017; Hu & Zwickl, 2018). This expectation is largely undesirable: labs that emphasize confirmation of textbook physics concepts are unsuccessful at teaching those concepts (Wieman & Holmes, 2015; Holmes et al., 2017) and even in contexts that don't emphasize confirmation, such expectations can lead to students disregarding or manipulating their data in order to obtain the expected result (Smith et al., 2020). In other words, when students expect their lab activities to confirm a known result, they may relinquish epistemic agency and violate disciplinary practices. We claim that, in other cases, confirmatory expectations can actually support productive disciplinary engagement. In particular, when an expected result is not confirmed, students may enter a productive "troubleshooting" mode (Smith et al., 2020). We analyze the complex dynamics of students' epistemological framing in a lab where student's confirmatory expectations support and even generate epistemic agency and disciplinary practices, including developing original ideas, measures, and apparatuses to apply to the material world.

physics.ed-ph↗

arXiv2024

Who and what gets recognized in peer recognition

Previous work has identified that recognition from others is an important predictor of students' participation, persistence, and career intentions in physics. However, research has also found a gender bias in peer recognition in which student nominations of strong peers in their physics course disproportionately favor men over women. In this study, we draw on methods from social network analysis and find a consistent gender bias in which men disproportionately under-nominate women as strong in their physics course in two offerings of both a lecture course (for science and engineering, but not physics, majors) and a distinct lab course (for science, engineering, and physics majors). We also find in one offering of the lecture course that women disproportionately under-nominate men, contrary to what previous research would predict. We expand on prior work by also probing two data sources related to who and what gets recognized in peer recognition: students' interactions with their peers (who gets recognized) and students' written explanations of their nominations of strong peers (what gets recognized). Results suggest that the nature of the observed gender bias in peer recognition varies between the instructional contexts of lecture and lab. In the lecture course, the gender bias is related to who gets recognized: both men and women disproportionately over-nominate their interaction ties to students of their same gender as strong in the course. In the lab course, the gender bias is also related to what gets recognized: men nominate men more than women because of skills related to interactions, such as being helpful. These findings illuminate the different ways in which students form perceptions of their peers and add nuance to our understanding of the nature of gender bias in peer recognition.

physics.ed-ph↗

arXiv2023

A method to assess trustworthiness of machine coding at scale

Physics education researchers are interested in using the tools of machine learning and natural language processing to make quantitative claims from natural language and text data, such as open-ended responses to survey questions. The aspiration is that this form of machine coding may be more efficient and consistent than human coding, allowing much larger and broader data sets to be analyzed than is practical with human coders. Existing work that uses these tools, however, does not investigate norms that allow for trustworthy quantitative claims without full reliance on cross-checking with human coding, which defeats the purpose of using these automated tools. Here we propose a four-part method for making such claims with supervised natural language processing: evaluating a trained model, calculating statistical uncertainty, calculating systematic uncertainty from the trained algorithm, and calculating systematic uncertainty from novel data sources. We provide evidence for this method using data from two distinct short response survey questions with two distinct coding schemes. We also provide a real-world example of using these practices to machine code a data set unseen by human coders. We offer recommendations to guide physics education researchers who may use machine-coding methods in the future.

physics.ed-ph↗

arXiv2023

Context affects student thinking about sources of uncertainty in classical and quantum mechanics

Measurement uncertainty is an important topic in the undergraduate laboratory curriculum. Previous research on student thinking about experimental measurement uncertainty has focused primarily on introductory-level students' procedural reasoning about data collection and interpretation. In this paper, we extended this prior work to study upper-level students' thinking about sources of measurement uncertainty across experimental contexts, with a particular focus on classical and quantum mechanics contexts. We developed a survey to probe students' thinking in the generic question "What comes to mind when you think about measurement uncertainty in [classical/quantum] mechanics?" as well as in a range of specific experimental scenarios and interpreted student responses through the lens of availability and accessibility of knowledge pieces. We found that limitations of the experimental setup were most accessible to students in classical mechanics while principles of the underlying physics theory were most accessible to students in quantum mechanics, even in a context in which this theory was not relevant. We recommend that future research probe which sources of uncertainty experts believe are relevant in which contexts and how instruction in both classical and quantum contexts can help students draw on appropriate sources of uncertainty in classical and quantum experiments.

physics.ed-ph↗

arXiv2023

Comparing introductory and beyond-introductory students' reasoning about uncertainty

Uncertainty is an important concept in physics laboratory instruction. However, little work has examined how students reason about uncertainty beyond the introductory (intro) level. In this work we aimed to compare intro and beyond-intro students' ideas about uncertainty. We administered a survey to students at 10 different universities with questions probing procedural reasoning about measurement, student-identified sources of uncertainty, and predictive reasoning about data distributions. We found that intro and beyond-intro students answered similarly on questions where intro students already exhibited expert-level reasoning, such as in comparing two data sets with the same mean but different spreads, identifying limitations in an experimental setup, and predicting how a data distribution would change if more data were collected. For other questions, beyond-intro students generally exhibited more expert-like reasoning than intro students, such as when determining whether two sets of data agree, identifying principles of measurement that contribute to spread, and predicting how a data distribution would change if better data were collected. Neither differences in student populations, lab courses taken, nor research experience were able to fully explain the variability between intro and beyond-intro student responses. These results call for further research to better understand how students' ideas about uncertainty develop beyond the intro level.

physics.ed-ph↗

arXiv2023

What topics of peer interactions correlate with student performance in physics courses?

Research suggests that interacting with more peers about physics course material is correlated with higher student performance. Some studies, however, have demonstrated that different topics of peer interactions may correlate with their performance in different ways, or possibly not at all. In this study, we probe both the peers with whom students interact about their physics course and the particular aspects of the course material about which they interacted in six different introductory physics courses: four lecture courses and two lab courses. Drawing on methods in social network analysis, we replicate prior work demonstrating that, on average, students who interact with more peers in their physics courses have higher final course grades. Expanding on this result, we find that students discuss a wide range of aspects of course material with their peers: concepts, small-group work, assessments, lecture, and homework. We observe that in the lecture courses, interacting with peers about concepts is most strongly correlated with final course grade, with smaller correlations also arising for small-group work and homework. In the lab courses, on the other hand, small-group work is the only interaction topic that significantly correlates with final course grade. We use these findings to discuss how course structures (e.g., grading schemes and weekly course schedules) may shape student interactions and add nuance to prior work by identifying how specific types of student interactions are associated (or not) with performance.

physics.ed-ph↗

arXiv2023

New perspectives on student reasoning about measurement uncertainty: More or better data

Uncertainty is an important and fundamental concept in physics education. Students are often first exposed to uncertainty in introductory labs, expand their knowledge across lab courses, and then are introduced to quantum mechanical uncertainty in upper-division courses. This study is part of a larger project evaluating student thinking about uncertainty across these contexts. In this research, we investigate advanced physics student thinking about uncertainty by asking them conceptual questions about how a hypothetical distribution of measurements would change if `more' or `better' data were collected in four different experimental scenarios. The scenarios include both classical and quantum experiments, as well as experiments that theoretically result in an expected single value or an expected distribution. This investigation is motivated by our goal of finding insights into students' potential point- and set-like thinking about uncertainty and of shining light on the limitations of those binary paradigms.

physics.ed-ph↗

arXiv2023

Instructing nontraditional physics labs: Toward responsiveness to student epistemic framing

Research on nontraditional laboratory (lab) activities in physics shows that students often expect to verify predetermined results, as takes place in traditional activities. This understanding of what is taking place, or epistemic framing, may impact their behaviors in the lab, either productively or unproductively. In this paper, we present an analysis of student epistemic framing in a nontraditional lab to understand how instructional context, specifically instructor behaviors, may shape student framing. We present video data from a lab section taught by an experienced teaching assistant (TA), with 19 students working in seven groups. We argue that student framing in this lab is evidenced by whether or not students articulate experimental predictions and by the extent to which they take up opportunities to construct knowledge (epistemic agency). We show that the TA's attempts to shift student frames generally succeed with respect to experimental predictions but are less successful with respect to epistemic agency. In part, we suggest, the success of the TA's attempts reflects whether and how they are responsive to students' current framing. This work offers evidence that instructors can shift students' frames in nontraditional labs, while also illuminating the complexities of both student framing and the role of the instructor in shifting that framing in this context.

physics.ed-ph↗

arXiv2022

Understanding interaction network formation across instructional contexts in remote physics courses

Engaging in interactions with peers is important for student learning. Many studies have quantified patterns of student interactions in in-person physics courses using social network analysis, finding different network structures between instructional contexts (lecture and lab) and styles (active and traditional). Such studies also find inconsistent results as to whether and how student-level variables (e.g., grades and demographics) relate to the formation of interaction networks. In this cross-sectional research study, we investigate these relationships further by examining lecture and lab interaction networks in four different remote physics courses spanning various instructional styles and student populations. We apply statistical methods from social network analysis -- exponential random graph models -- to measure the relationship between network formation and multiple variables: students' discussion and lab section enrollment, final course grades, gender, and race/ethnicity. Similar to previous studies of in-person courses, we find that remote lecture interaction networks contain large clusters connecting many students, while remote lab interaction networks contain smaller clusters of a few students. Our statistical analysis suggests that these distinct network structures arise from a combination of both instruction-level and student-level variables, including the learning goals of each instructional context, whether assignments are completed in groups or individually, and the distribution of gender and major of students enrolled in a course. We further discuss how these and other variables help to understand the formation of interaction networks in both remote and in-person physics courses.

physics.ed-ph↗

arXiv2022

Introductory physics students' recognition of strong peers: Gender and racial/ethnic bias differ by course level and context

Researchers have pinpointed recognition from others as one of the most important dimensions of students' science and engineering identity. Studies, however, have found gender biases in students' recognition of their peers, with inconsistent patterns across introductory science and engineering courses. Toward finding the source of this variation, we examine whether a gender bias exists in students' nominations of strong peers across three different remote, introductory physics courses with varying student populations (varying demographics, majors, and course levels). We also uniquely evaluate possible racial/ethnic biases and probe the relationship between instructional context (whether lecture or laboratory) and recognition. Some of our results replicate previous findings (such as the the association of course grade and small class section enrollment with nominations), while others offer contradictions. Comparing across our three courses and the prior work, results suggest that course level (whether first-year students or beyond-first-year students) might be more associated with a gender bias in peer recognition than other variables. Surprisingly, we also find instances of racial/ethnic biases in favor of students from backgrounds historically underrepresented in science. Finally, we find that the nomination patterns differ when students nominate individuals strong in the lecture material versus laboratory material. This work serves as an important step in determining which courses and contexts exhibit biases in peer recognition, as well as how students' perceptions of one another form in remote teaching environments.

physics.ed-ph↗

arXiv2021

Not engaging with problems in the lab: Students' navigation of conflicting data and models

With the adoption of instructional laboratories (labs) that require students to make their own decisions, there is a need to better understand students' activities as they make sense of their data and decide how to proceed. In particular, understanding when students do not engage productively with unexpected data may provide insights into how to better support students in more open-ended labs. We examine video and audio data from groups within a lab session where students were expected to find data inconsistent with the predictions of two models. In prior work, we examined the actions of the four groups that productively grapple with this designed problem. Here, we analyze the engagement of the three groups that do not. We conducted three phases of analysis: 1) documenting large scale behaviors and time spent in on-topic discussion, 2) analyzing interactions with the teaching assistant, and 3) identifying students' framing--their expectations for what is taking place--when they were discussing their data. Our Phase 1 and 2 analysis show only minor differences between the groups that engaged with the problem and those that did not. Our Phase 3 analysis demonstrated that the groups that did not engage with the problem framed the lab activity as about confirming a known result or as a series of hoops to jump through to fulfill assignment requirements. Implications for instruction include supporting teaching assistants to attend to students' framing and agency within laboratory classrooms.

physics.ed-ph↗