arXiv subjects

Marcos D. Caballero

Publications and source records attributed to Marcos D. Caballero.

At least 19 recordsLinked to original sources

arXiv2025

Characterizing Faculty Online Learning Community Interactions Using Social Network Analysis

The Partnership for Integration of Computation into Undergraduate Physics (PICUP) was founded in the mid-2010s to assist educators with the challenges of integrating computation into physics curricula. In addition to in-person trainings and hosted educational materials, PICUP uses a Slack Workspace to continue collaboration and discussion offline. In this work, we use Social Network Analysis (SNA) to study the communication patterns of PICUP and assess if PICUP is meeting their goals in the Slack environment. Through our analysis, we discuss PICUP's community structure and define a conceptual framework to evaluate if the goals are being met through SNA metrics. We present a comprehensive analysis of eight channels in the Slack Workspace using various SNA metrics, identifying three distinct levels of user engagement. We conclude with implications for PICUP and provide recommendations for the community.

physics.ed-ph↗

arXiv2024

Overview of couplet scoring in content-focused physics assessments

Content-focused research-based assessment instruments typically use items (i.e., questions) as the unit of assessment for scoring, reporting, and validation. Couplet scoring employs an alternative unit of assessment called a couplet, which is essentially an item viewed and scored through the lens of a specific assessment objective. With couplet scoring, a single item may have more than one assessment objective and therefore more than one couplet and thus more than one score. We outline the components of traditional item scoring, discuss couplet scoring and its benefits, and use both a recently developed content research-based assessment instrument and an existing one to ground our discussion.

physics.ed-ph↗

arXiv2023

How do we assess computation in physics?

In recent years, computing has become an important part of the way we teach and learn physics. Teachers, both at high school and college levels, now use computational activities in many of their courses. Physics departments are offering specialized courses and degrees in computational physics. And many countries are adding programming or computational thinking to their secondary science education standards. Although we know more about how to teach computation, that's only half the picture; we need to know how to assess it. In this paper, we provide a snapshot of some commonly used assessment activities and forms.

physics.ed-ph↗

arXiv2023

Survey of physics reasoning on uncertainty concepts in experiments: an assessment of measurement uncertainty for introductory physics labs

Measurement uncertainty is a critical feature of experimental research in the physical sciences, and the concepts and practices surrounding measurement uncertainty are important components of physics lab courses. However, there has not been a broadly applicable, research-based assessment tool that allows physics instructors to easily measure students' knowledge of measurement uncertainty concepts and practices. To address this need, we employed Evidence-Centered Design to create the Survey of Physics Reasoning on Uncertainty Concepts in Experiments (SPRUCE). SPRUCE is a pre-post assessment instrument intended for use in introductory (first- and second-year) physics lab courses to help instructors and researchers identify student strengths and challenges with measurement uncertainty. In this paper, we discuss the development of SPRUCE's assessment items guided by Evidence-Centered Design, focusing on how instructors' and researchers' assessment priorities were incorporated into the assessment items and how students' reasoning from pilot testing informed decisions around item answer options. We also present an example of some of the feedback an instructor would receive after implementing SPRUCE in a pre-post fashion, along with a brief discussion of how that feedback could be interpreted and acted upon.

physics.ed-ph↗

arXiv2022

Students' perspectives on computational challenges in physics class

High school science classrooms across the United States are answering calls to make computation a part of science learning. The problem is that there is little known about the barriers to learning that computation might bring to a science classroom or about how to help students overcome these challenges. This case study explores these challenges from the perspectives of students in a high school physics classroom with a newly revamped, computation-integrated curriculum. Focusing mainly on interviews to center the perspectives of students, we found that computation is a double-edged sword: It can make science learning more authentic for students who are familiar with it, but it can also generate frustration and an aversion towards physics for students who are not.

physics.ed-ph↗

arXiv2021

Rubric-based holistic review represents a change from traditional graduate admissions approaches in physics

Rubric-based admissions are claimed to help make the graduate admissions process more equitable, possibly helping to address the historical and ongoing inequities in the U.S. physics graduate school admissions process that have often excluded applicants from minoritized races, ethnicities, genders, and backgrounds. Yet, no studies have examined whether rubric-based admissions methods represent a fundamental change of the admissions process or simply represent a new tool that achieves the same outcome. To address that, we developed supervised machine learning models of graduate admissions data collected from our department over a seven-year period. During the first four years, our department used a traditional admissions process and switched to a rubric-based process for the following three years, allowing us to compare which parts of the applications were used to drive admissions decisions. We find that faculty focused on applicants' physics GRE scores and grade point averages when making admissions decisions before the implementation of the rubric. While we were able to develop a sufficiently good model whose results we could trust for the data before the implementation of the rubric, we were unable to do so for the data collected after the implementation of the rubric, despite multiple modifications to the algorithms and data such as implementing Tomek Links. Our inability to model the second data set despite being able to model the first combined with model comparison analyses suggests that rubric-based admissions does change the underlying process. These results suggest that rubric-based holistic review is a method that could make the graduate admissions process in physics more equitable.

physics.ed-ph↗

arXiv2021

Predictive and explanatory models might miss informative features in educational data

We encounter variables with little variation often in educational data mining (EDM) due to the demographics of higher education and the questions we ask. Yet, little work has examined how to analyze such data. Therefore, we conducted a simulation study using logistic regression, penalized regression, and random forest. We systematically varied the fraction of positive outcomes, feature imbalances, and odds ratios. We find the algorithms treat features with the same odds ratios differently based on the features' imbalance and the outcome imbalance. While none of the algorithms fully solved how to handle imbalanced data, penalized approaches such as Firth and Log-F reduced the difference between the built-in odds ratio and value determined by the algorithm. Our results suggest that EDM studies might contain false negatives when determining which variables are related to an outcome. We then apply our findings to a graduate admissions data set. We end by proposing recommendations that researchers should consider penalized regression for data sets on the order of hundreds of cases and should include more context about their data in publications such as the outcome and feature imbalances.

stat.ME↗

arXiv2021

Rubric-based holistic review: a promising route to equitable graduate admissions in physics

As systematic inequities in higher education and society have been brought to the forefront, graduate programs are interested in increasing the diversity of their applicants and enrollees. Yet, structures in place to evaluate applicants may not support such aims. One potential solution to support those aims is rubric-based holistic review. Starting in 2018, our physics department implemented a rubric-based holistic review process for all applicants to our graduate program. The rubric assessed applicants on 18 metrics covering their grades, test scores, research experiences, noncognitive competencies, and fit with the program. We then compared faculty's ratings of applicants by admission status, sex, and undergraduate program over a three-year period. We find that the rubric scores show statistically significant differences between admitted and non-admitted students as hoped and that statistically significant differences based on sex or undergraduate program aligned with known disparities in GRE scores and service work expectations. Our results then suggest rubric-based holistic review as a possible route to making graduate admissions in physics more equitable.

physics.ed-ph↗

arXiv2021

A Framework for Evaluating Statistical Models in Physics Education Research

Across the field of education research there has been an increased focus on the development, critique, and evaluation of statistical methods and data usage due to recently created, very large data sets and machine learning techniques. In physics education research (PER), this increased focus has recently been shown through the 2019 Physical Review PER Focused Collection examining quantitative methods in PER. Quantitative PER has provided strong arguments for reforming courses by including interactive engagement, demonstrated that students often move away from scientist-like views due to science education, and has injected robust assessment into the physics classroom via concept inventories. The work presented here examines the impact that machine learning may have on physics education research, presents a framework for the entire process including data management, model evaluation, and results communication, and demonstrates the utility of this framework through the analysis of two types of survey data.

physics.ed-ph↗

arXiv2021

Developing a learning goal framework for computational thinking in computationally integrated physics classrooms

Computational thinking has been a recent focus of education research within the sciences. However, there is a dearth of scholarly literature on how best to teach and to assess this topic, especially in disciplinary science courses. Physics classes with computation integrated into the curriculum are a fitting setting for investigating computational thinking. In this paper, we lay the foundation for exploring computational thinking in introductory physics courses. First, we review relevant literature to synthesize a set of potential learning goals that students could engage in when working with computation. The computational thinking framework that we have developed features 14 practices contained within 6 different categories. We use in-class video data as existence proofs of the computational thinking practices proposed in our framework. In doing this work, we hope to provide ways for teachers to assess their students' development of computational thinking, while also giving physics education researchers some guidance on how to study this topic in greater depth.

physics.ed-ph↗

arXiv2020

Investigating institutional influence on graduate program admissions by modelling physics GRE cut-off scores

Despite limiting access to applicants from underrepresented racial and ethnic groups, the practice of using hard or soft GRE cut-off scores in physics graduate program admissions is still a popular method for reducing the pool of applicants. The present study considers whether the undergraduate institutions of applicants have any influence on the admissions process by modelling a physics GRE cut-off score with application data from admissions offices of five universities. Two distinct approaches based on inferential and predictive modelling are conducted. While there is some disagreement regarding the relative importance between features, the two approaches largely agree that including institutional information significantly aids the analysis. Both models identify cases where the institutional effects are comparable to factors of known importance such as gender and undergraduate GPA. As the results are stable across many cut-off scores, we advocate against the practice of employing physics GRE cut-off scores in admissions.

physics.ed-ph↗

arXiv2020

The Physics GRE does not help applicants "stand out"

One argument for keeping the physics GRE is that it can help applicants who might otherwise be missed in the admissions process stand out. In this work, we evaluate whether this claim is supported by physics graduate school admissions decisions. We used admissions data from five PhD-granting physics departments over a 2-year period (N=2537) to see how the fraction of applicants admitted varied based on their physics GRE scores. We compared applicants with low GPAs to applicants with higher GPAs, applicants from large undergraduate universities to applicants from smaller undergraduate universities, and applicants from selective undergraduate institutions to applicants from less selective undergraduate institutions. We also performed a mediation and moderation analysis to provide statistical rigor and to better understand the previous relationships. We find that for applicants who might otherwise have been missed (e.g. have a low GPA or attended a small or less selective school) having a high physics GRE score did not seem to increase the applicant's chances of being admitted to the schools. However, having a low physics GRE score seemed to penalize otherwise competitive applicants. Thus, our work suggests that the physics GRE does not, in fact, help applicants who might otherwise be missed stand out.

physics.ed-ph↗

arXiv2020

Predicting time to graduation at a large enrollment American university

The time it takes a student to graduate with a university degree is mitigated by a variety of factors such as their background, the academic performance at university, and their integration into the social communities of the university they attend. Different universities have different populations, student services, instruction styles, and degree programs, however, they all collect institutional data. This study presents data for 160,933 students attending a large American research university. The data includes performance, enrollment, demographics, and preparation features. Discrete time hazard models for the time-to-graduation are presented in the context of Tinto's Theory of Drop Out. Additionally, a novel machine learning method: gradient boosted trees, is applied and compared to the typical maximum likelihood method. We demonstrate that enrollment factors (such as changing a major) lead to greater increases in model predictive performance of when a student graduates than performance factors (such as grades) or preparation (such as high school GPA).

physics.ed-ph↗

arXiv2020

Thematic Analysis of 18 Years of PERC Proceedings using Natural Language Processing

We have used an unsupervised machine learning method called Latent Dirichlet Allocation (LDA) to thematically analyze all papers published in the Physics Education Research Conference Proceedings between 2001 and 2018. By looking at co-occurrences of words across the data corpus, this technique has allowed us to identify ten distinct themes or "topics" that have seen varying levels of prevalence in Physics Education Research (PER) over time and to rate the distribution of these topics within each paper. Our analysis suggests that although all identified topics have seen sustained interest over time, PER has also seen several waves of increased interest in certain topics, beginning with initial interest in qualitative, theory-building studies of student understanding, which has given way to a focus on problem solving in the late 2010s. Since 2010 the field has seen a shift towards more sociocultural views of teaching and learning with a particular focus on communities of practice, student identities, and institutional change. Based on these results, we suggest that unsupervised text analysis techniques like LDA may hold promise for providing quantitative, independent, and replicable analyses of educational research literature.

physics.ed-ph↗

arXiv2019

Physics Computational Literacy: An Exploratory Case Study Using Computational Essays

Computation is becoming an increasingly important part of physics education. However, there are currently few theories of learning that can be used to help explain and predict the unique challenges and affordances associated with computation in physics. In this study, we adapt the existing theory of computational literacy, which posits that computational learning can be divided into material, cognitive, and social aspects, to the context of undergraduate physics. Based on an exploratory study of undergraduate physics computational literacy, using a newly-developed teaching tool known as a computational essay, we have identified a variety of student practices, knowledge, and beliefs across these three aspects of computational literacy. We illustrate these categories with data collected from students who engaged in an initial implementation of computational essays in an introductory electricity and magnetism class. We conclude by arguing that this framework can be used to theoretically diagnose student difficulties with computation, distinguish educational approaches that focus on material vs. cognitive aspects of computational literacy, and highlight the benefits and limitations of open-ended projects like computational essays to student learning.

physics.ed-ph↗

arXiv2019

Using machine learning to understand physics graduate school admissions

Among all of the first-year graduate students enrolled in doctoral-granting physics departments, the percentage of female and racial minority students has remained unchanged for the past 20 years. The current graduate program admissions process can create challenges for achieving diversity goals in physics. In this paper, we will investigate how the various aspects of a prospective student's application to a physics doctoral program affect the likelihood the applicant will be admitted. Admissions data was collected from a large, Midwestern public research university that has a decentralized admissions process and included applicants' undergraduate GPAs and institutions, research interests, and GRE scores. Because the collected data varied in scale, we used supervised machine learning algorithms to create models that predict who was admitted into the PhD program. We find that using only the applicant's undergraduate GPA and physics GRE score, we are able to predict with 75% accuracy who will be admitted to the program.

physics.ed-ph↗

arXiv2019

Computational Essays: An Avenue for Scientific Creativity in Physics

Computation holds great potential for introducing new opportunities for creativity and exploration into the physics curriculum. At the University of Oslo we have begun development of a new class of assignment called computational essays to help facilitate creative, open-ended computational physics projects. Computational essays are a type of essay or narrative that combine text and code to express an idea or make an argument, usually written in computational notebooks. During a pilot implementation of computational essays in an introductory electricity and magnetism course, students reported that computational essays facilitated creative investigation at a variety of levels within their physics course. They also reported finding this creativity as being both challenging and motivating. Based on these reflections, we argue that computational essays are a useful tool for leveraging the creative affordances of programming in physics education.

physics.ed-ph↗

arXiv2019

Modeling student pathways in a physics bachelor's degree program

Physics education research has used quantitative modeling techniques to explore learning, affect, and other aspects of physics education. However, these studies have rarely examined the predictive output of the models, instead focusing on the inferences or causal relationships observed in various data sets. This research introduces a modern predictive modeling approach to the PER community using transcript data for students declaring physics majors at Michigan State University (MSU). Using a machine learning model, this analysis demonstrates that students who switch from a physics degree program to an engineering degree program do not take the third semester course in thermodynamics and modern physics, and may take engineering courses while registered as a physics major. Performance in introductory physics and calculus courses, measured by grade as well as a students' declared gender and ethnicity play a much smaller role relative to the other features included the model. These results are used to compare traditional statistical analysis to a more modern modeling approach.

physics.ed-ph↗