United States

Classifying students into groups that exhibit similar behavioral patterns so they can be treated with a tailored instruction is a new research direction that attracts both the Physics Education and the Learning Analytics research communities. Typically, personalization approaches center on the individual learner, and identify the knowledge components required for solving each problem based on expert task analysis or educational data mining. However, individual-level personalization is less appropriate in the reality of science classes, and may also miss the advantages of group work and discussions. Moreover, previous studies have shown that experts may miss latent knowledge structures that bottom-up analyses such as Exploratory Factor Analysis (EFA) may help to reveal. 

In this study we examined how cluster analysis can reveal additional cognitive dimensions not identified by the experts. We used cluster analysis and not EFA which is typically used for this purpose because our end goal was using the cognitive dimensions to provide the teachers with sub-groups of students sharing similar profiles. 

The research tool that we used was an instrument on Magnetism. The items were analyzed by experts and the knowledge components identified for each item were represented in a Q-matrix. The instrument was administered to ~300 12th grade Physics majors. K-means cluster analysis revealed that subgroups can be differentiated by type of knowledge structures (e.g. procedural vs. declarative), resulting in improved labeling of the questions. In follow-up interviews the teachers expressed their satisfaction with the usefulness of this representation for tailoring their teaching to each group&#39;s difficulties and strengths.

PERC SM 21

Combining expert- and cluster-analysis for Group-level personalization

cluster-analysis

magnetism

personalization

**Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics**

As a gateway discipline to STEM majors and careers, physics plays a central role in deciding who does, and who doesn't, get to discover and pursue scientific careers. Technical and scientific professionals are rewarded with economic privilege and status which affords them both social and cultural power. Therefore, social justice will be realized only when the opportunity, and the choice, to excel in physics is open to all students. The culture and practices of the physics classroom reflect the culture of physics at large, which raises the question of how our professional physics communities can embrace, rather than inadvertently repel, a diverse group of academics that can help improve physics culture.  To this end, the 2021 Physics Education Research Conference will focus on defining inclusive spaces and practices. Additionally, we will explore strategies for eliminating exclusionary practices in our classrooms, research spaces, and professional organizations. We call on the community to propose sessions that will drive conversations related to these initiatives throughout PERC 2021.

Please download the full program [here. ](https://drive.google.com/file/d/1TRMC_SYJ3PUqXyP7sskA7LRSA8ow0eRE/view?usp=sharing)

August 4, 2021 - August 5, 2021 in a Virtual Conference Format

Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics

Classifying students into groups that exhibit similar behavioral patterns so they can be treated with a tailored instruction is a new research direction that attracts both the Physics Education and the Learning Analytics research communities. Typically, personalization approaches center on the individual learner, and identify the knowledge components required for solving each problem based on expert task analysis or educational data mining. However, individual-level personalization is less appropriate in the reality of science classes, and may also miss the advantages of group work and discussions. Moreover, previous studies have shown that experts may miss latent knowledge structures that bottom-up analyses such as Exploratory Factor Analysis (EFA) may help to reveal. 

In this study we examined how cluster analysis can reveal additional cognitive dimensions not identified by the experts. We used cluster analysis and not EFA which is typically used for this purpose because our end goal was using the cognitive dimensions to provide the teachers with sub-groups of students sharing similar profiles. 

The research tool that we used was an instrument on Magnetism. The items were analyzed by experts and the knowledge components identified for each item were represented in a Q-matrix. The instrument was administered to ~300 12th grade Physics majors. K-means cluster analysis revealed that subgroups can be differentiated by type of knowledge structures (e.g. procedural vs. declarative), resulting in improved labeling of the questions. In follow-up interviews the teachers expressed their satisfaction with the usefulness of this representation for tailoring their teaching to each group's difficulties and strengths.

poster

The Force Concept Inventory (FCI), one of the most widely used tools in the physics community, is commonly used as a pre and posttest assessment to gauge the effectiveness of various teaching strategies. Over the years there have been various studies on the FCI itself, and different formats have been created. In this preliminary study we are looking at the incorporation of videos that illustrate the problem statements as a way to see if these additions have positive effects on students’ responses to FCI questions. We took a subset of four questions from the FCI and varied the format of the questions and videos in quizzes administered to multiple introductory algebra-based physics courses. Though there were no quantitatively significant differences in students’ performance on questions between those who answered the as-written FCI questions versus the questions with the videos, we found significant differences on some questions with repeated responses to the same questions with and without video. Furthermore, students overall felt the videos helped them visualize the question scenarios.

Preliminary results on a video-based force concept inventory

Due to the COVID-19 pandemic, many universities have had to transition to online instruction. For large-enrollment introductory physics classes, this situation raises questions of ensuring academic honesty and fairness in assessments. Short, low-stakes quizzes have been suggested as an alternative to traditional exams, in order to minimize opportunity for students to engage in academic dishonesty by way of unauthorized online resources that can be problematic for longer online exams, and to reduce the pressure students feel that may tempt them to cheat on high-stakes exams. However, short quizzes could still be vulnerable to academic dishonesty by way of student sharing of assessment content with each other through various means that may be difficult to detect. Moreover, students may not perceive that short quizzes are meant to be low-stakes. In this investigation, the prevalence of academic dishonesty via unauthorized information sharing was explored in two remote large-enrollment introductory physics courses. In addition, student perception of assessments was investigated. It was found that students learning remotely had a preference for less-frequent, longer assessments, compared to frequent, short quizzes. It was also found that evidence for academic dishonesty was reduced when students perceived the assessments as fair, even if the online format was perceived as more difficult than an in-class experience.

Creating honest and fair remote introductory physics assessments

We use quantitative measures and visual inspection to compare the item response curves (IRCs) of matched pre-/post-instruction Force Concept Inventory (FCI) data. We find that the IRCs are not static; the pre-instruction IRCs differ from the post-instruction IRCs by more than can be explained by random chance. This result is also the case for a subpopulation consisting of students who make little or no gains on the FCI, suggesting that learning is taking place even when scores do not change appreciably. We consider three items where students make substantial progress (item 4) or little progress (items 14 and 21) compared to overall changes in FCI scores.

Comparing pre/post item response curves to identify changes in misconceptions

Conceptual inventories such as the FCI are widely used to assess the impact of course reform efforts in physics education. Gains on these instruments, however, are known to correlate with other student variables such as students’ SAT score. I make use of this type of underlying correlation to communicate the impact of course reforms to broad audiences outside of physics education for which the FCI is not a useful benchmark. To illustrate this approach, I present an analysis of FCI learning gains that are disaggregated by ACT score before and after a major course reform in an introductory physics course. In this sample, normalized learning gains shifted from 0.28 to 0.46 were found to be equivalent to a shift in average ACT score from 24 to 29. Implications for the communicating research and evaluation results with different audiences are discussed.

Making use of FCI–ACT Correlations to Communicate the Impact of Course Reform

The updated Next Generation Science Standards place an emphasis on computational integration into science classrooms through the inclusion of mathematics and computational thinking; however, research focused on understanding the impacts and methods of computational integration is still in its infancy. Important questions around activity design and assessment remain unanswered and understudied. In this paper, we present an initial case study of one classroom that integrates computation into a second semester, introductory level physics class. In the course, instructional staff use formative feedback as an assessment tool to both evaluate students and help them develop computational practices. Instructor feedback for several years was coded for repeating trends, resulting in 3 categories of feedback: praise, notice and suggestion. An examination of the data revealed that the feedback is helping to develop positive norms around collaborative computational practices, but little attention is paid to the development of computational thinking practices.

Analysis of computation-based formative feedback

Three-dimensional assessments address application of knowledge and are based on a combination of core ideas, scientific practices, and crosscutting concepts. In this paper, we present one portion of a larger project aimed at developing a flexible, three-dimensional assessment for upper-division thermal physics: the Thermal and Statistical Physics Assessment (TaSPA). The TaSPA tasks are guided by ``learning performances'' (LPs), which faculty interface with to customize the assessment version they prefer to administer in their class. Here, we focus on interviews conducted with 7 physics faculty who teach upper-division thermal physics to gain insights about the three-dimensional LPs developed for the TaSPA. We found faculty rated our LPs variably, but highly overall, and their reasons for their ratings often focused on core concepts as opposed to scientific practices. This finding informs the design of LPs for the TaSPA and supports the need for a flexible assessment which can address varying course foci.

Faculty Perceptions of Three-Dimensional Learning Performances Addressing Thermal Physics

PER-developed assessments are used to track students' progress in a curriculum. If carefully designed, such assessments can also be used by instructors to inform explicit course modifications beyond using them just as tools to evaluate student learning. However, this requires a mechanism to translate assessment outcomes, which are usually presented as a score, into meaningful information about student learning. This information can in turn inform actionable feedback for instructors. In this work, we advocate for PER-developed assessments that explicitly include actionable feedback for instructors. We demonstrate a mechanism to generate feedback for coupled, multiple-response tasks in the context of developing a new standardized assessment for upper-division thermal physics. The inherent features of the coupled, multiple-response tasks such as the ability to administer online with streamlined scoring inform its broader utility in classrooms with small to large student enrollment.

Conceptualizing Assessments in PER Informed by Explicit Feedback for Instructors

Qualitative data coding is a powerful research tool for analyzing text statements; however, having human researchers manually code can be time-consuming and hindering to a project with a large data set. Modern machine learning programs have advanced such that they currently play large roles in automating data categorization and may be useable as a method of semi-automating categorization in PER: human coders can prepare a moderate sample of the data for training, and machine learning software can be left to sort the remaining data, leaving only complex responses to be finally resolved by humans. We have attempted this process using data from a study seeking to code several hundred answers to a single free-response question. Our preliminary work provides Google’s AutoML API with a minimal training sample. In this poster, we compare the accuracy and efficiency of their software to human coders’ interrater reliability to demonstrate how we need to continue developing our training sets moving forward.

Understanding the Efficacy of Machine Learning for Coding Practices

A small number of prior studies have investigated whether instructor gender has a measurable impact on student course grades, though to our best knowledge no study has specifically examined physics courses. We explored the effect of instructor gender on student performance in calculus-based introductory physics courses at a large land grant university. The data set included the scores on midterm and final exams and final letter grades for more than 8200 students taught by 11 male and 4 female instructors between 2007-2018. We found no persistent correlation between student gendered performance and their instructor gender. While statistically significant differences in student performance were found for particular exams or in final letter grades, the effect sizes were weak or small. When comparing male and female instructors, we found that the instructor gender had a larger effect on female students than on their male classmates.

Does Instructor Gender Matter for Student Performance in Introductory Physics?

Response-shift bias explains how changes in students' criteria for self-evaluation can suppress the effect of educational interventions in pre-to-post shifts of self-reported metrics, such as the Colorado Learning Attitudes about Science Survey (CLASS). Therefore, the static or declining pre-to-post shifts seen in the CLASS might not reflect development in students' reasoning about themselves based on new knowledge of physics. We investigated possible causes of response-shift bias in the CLASS by adding a free response question after select CLASS items during a two-pass survey administration, asking students to explain their reasoning behind their expected and perceived shifts. We present the themes that occurred most clearly in students' free responses to one survey item, which highlight the established forms of response-shift bias: recalibration of the students' standards of measurement, reprioritization of the students' values, and reconceptualization of the subject's content.

Next from PERC SM 21

Preliminary results on a video-based force concept inventory

Similar lecture

Middle grade students reasoning about temporary magnetism