Erin Arruda, Arturo Zavala, Panadda Marayong, Jesse Dillon, Chi-ah Chun, Kim-Phuong L. Vu (2024). Research-Infused Courses are Effective for Online and In-Person Education. In: Human Factors in Design, Engineering, and Computing. AHFE Open Access, vol. 159, pp. 1860–1870. https://doi.org/10.54941/ahfe1005753
Engaging students in research is a high impact practice known to increase underrepresented students’ persistence in Science, Technology, Engineering, and Mathematics (STEM) fields and improve their graduation rates. For broad impact, research infusion can be implemented through careful redesign of courses or through the adoption of research modules to supplement class instruction and existing student training programs. In this paper, we present data on a program for the integration of research-infused curriculum in major courses across a variety of disciplines in STEM. Specifically, the program's goal is to have faculty engage in a redesign of the class to exemplify how specific disciplines engage in research. The course redesign can be in the form of activities for in-person education or as interactive activities for online learning. Fourteen courses were included in this analysis. Anonymous surveys were administered at the end of the fall or spring semester to 864 students in the 14 courses from Fall 2020 to Spring 2023. A total of 643 students (74% response rate) participated in the evaluation.Gains in research skills: Students indicated how much they gained in various skills (on a scale of 1: no Gain to 5: Great Gain) as a result of the course. Overall, a high percentage of students (92-96%) reported gains of some kind, while only about 4% of students reported no gains, though a slightly higher proportion reported no gains in understanding what everyday research work is like (8%). Results suggest that on average, students reported moderate to good gains (Ms ranged from 3.36 to 3.70) for all items. Most students felt they had good gains in problem-solving, understanding the relevance of research, understanding research reports, and interpreting results, and moderate gains in comfort discussing research concepts and confidence in ability to do well in research courses. Course Redesign Goals: Students also indicated high levels of agreement about three redesigned course goals: Understanding research-related ethical issues, effective communication of research, and stimulating research interest on a scale ranging from 1 (Strongly Disagree) to 4 (Strongly Agree). Most students agreed/strongly agreed that due to the course, they were able to effectively communicate research information to different audiences (80% Mode=3). Although most students agreed/strongly agreed that the course stimulated interest in research (71%, Mode=3) and helped them understand research ethics (73%; Mode=3), close to one-third of students disagreed (17-23%) or strongly disagreed (3-7%) with these statements.Intention to Pursue Research: More than half of students in the redesigned course indicated that they increased their likelihood of exploring research careers (62%), joining a research/lab training program (64%), and pursuing research-focused courses (62%).Effects of Course Modality: When examining differences by the modality of the course, there were no statistically significant differences between courses that were virtual and courses that were face-to-face for gains in research skills, (X2(1)=0.22, p=.641), research intentions X2(1)=1.30, p=.254, research career perceptions (X2(1)=2.05, p=.152), or course redesign goals, X2(1)=0.08, p=.778.Faculty who received the award found it helpful in redesigning courses to make them more interactive, problem- and research- focused. One awardee shared, “through this program, I can see more underrepresented students more interested in machine learning, data science and artificial intelligence.” Overall, the redesigned courses had a large impact on student’s research skill gains. Students reported moderate to good gains in understanding relevance of research in their discipline and skills important to research like problem solving, understanding research papers, and interpreting research results. The modality of the course (virtual vs. face-to-face) had little impact. Thus, research-focus activities intentionally embedded in courses strengthening the research foundation of students and should be encouraged as a high-impact practice.
Formal undergraduate research training programs, such as the Maximizing Access to Research Careers Undergraduate Student Training in Academic Research (MARC U-STAR) and BUilding Infrastructure Leading to Diversity (BUILD) programs, funded by the National Institutes of Health (NIH), are effective at preparing underrepresented students for the pursuit of advanced degrees in STEM (see e.g., Hall et al., 2016; Vu et al., 2023). However, programs such as these typically can only offer research training to a small percentage of high-achieving undergraduate students. Course- based undergraduate research experiences can increase the percentage of undergraduate students participating in meaningful research experiences before they graduate (Jones & Lerner, 2019). Moreover, early exposure to research in courses can broadly promote awareness of research opportunities
Research-Infused Courses are Effective for Online and In-Person Education 1861
available on campus and the possible benefits of research experiences (Bangera & Brownell, 2014), which can reduce inequities relating to access. For example, Taing et al. (2022) found that research-infused courses promoted research skills gains (e.g., understanding research articles, preparing a research poster, and making oral presentations) as well as personal gains (e.g., personal and professional development and preparation for graduate school). Furthermore, there is evidence that such courses may be especially beneficial for underrepresented minority (URM) students. Specifically, URM students in a research-infused introductory research methods course reported greater interest in pursuing a research career and higher confidence to do well in future advanced research courses compared to non-URM student counterparts. Thus, infusing research across the existing undergraduate curriculum could be an effective method for broadening access to research for students.
IMPLEMENTING RESEARCH COURSE INFUSION AT AN R2 UNIVERSITY The Research Across the Curriculum (RAC) Program was a strategy for institutionalizing health-related research across the curriculum in biomedical and behavioral science courses at the California State University, Long Beach (CSULB), a research intensive (R2) Hispanic-Serving Institution with a large proportion of transfer, financial aid eligible, and first-generation students. RAC was developed to achieve the CSULB BUILD’s overarching goals to strengthen student research training and enhance preparation for graduate studies and success in health-related research careers among underrepresented and underserved students. An essential component of the initiative included supporting curriculum changes that highlight the inquiry and discovery aspects of research across health-related disciplines to be sustained in future offerings of the course. Such changes are designed to foster students’ scientific interest through courses in the discipline- specific curriculum and entice students to consider pursuing research-focused careers. Faculty in the College of Health and Human Services (CHHS), College of Liberal Arts (CLA), College of Natural Sciences & Mathematics (CNSM), and College of Engineering (COE) applied for a course redesign award funded by BUILD to receive one course release time for curriculum development. Their existing courses were redesigned by incorporating relevant hands-on research infused activities and demonstrations, as well as novel scientific approaches that epitomize research in health-related disciplines. The redesigned course was offered in a subsequent semester. Additional adjustments were made, if needed, based on the instructors’ assessment of course outcomes and student feedback.
EFFECTIVENESS OF THE COURSE REDESIGN AWARD To evaluate the effectiveness of the course redesign, end-of-semester student feedback was collected via a brief, anonymous survey. Data were collected from 14 redesigned courses (N=643, 74% response rate). Eight courses were
online, five were face-to-face (F2F), and one was hybrid (see Table 1). The majority of courses were upper-division courses (11), two were lower-division courses, and one was a graduate-level course. Four courses were small (<20 students), seven moderate (21–49 students), and three large (>50 students). Disciplines were from fields related to biomedical and behavioral sciences relevant to the NIH’s mission and included chemistry, biology, psychology, healthcare, nutrition, and engineering.
Table 1. Course list by modality.
Table available in the original PDF paper.
Online Face-to-Face Advanced physical chemistry Finite element methods I** General chemistry Analysis and evaluation of health care services*
Machine learning
Psychobiology honors** Research in social psychology General genetics** Advanced organic chemistry lab Fitness for adult populations with unique health considerations* Quality assurance in health care** Hybrid
Computational fluid dynamics I** Applied sports nutrition Chemical engineering design** Note. Courses were pivoted to virtual modality due to the COVID-19 pandemic (*traditionally hybrid course, ** traditionally face-to-face course).
Survey Design Items were selected or adapted to best match the goals of the BUILD program. Three items measured course redesign goals (e.g., “This course helped me understand research-related ethical issues.”) and three items measured perceptions of research careers (e.g., “As a result of this course, I believe a career as a scientist or researcher would be enjoyable”, Enriquez et al., 2015) on an agreement scale from 1 (Strongly disagree) to 4 (Strongly agree). Three items tapped students’ intentions to pursue research on a likelihood scale (e.g., “Compared to your intentions BEFORE taking this course, how likely are you now to join a research lab or formal research training program?” 1 = Less likely to 6=Extremely likely). Seven items measured gains in research skills on a scale from 1 (No gain) to 5 (Great gain). Sample items included, “How much did you gain in the following areas as a result of the course? Skill in interpreting results” (URSSA, Weston & Laursen, 2015; SURE, Lopatto, 2004). The initial survey was piloted and then shortened to 16 items by removing items that did not apply to all course types. To examine construct validity, an exploratory factor analysis was performed on these final 16 items using principal axis factoring with an Oblimin rotation to allow factors to correlate.
Results indicated that the four-factor solution was supported with adequate factor loadings (λ > .35) for research skill gains (λs =.68−.95), research intentions (λs =.85–.93), research career perceptions (λs =.47– .84), and course redesign goals (λs =.35–.77). Cronbach’s alphas were all above .70, demonstrating adequate reliability of subscale scores (see Table 2).
Research-Infused Courses are Effective for Online and In-Person Education 1863
Based on the item development and exploratory factor analysis results, four composites were computed by averaging across items for each construct. Descriptive statistics were computed (see Table 2), and differences by course characteristics were tested using multilevel modeling, which is described before the group differences results.
Table 2. Descriptive statistics and correlations of research composites.
Table available in the original PDF paper.
Research Composite 1 2 3 4
1. Research gains 620 2. Research interest .55*** 612 3. Research career perceptions .53*** .52*** 637 4. Course redesign goals .63*** .44*** .61*** 637 Cronbach’s alpha .94 .95 .76 .75 M(SD) 3.54(1.00) 3.40(1.52) 2.87(0.60) 2.84(0.59) Mode 4 2 3 3 Note. Pairwise Ns ranged from 592–634, Ns for each construct are displayed on the diagonal, ***p<.001.
Gains in Research Skills Students indicated how much they gained in various skills (listed in Table 3) as a result of the course. Research skill objectives are ordered from highest to lowest average perceived gains in Table 3. Results indicate that, on average, students reported moderate to good gains (Ms = 3.36 to 3.70) for all skills. Most students felt they had good gains in problem-solving, understanding the relevance of research, understanding research reports, and interpreting results, and moderate gains in comfort discussing research concepts and confidence in their ability to do well in research courses.
Table 3. Descriptive statistics of research skill gains.
Table available in the original PDF paper.
Min, Max Mode Mean (SD) N (Missing)
How much did you GAIN in the following areas as a result of the course?
1,5 4 3.70(1.17) 629(14)
Problem-solving in general 1,5 4 3.70(1.14) 638(5) Understanding the relevance of research in this discipline 1,5 4 3.61(1.20) 636(7) Understanding research reports or papers
1,5 4 3.44(1.14) 640(3)
Skills in interpreting results 1,5 4 3.58(1.14) 637(6) Comfort in discussing research concepts with others 1,5 4 3.39(1.21) 639(4) Confidence in my ability to do well in future research courses
1,5 4 3.36(1.22) 639(4)
Understanding what everyday research work is like Note. The scale ranged from 1 (No gain) to 5 (Great gain).
Overall, a high percentage of students (92–96%) reported gains of some kind, while only about 4% of students reported no gains. A similarly high proportion reported gains in understanding what everyday research work is like (92%). Out of all the research topics, there was the most
variation in student gains in understanding what everyday research work is like (see Figure 1). Most students reported good gain (30%), some students indicated a great amount (20%), while a similar number of students reported moderate gain (23%) and little gain (19%). Similarly, there was some variation in confidence in future research courses. While most students reported good gain (27%), some students indicated a great amount (22%), and a similar number of students reported moderate gain (25%) and little gain (19%).
Course Redesign Goals Students indicated high levels of agreement about three redesigned course goals: Understanding research-related ethical issues, effective communication of research, and stimulating research interest (see Figure 2) on a scale ranging from 1 (Strongly Disagree) to 4 (Strongly Agree). Most students agreed/strongly agreed that due to the course, they were able to effectively communicate research information to different audiences (80%, Mode = 3, Agree). Additionally, most students agreed/strongly agreed that the course stimulated interest in research (71%, Mode = 3) and helped them understand research ethics (73%; Mode = 3), although about one-third of students disagreed (17–23%) and a few strongly disagreed (3–7%) with these statements.
Research Career Perceptions Additionally, students indicated how course experiences influenced their perceptions and attitudes about a potential research career (see Figure 3) on a scale ranging from 1 (Strongly Disagree) to 4 (Strongly Agree). Almost all students agreed/strongly agreed that they have a more positive impression of scientists/researchers after the course (89%; Mode = 3, Agree). Only 10% disagreed/slightly disagreed that they had a positive impression. A large majority (70%) agreed/strongly agreed a science/research career would be enjoyable (Mode = 3). More than half agreed/strongly agreed that they were ready for more demanding research (63%, Mode = 3), while a third disagreed/strongly disagreed.
Intention to Pursue Research More than half of students in a redesigned course indicated that the course experience increased their likelihood of exploring research careers (62%), joining a research/lab training program (64%), and pursuing research- focused courses (62%). Figure 4 displays percentages for each response option. About a quarter of students were equally likely to engage in these types of future research opportunities after taking the course than before (ranged 25–26%) while fewer students responded that they were less likely to engage in these opportunities (12–14%). Students were very consistent in their responses across the three types of future research opportunities (rs = .82 to .87). That is, a high likelihood of pursuing a research-focused course was related to a high likelihood of joining a research lab or program and exploring a research-focused career.
Research-Infused Courses are Effective for Online and In-Person Education 1865
Figure 1: Differential gains: Understanding research & confidence in future courses.Figure 2: Research ethics, communication, and interest.Figure 3: Student perceptions of the research career as a result of the course.
Group Comparisons by Course Characteristics Data Analysis Procedures Since data were nested (students nested within courses), multilevel models were analyzed to test differences among groups of course-level characteristics
while accounting for the hierarchical structure of the data and the violation of the assumption of independence of errors. First, empty models were tested using REML to estimate intra-class correlation coefficients (ICCs) and the degree of the violation of the assumption. An ICC (ρ) of zero indicates independence of errors (0% between groups). ICCs indicated multilevel modeling was appropriate to account for non-independence for all course redesign outcomes (ρs ranged from. 12–.22). Next, an intercept-only model (“empty” model) using ML was estimated as the baseline for model comparisons. The following course level predictors were then included in the models: College (Engineering/COE, Health and Human Services/CHHS, Liberal Arts/CLA, or Natural Sciences and Mathematics/CNSM), modality (partially/fully virtual or fully F2F), year the course ran (2019, 2020, 2021, or 2022), semester (Fall or Spring), course level (lower division or upper division), and discipline of the course (COE/CNSM and CLA/CHSS). Change tests were calculated based on the deviance test (i.e., the difference in the – 2LL functions between two models) using a chi-square distribution to test if the predictor was significant (α = .05). If significant, fixed effects tests were examined for group differences. In the case of multiple group comparisons, a Bonferroni corrected alpha was employed for post-hoc tests since all possible pairwise tests were examined by switching referent groups.
Figure 4: Influence of redesigned course on intention to pursue research.
Group Comparison Results Differences in outcomes by course level characteristics: year, semester, level, and discipline were non-significant. However, college was a significant predictor for differences in some course redesign constructs. Research gains differed by college, X2(3)=7.91, p=.048. Students in redesigned courses in the COE reported higher research gains due to the course (M=4.08, SD=0.90) compared to students in CNSM courses (M=3.41, SD=0.96), t(1)=3.95, p=.009. Research intentions also differed by college, X2(3)=12.83, p=.048. Students in COE redesigned courses had higher research intentions (M=4.31, SD=1.44) than both CNSM (M=3.26, SD=1.46; t(1)=4.67, p=.005) and CHHS (M=2.99, SD=1.64; t(1)=4.68,
Research-Infused Courses are Effective for Online and In-Person Education 1867
p<.001) redesigned courses. Lastly, students in COE redesigned courses had higher research career perceptions (M=3.22, SD=0.56) compared to students in CNSM (M=2.82, SD=0.57; t(1)=4.68, p<.001) and CHHS (M=2.72, SD=0.70; t(1)=4.68, p<.001) redesigned courses. Additionally, students in CLA redesigned courses also had higher research career perceptions (M=3.14, SD=0.56) compared to those in the CHHS redesigned courses, (M=2.72, SD=0.70 t(1)=4.68, p<.001). No differences were found in course redesign goals by college.
Modality When examining differences by the modality of the course (see Table 4), there were no statistically significant differences between courses that were virtual and courses that were face-to-face for research skills, (X2(1)=0.22, p=.641), research intentions X2(1)=1.30, p=.254, research career perceptions (X2(1)=2.05, p=.152) or course redesign goals, X2(1)=0.08, p=.778.
Table 4. Descriptive statistics of research outcomes by modality.
Table available in the original PDF paper.
Modality Research Gains
Research Interest
Course Redesign Goals
Research Career Perceptions Virtual (Online/Hybrid) M 3.67 3.45 2.87 2.94 SD 1.04 1.55 0.62 0.60 N 299 300 313 312 Face-to-face M 3.41 3.35 2.87 2.75 SD 0.94 1.49 0.58 0.57 N 314 305 318 318 Total M 3.54 3.40 2.87 2.84 SD 1.00 1.52 0.60 0.59 N 620 612 637 637
Note. Only one course was hybrid (N=7).
CONCLUSION
Overall, redesigned courses had a broad impact on student’s ratings of research skill gains. Students reported moderate to good gains in understanding the relevance of research in their discipline and skills important to research, such as problem-solving, understanding research papers, and interpreting research results. On the other hand, students varied in terms of the amount of gain, in two areas: Understanding what research work is like and doing well in future research courses. The redesigned curriculum may not have reached beyond the classroom for some of the more abstract and future-oriented goals for some students (e.g., if the redesign included doing hands-on research experiments in a course with known outcomes rather than those with unknown outcomes, like course- based undergraduate research experiences). This might signal that gains differed by approaches of enhancing research activities in the course redesign.
Different approaches may have different effectiveness. Differential gains may also relate to efficacy and achievement in the course. For example, students who perceived they were not doing well (lower grade) in the course may not feel confident in future research-focused classes. Remarkably, gains in research skills were similar by modality, level, year, semester, discipline and across colleges, though on average, students in the CNSM redesigned courses reported less gains than students in redesigned courses of other colleges. The latter counter-intuitive finding may be due to a higher baseline level of research exposure, knowledge and/or skills for students in CNSM that might have resulted in lower amounts of gains. A baseline measure could illuminate what might have contributed to this finding.
On the other hand, the impact of the course redesign on interest in research and the likelihood of pursuing research activities was mixed. A large majority of students reported that the redesigned course helped them understand research ethics and that they could effectively communicate research to different audiences. The results also indicated that the course stimulated interest in research for a large percentage of students, a key objective of the program. However, the redesigned curriculum may not be reaching all students in these areas, as about a third of students disagreed. Again, these results may provide evidence of the differential effectiveness of the approaches to redesign the course. At the student-level, it is difficult to address why students differed in their course experiences and/or survey responses (data were collected anonymously without demographics). Baseline data was not obtained. It could be that students have different research experiences at entry to the courses and this mixture is driving these differential results.
On average, the overall student ratings for the items that measured course redesign goals were moderately high and remarkably equal across college, modality, semester, year, discipline, and level, revealing strengths of the research across the curriculum program. Furthermore, a little more than half of students linked the redesigned research-infused course to a higher likelihood of pursuing more research-focused activities/opportunities. Group differences across colleges could explain lower ratings as students in the CNSM and CHHS redesigned courses lagged behind students in CLA and COE redesigned courses in intentions to pursue research and perception of research careers. It may be that students from CNSM and CHHS colleges are more likely to be interested in pursuing experiences like internships that would be beneficial for advanced degrees and/or careers in applied fields (e.g., a degree from a professional school, a medical career, etc.) rather than research-focused activities compared to CLA and COE.
The redesigned courses’ most notable impact was on the students’ positive perception of researchers/scientists. Hands-on research-infused activities may not only build research skills but also enhance students’ understanding and appreciation of the role of a researcher. The impact of the course redesign on impressions of science and research is particularly important given the climate of misunderstanding of the role of the researcher and distrust/mistrust of science and research (e.g., medical research, climate
Research-Infused Courses are Effective for Online and In-Person Education 1869
research; Rowland et al., 2022, etc.). Furthermore, in line with expectancy- value theory (Eccles et al., 1983) increasing the value of a researcher could also relate to a potential increase in future motivation to pursue research activities, a goal of RAC.
In summary, both online and in-person redesigned courses contributed to strengthening the research foundation of students and provided evidence that the course design award successfully contributed to the goals of strengthening training, interest, and preparation for more demanding research-focused opportunities. Course redesign award also offers a cost-effective mechanism to integrate research within an existing curriculum that can have broad impact on students.
ACKNOWLEDGMENT
This research was supported in part by the National Institute of General Medical Sciences of the National Institutes of Health under Award Numbers; UL1GM118979; TL4GM118980; RL5GM118978. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
REFERENCES
Bangera, G., & Brownell, S. E. (2014). Course-based undergraduate research experiences can make scientific research more inclusive. CBE Life Sciences Education, 13(4), 602–606. https://doi.org/https://doi.org/10.1187/cbe.14-06- 0099 Eccles, J. (1983). Expectancies, values, and academic behaviors. In J. T. Spence
(Ed.), Achievement and achievement motives: Psychological and sociological approaches (pp.75–146). W. H. Freeman. Enriquez, A. G., Pong, W., Shahnasser, H., Mahmoodi, H., Chen, C., Zhang, X., Teh, K. S., & Rentsch, N. P. (2015). Assessing the impact of research experiences on the success of underrepresented community college engineering students [paper presentation]. 122nd American Society for Engineering Education Annual Conference & Exposition, Seattle, Washington, United States. https://peer.asee.org/23591 Hall, A. K., Miklos, A., Oh, A., & Gaillard, S. D., (2016). Educational
Outcomes from the Maximizing Access to Research Careers Undergraduate Student Training in Academic Research (MARC U-STAR) Program. Retrieved on July 10, 2022 from: https://www.nigms.nih.gov/News/reports/Documents/MAR Cpaper031416.pdf Jones, C. K., & Lerner, A. B. (2019). Implementing a course-based undergraduate research experience to grow the quantity and quality of undergraduate research in an animal science curriculum. Journal of Animal Science, 97(11), 4691–4697. https://doi.org/https://doi.org/10.1093/jas/skz319 Lopatto, D. (2004). Survey of Undergraduate Research Experiences (SURE): First
findings. Cell Biology Education, 3(4), 270–277. https://doi.org/10.1187/cbe.04– 07-0045 Rowland, J., Estevens, J., Krzewi´nska, A., Warwas, I., & Delicado, A. (2022). Trust and mistrust in sources of scientific information on climate change and vaccines: Insights from Portugal and Poland. Science & Education, 31(5), 1399–1424. https://doi.org/10.1007/s11191-021-00304-0
Vu, K.-P. L., Mendoza, R., Chun, C.-A., Dillon, J., & Kingsford, L. (2023). The CSULB BUILD Scholars Program: A research-intensive, upper-division program to broaden and diversify the behavioral and biomedical research workforce. Understanding Interventions, 14(1), 1–37. Weston, T. J., & Laursen, S. L. (2015). The Undergraduate Research Student Self-
Assessment (URSSA): Validation for use in program evaluation. CBE Life Sciences Education, 14(3), 1–10. https://doi.org/10.1187/cbe.14–11-0206
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Design, Engineering, and Computing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-35-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional