Michael-brian Ogawa, Sara Mostowfi, Jung Hyup Kim, Kristen Shinohara, Danielle Oprean, Curtis Ikehara, Yuanyuan Gu, Leah Ten Eyck, Martha Crosby (2025). Impact of Retrospective Confidence Prompts on Students’ Metacognitive Awareness and Skills. In: Human Factors in Design, Engineering, and Computing. AHFE Open Access, vol. 199, pp. 1412–1419. https://doi.org/10.54941/ahfe1006953
The capacity to comprehend and control one's mental processes, such as observation, assessment, and problem-solving, is known as metacognitive awareness. This research focused on a group of undergraduate students taking a foundational programming course during the semester. The aim was to explore how retrospective confident judgment questions influence students' metacognitive awareness—a crucial skill in self-regulated learning. To assess their metacognitive awareness, we employed the Metacognitive Awareness Inventory (MAI), a tool designed to illuminate the nuances of their cognitive processes and self-reflection (Schraw & Sperling, 1994). This is a validated self-report tool that measures two fundamental dimensions—knowledge of cognition and regulation of cognition, as well as their subcategories—and was used to assess metacognitive awareness. This research involved administering the MAI at two distinct points in time: the initial assessments took place in January and February, which we designated as the "beginning" phase, while the follow-up evaluations occurred in April and May, referred to as the "end" phase.
Jobs in computer and mathematical occupations are expected to grow by 12.9% through 2033 due to a demand for professionals in sectors such as AI, data analytics, and cybersecurity (U.S. Bureau of Labor Statistics, 2023). Enrollment in computer science programs has increased at a high rate to account for this demand (Zweben & Bizot, 2021). Although enrollment increased at high rates, degrees awarded did not increase at the same rate and stayed relatively stable. Attrition became a major factor impacting the development of professionals in these fields. Thus, it is critical to consider factors that support students’ growth in these fields. An internal factor that
supports academic achievement is students’ metacognitive awareness; the ability to comprehend, monitor, and control one’s mental processes, such as observation, assessment, and problem-solving (Dunlosky & Metcalfe, 2008). Learners with high levels of metacognitive awareness perform better academically than those who do not possess this self-awareness.
Metacognitive awareness includes two dimensions, knowledge and regulation (Schraw & Dennison, 1994). The knowledge dimension includes declarative, procedural, and conditional knowledge, while the regulation dimension includes planning, information management strategies, monitoring, debugging strategies, and evaluation. A short description of each of the elements are listed below:
Declarative knowledge: Understanding of one’s factual and conceptual knowledge
Procedural knowledge: Understanding of one’s approaches to learning and task process
Conditional knowledge: Knowing when to use learning strategies
Planning: Ability to organize oneself, select approaches, and determine goals
Information management strategies: Ability to organize and retrieve information
Monitoring: Following up with oneself regarding content and misconceptions
Debugging strategies: Determining inaccuracy of knowledge/content and implementing corrective actions
Evaluation: Determining effectiveness of learning approaches Reflecting and continuous evaluation of knowledge and regulation are critical to academic achievement (Dunlosky & Metcalfe, 2008). Academic achievement also depends on the situation and context to inform regulation approaches to learning (Händel & Dresel, 2022; Jameson et al., 1990). The process is referred to as metacognitive judgments, or how well learners align their performance with their knowledge and regulation techniques. These judgments are key components of metacognitive monitoring. This process includes concepts such as Retrospective Confidence Judgments (RCJs), which includes learners’ perceptions of how well they performed.
An important area of metacognitive monitoring and control is the accuracy of these assessments (Morphew, 2021). Awareness of one’s thought process supports their ability to be cognizant of actions, motivations, and the application of the skills across contexts (Ehrlinger et al., 2008). This awareness comes from both retrospective and prospective vantage points (Baars et al., 2014). Retrospective monitoring focuses on evaluating one’s recent prior performance, while prospective monitoring targets perceptions of performance on a similar future task. In both cases, students need to have accurate perceptions of their prior performance or likeliness of accomplishing a future task. Students with lower ability to make these judgments accurately typically perform lower due to the misalignment of their abilities and task challenge.
To improve retrospective monitoring, Retrospective Confidence Judgments (RCJs) can allow learners to practice this type of monitoring. RCJs have been beneficial in enhancing students’ self-monitoring and self- regulation of learning abilities (Guo & Kim, 2022). This approach focuses on the accuracy of past responses, where students compare their confidence to performance. Comparing these areas lead to students better understanding their current state and supports needed to achieve their goals which leads to improved metacognitive awareness (Mostowfi et al., 2025).
With the benefits of metacognition to learning, we build on prior work (Mostowfi et al., 2025) by investigating how retrospective RCJ impacted students’ metacognitive awareness in computer-based learning (CBL) scenarios. This study provided insights into the use of RCJs with different levels of CBL students to improve our understanding of approaches to improve metacognition in students. These findings have the potential to be applied to various instructional context and may provide additional insights to advance metacognitive awareness for learners.
The eight aspects of metacognition serve as the primary framework for this study. We quantitatively used the Metacognitive Awareness Inventory (MAI) to measure the two dimensions across the eight aspects of metacognition with our participants before coding the data using the same framework.
METHODS
Participants The study was conducted at a research university in the western United States. Participants were recruited from two classes, an introductory computer science course for non-majors and an upper division computer science course. The courses selection enabled the research team to draw comparisons between students with different backgrounds and experiences.
Introductory Computer Science Course The introductory course focuses on computational thinking for non-majors. It included 121 participants from 18 majors including multiple sectors of business, social sciences, natural sciences, and humanities. These students’ age ranged from 17 to 69 years old with a median value of 19. The majority were 18–22 years old.
The course was taught using a hybrid model with a focus on computer- based learning (CBL). It included a lecture and lab that each met in-person once a week and asynchronous content. Each lecture included confidence prompts to support students identifying their current knowledge level and RCJ targeting how much they learned from each session. The students were also surveyed in the middle of the semester to reflect on their learning to identify how best to meet their learning goals in the second half of the term.
Upper Division Computer Science Course The upper division course, Cognitive Science, is an upper division computer science major elective course. It was comprised of computer science majors typically in their senior year. There was a total of 17 participants in this group with an age range from 21 to 33 years old with a median age of 23. Although
Impact of Retrospective Confidence Prompts 1415
the age range was smaller than the introductory course, a majority were in the 21–24 age range.
The instructional method included a hybrid online structure that included asynchronous and synchronous online sessions. The asynchronous sessions focused on online readings and postings to support the synchronous sessions. The synchronous sessions included both group-based and individual discussions to expand on the course content and develop a deeper understanding of course material. Both sessions included opportunities to improve metacognitive awareness through RCJ.
Research Design A sequential mixed methods design was implemented to address the research goals (Figure 1). The first phase of the study was a quantitative analysis of the MAI to determine which student groups had a significant increase in the elements of metacognitive awareness. A t-test was used for the two groups, majors and non-majors with the dependent variable being each of the aspects of metacognition. After determining the student groupings with significant gains, we conducted a qualitative analysis of surveys to identify the practices and approaches that had the greatest influence on their growth. The eight aspects of metacognition were used as primary codes for the qualitative analysis.
Figure 1: Sequential mixed methods design.
Due to the sequential mixed methods approach, we collected the data in stages (Figure 2). For the non-major group, we collected the MAI from all of the students at the beginning of the term. To promote metacognitive awareness, the instructor used RCJ prompts weekly during in-class lectures. Students were initially asked how much they know about a topic at the start of the class and each class had students indicate the amount they learned to align their background knowledge and content learned with the learning objectives. In the middle of the semester, the students were surveyed about their learning to reflect on their current progress and how they may adjust their learning approaches to meet their learning goals. After the mid semester survey, the instructor continued with the RCJ prompts. At the end of the term, the MAI was administered with the end of semester course evaluation.
After the MAI was distributed to the major group, they received instruction about metacognition and how it applies to their learning. They responded to asynchronous prompts including course material and RCJ weekly which led to in-class discussions focusing on course content. In-class discussions about course content led to RCJ-based prompts for discussion and internalization of learning. At the end of the term, the participants completed the MAI and a course evaluation survey.
Figure 2: Data collection sequence.
Figure available in the original PDF paper.
RESULTS
The t-test results show significant MAI differences for the non-major group for the Evaluation and Planning components for the MAI (Table 1), whereas the CS major students did not include significant differences between (Table 2). For Evaluation, non-major students improved their metacognitive awareness by 8.04% (p<.05). They also increased their Planning awareness by 5.79% (p<.05).
Table 1: The comparison between non-CS student from beginning to the end of the course in terms of evaluation and planning.
Table available in the original PDF paper.
Category Time N Mean Std. Dev Std. Error F Ratio p-Value Evaluation Jan - Feb 121 0.6721 0.2545 0.0231 5.27 0.0225* April - May 99 0.7525 0.2622 0.0263 Planning Jan - Feb 121 0.7285 0.2098 0.0190 4.369 0.0378 * April - May 99 0.7864 0.1983 0.0199 Information Management Strategies Jan - Feb 121 0.8024 0.1864 0.0169 0.893 0.3458 April - May 99 0.8262 0.1849 0.0186 Debugging Strategies Jan - Feb 121 0.7652 0.2231 0.0202 0.077 0.7815 April - May 99 0.7734 0.2261 0.0227 Comprehension Monitoring Jan - Feb 121 0.7741 0.2028 0.0018 1.082 0.2994 April - May 99 0.8030 0.2080 0.0209 Conditional Knowledge Jan - Feb 121 0.7140 0.2349 0.0213 0.024 0.8771 April - May 99 0.7192 0.2570 0.0258 Procedural Knowledge Jan - Feb 121 0.876 0.1681 0.0152 0.599 0.4397 April - May 99 0.856 0.2145 0.0215 Declarative Knowledge Jan - Feb 121 0.789 0.1818 0.0165 0.973 0.3249 April - May 99 0.813 0.1743 0.0175
Table 2: The comparison between CS student from beginning to the end of the course in terms of evaluation and planning.
Table available in the original PDF paper.
Category Time N Mean Std. Dev Std. Error F Ratio P-Value Evaluation Jan - Feb 14 0.69 0.1945 0.0519 0.005 0.9445 April - May 11 0.69 0.2668 0.0804 Planning Jan - Feb 14 0.755 0.1305 0.0349 0.0746 0.7872 April - May 11 0.740 0.1402 0.0423 Information Management Strategies Jan - Feb 14 0.821 0.1368 0.0365 0.2792 0.6023 April - May 11 0.791 0.1513 0.0456 Debugging Strategies Jan - Feb 14 0.671 0.2016 0.0538 0.5590 0.4622 April - May 11 0.727 0.1618 0.0487 Comprehension Monitoring Jan - Feb 14 0.714 0.1875 0.050 0.0199 0.8890 April - May 11 0.727 0.2723 0.082 Conditional Knowledge Jan - Feb 14 0.726 0.1680 0.0449 0.9072 0.3508 April - May 11 0.787 0.1507 0.0454 Procedural Knowledge Jan - Feb 14 0.875 0.2137 0.057 2.024 0.1683 April - May 11 0.75 0.2236 0.067 Declarative Knowledge Jan - Feb 14 0.767 0.1459 0.0389 0.005 0.9431 April - May 11 0.772 0.1921 0.0579
Impact of Retrospective Confidence Prompts 1417
These statistically significant findings led the researchers to focus qualitative examination on survey data from the non-major group. Although the eight aspects of metacognitive awareness were used as a coding structure, the primary themes emerged from the evaluation and planning constructs. The remaining six aspects did not include sufficient evidence to support additional themes. Planning included two major themes, time management and course attendance. Time management targeted starting assignments early, organization of assignments, and scheduling study time. A student noted, “I would tell myself to stop procrastinating and start making a better study/sleep schedule for the rest of the semester.” This quote highlighted the importance of starting assignments early and reserving time to study while considering their health. A majority of the responses that focused on course attendance considered changing their behavior to attend class regularly as a critical preliminary step to enable them to implement other metacognitive learning strategies. Overall, time management covered approximately 67% of the coded data.
The evaluation construct accounted for approximately 20% of the coded data with 3 major themes. The themes included pre-class preparation, in-class engagement, and post-class follow-up. Pre-class preparation considered how students were ready for live class sessions. Videos, readings, and assignments were available for students to prepare for class sessions. Many found these valuable in the middle of the term and modified their behavior accordingly. A student stated, “I would tell myself to be more prepared for in-class workdays so that I can ask all my questions then instead of trying to do my work in class and possibly going over to office hours which didn’t always happen.” This highlighted the value of preparing for workdays, where students could utilize “live”support for areas that were unclear to them rather than studying during work time and depending on office hours for additional queries. In-class engagement emphasized the importance of maintaining and active learning state by being engaged throughout class sessions. An illustrative quote was, “Try to participate more during classes by thinking about content, trying to complete practice problems on my own, and asking more questions.” This student evaluated their learning and discussed the importance of the strategies they used in class and how they should refine them in the future. The final theme targeted post-class follow-up, where participants discussed the value of practicing what they learned to better internalize content. Another student discussed the importance of post-class learning by stating, “Keep practicing as we learn more to make sure I got it and make sure I stay ahead on assignments.” This theme demonstrated the importance of how students approach content after initially learning it in class and how it can help them in building their knowledge. A quote that exemplified the interaction between these three themes was when a student indicated that they should, “Prepare early, familiarize yourself with the SKILLS. stay focused in class and take time to review after class.”
The themes across the planning and evaluation constructs showcased the students’ metacognitive thinking processes and changes in behavior. Many of the participants indicated that their changes in these areas were due to the
RCJ practices; especially the type of advice they would give to themselves in the middle of the semester after seeing how their thinking and performance were related.
DISCUSSION
The results indicated that introductory non-major computer science students had significant improvement in metacognitive awareness for planning and evaluation. Although upper division computer science majors did not have a significant improvement in metacognitive awareness, we believe that it may have been due to their metacognitive awareness being established during the first few years of their undergraduate program of study. It is possible that purposeful implementation of RCJ in early college courses may have a larger influence on improving metacognitive awareness. The statistical results from the upper division majors were not significant, which could have been due to a smaller participant pool than the introductory non-major group. We would like to continue our study of this group by utilizing the qualitative data to better understand the nuances of furthering metacognitive awareness using approaches such as RCJ.
The qualitative themes provided deep insight into the growth of metacognitive awareness of the introductory non-major group and how the practices were influenced by in-class RCJ activities. This group found time management and course attendance to be critical components of the planning element of metacognitive awareness. These themes were a practical approach to improve study habits (attending class) and managing one’s time to practice what one learns. Similarly, the evaluation themes were practical as they discussed the importance of pre-class preparation, in-class engagement, and post-class follow-up. These are metacognitive approaches that upper division students may have developed though practices that are not likely via metacognitive awareness practice such as RCJ. Thus, we would like to investigate the differences that exist when students purposefully interact with their learning in this way through RCJ rather than not having metacognition as a core method of learning.
The study opened additional avenues for future research. We plan to conduct detailed qualitative analysis on the student populations to tease out details that may not be feasible with a quantitative approach. These future studies may allow us to ascertain the differences between groups of students such as our introductory non-majors and upper division majors using within subject approaches to account for individual differences. This has the potential to illuminate themes that are only possible with larger scale within subject approaches. We are also considering comparison of RCJ approaches that compare the differences between physical (hands- on) and mental constructs to determine the affordances and constraints of metacognitive approaches.
CONCLUSION
This study revealed the impact of RCJ metacognitive prompts on metacognitive awareness for introductory non-major students. The findings
Impact of Retrospective Confidence Prompts 1419
highlighted the significant growth in the planning and evaluation constructs with their practical implementations focusing on time management, course attendance, pre-class preparation, in-class engagement, and post-class follow- up. These themes were approaches that students used as a result of their metacognitive awareness. Therefore, considering what they know, what they learned, and how they can improve are critical retrospective RCJ practices that supported students in developing their metacognitive awareness in planning and evaluation approaches.
REFERENCES
Baars, M., Vink, S., Gog, T., Bruin, A., & Paas, F. (2014). Effects of training self-assessment and using assessment standards on retrospective and prospective monitoring of problem solving. Learning and Instruction, 33, 92–107. Dunlosky, J. and Metcalfe, J. (2008). Metacognition. Sage Publications. Ehrlinger, J., Johnson, K., Banner, M., Dunning, D., & Kruger, J. (2008) Why
the unskilled are unaware: Further explorations of (absent) self-insight among the incompetent. Organizational behavior and human decision processes, 105(1), 98–121. Guo, W. & Kim, J. (2022). Investigating Academic Performance Using an ARbased Learning Environment with Retrospective Confidence Judgments. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting. SAGE Publications Sage CA: Los Angeles, CA. Händel, M. & Dresel, M. (2022). Structure, relationship, and determinants of
monitoring strategies and judgment accuracy. An integrated model and evidence from two studies. Learning and Individual Differences, 2022. 100: 102229. Jameson, K., Narens, L., Goldfarb, K., & Nelson, T. (1990). The influence of nearthreshold priming on metamemory and recall. Acta Psychologica, 73(1), 55–68. Morphew, J. (2021). Changes in metacognitive monitoring accuracy in an introductory physics course. Metacognition and Learning, 16(1). 89–111. Mostowfi, S., Crosby, M. E., Ogawa, M. B., Ikehara, C., Gu, Y., Oprean, D., &
Kim, J. (2025). Augmented Reality vs. Computer-Based Learning: Understanding Metacognitive Awareness Differences. In: Schmorrow, D. D., Fidopiastis, C. M. (eds) Augmented Cognition. HCII 2025. Lecture Notes in Computer Science, vol 15779. Springer, Cham. https://doi.org/10.1007/978–3-031–93727-9_7. Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness.
Contemporary Educational Psychology, 19, 460–475. U.S. Bureau of Labor Statistics. (2023). Employment Projections - 2022–2032.
Human Factors in Design, Engineering, and Computing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-75-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional