Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Teaching Programming during COVID lockdowns – a positive lesson

Queen’s University Belfast, United Kingdom ↗
Publication year
Digital Object Identifier10.54941/ahfe100887

Abstract

Teaching Programming during COVID lockdowns – a positive lesson. As an immediate response to the economic impact of COVID, in 2020 regional UK governments promoted short-term funded university courses to enable individuals that were redundant or furloughed to reskill in the area of software development. One of the courses developed was a novel Post Graduate Certificate in Software Development (Cert), based on a well-established MSc in Software Development (conversion) course (MSc SD) delivered at a local University. The course attracted over 700 applications, enabling the application of higher entry criteria than the MSc SD. Differences in the entry criteria and delivery format (entirely online for Cert) presented a unique Action Based research opportunity. This paper initially reports on course performance of the Cert in comparison to the MSc SD. It then provides an analysis of impact and differences within the enhanced entry criteria for the Cert. Providing discussion and recommendations for entry criteria and delivery mechanisms for similar courses.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

INTRODUCTION

As an immediate response to the economic impact of COVID, in 2020 the Northern Ireland Assembly (regional UK government) promoted the concept of short-term funded university courses to provide individuals, facing redundancy or furlough, the opportunity to reskill in the area of software development. This initiative was driven in part by the long-standing need to provide highly qualified software engineers to satisfy industry demand for computing graduates, which has steadily increased over the past several years.

However, support for this measure necessitated an equivalent reactive response from a local university to provide suitable courses, available at short notice and at a sufficient level to facilitate local industry requirements. This resulted in the development of a postgraduate certificate course (Cert) in Software Development, derived from the first year of a well-established established MSc in Software Development (conversion) course (MSc SD) delivered at Queen’s University Belfast. In practical terms, the Cert and MSc SD had similar course entry requirements, namely 1). non-computing degree 2). classified at least of 2:2 standard and 3). the passing of an Aptitude test, thereby minimising duplication of administration in terms of the admission process.

The course eventually enrolled 163 students, with the regional government providing 150 fully funded places for the Cert with the remaining being funded by the university. However, the course attracted over 700 applications, requiring the course admissions team to apply higher entry criteria, based on previous academic qualifications and aptitude test scores. Specific emphasis on the use of aptitude testing to filter applications has necessitated scrutiny to determine effectiveness in identifying candidates with the best chance of a successful course outcome. This has particular importance given the nature of funding for the Cert initiative.

Previous research has reported aptitude testing as a good predictor of successful course performance, with students with previous STEM backgrounds and higher classification of previous degrees regularly outperforming non-STEM students in the MSc SD course.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

It became apparent that with such a large number of applications for the special offering of the Cert, shortlisting of applications using the normal baseline for entry into MSc SD program would not be enough. That is where the aptitude testing and degree profile were used to determine the final list of the successful applications for entry into the program.

The use of aptitude testing in this process was also important as there is evidence that aptitude testing is a good predictor of programme success. The critical role an aptitude test plays in academic admissions has been well discussed in the literature in a similar context. As an example in a study conducted by Kuncel & Hezlett (2007) conclude that standardised (aptitude) tests are effective predictors of performance when combined with a record of prior academic performance (Rahmen et al,2017).

Research into the validity of standardised testing for academic admission is typically not subject-specific. However, it can be argued that the composition of aptitude tests for some subjects such as Computer Science, would require a more tailored approach. In a subject-specific example, Cambridge University makes use of a Computer Science Aptitude Test (CSAT) to predict student performance (Partington, 2011). In this approach, tests are configured to address different areas of aptitude, such as logical thinking or mathematical ability but do not require students to take all tests available, allowing students to focus on what they are specifically interested in. This mode of testing is underpinned by the belief that “it is important to recognize that there is no single aptitude or set of aptitudes that qualifies a candidate for a subject”. However, the validation of this approach is based on post-delivery analysis: “Among known predictors, the CSAT has had the highest correlation with later undergraduate performance in the Cambridge course (aka Tripos), achieving a correlation coefficient of 0.492.”

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

In terms of degree background, most professional software developers have a bachelor’s degree (or higher). Moreover, a Stack Overflow survey (2018) indicates that 75% have a degree with 80% having studied Computer Science or Software Engineering as a major, with much of the remainder from STEM disciplines (Foster et al, 1999). While, the absence of a STEM background does not mean a lack of aptitude for programming, the promotion of STEM education does enhance inherent potential and goes some way to explain dominance in the employment of software developers. A major factor in this reality is that STEM Education promotes the solution of real-world problems using science, technology, and logic and develops aspects of Computational Thinking (CT) (Denning et al, 2019), a key ability in learning to program.

While some non-STEM students can become proficient programmers, the absence of CT education (and experience) can go some way to explain the disparity in employment and lower outcomes in MSc SD results over the past several years. Additionally, McGowan et al (2017) has also demonstrated that students that hold higher class degree classifications (1st) outperform others by +4.5% in the MSc SD.

Due to lockdowns, the normal face-to-face teaching format of the MSc SD was suspended and replaced by online delivery for the PGCert, retaining similar timescales for delivery and assessment structures as the MSc SD. The changes to the delivery mechanism, coupled with enhanced entry criteria applied against the PG Cert raises several issues worthy of consideration in the context of both programs: i. The overall impact of fully online delivery on module outcomes ii. The relevance of previous degree (discipline) and degree classification in determining outcome. iii. The relevance of aptitude test scores.

To address these issues, this paper presents several statistical comparisons between the PG Cert and previous MSc SD outcomes. Specifically, the comparison is made between the Programming module which is double weighted, compulsory, and present in both qualifications.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

This research paper provides a comparative analysis of the outcomes of the PG Cert and MSc SD Programming modules. It then provides an analysis of the impact of the enhanced entry criteria for the Cert. Specifically to answer the research questions:

  • Does online delivery adversely affect programme scores?
  • Do previous degree type (STEM vs. Non STEM) and classification affect programme scores?
  • Does aptitude test score predict programme scores?

METHODOLOGY

The Cert cohort consisted of 163 students (Male=93, Female= 70) all holding previous non-computing degrees of at least 2:1 standard. All students had met the socio-economic course entry-level, that being that they had at the time of application been furloughed or made redundant due to COVID. Due to the oversubscription of the Cert after shortlisting based on normal MSc entry criteria including passing of the aptitude test the enhanced criteria was applied. Applicants were ranked based on degree type (STEM or Non-STEM), degree classification and aptitude test score. As such the final demographics of the Cert cohort are summarised as STEM (n=102, 1st=30, 2:1=69) and Non-STEM (n=61, 1st=24, 2.1=34). The Cert Programming module was delivered entirely online but with the same content, learning outcomes, and contact hours (lectures and labs) as the MSc Programming module. The assessments for both modules covered all learning outcomes and were independently approved by an external examiner.

RESULTS

The following section reports on the outcomes of the Cert Programming module and makes a comparison between several themed areas between the same module and the historic outcomes from the comparable MSc SD Programming module. General performances In order to respond to the research to the research question “Does online delivery adversely affect program scores?”, in addition to a direct comparison with module scores it is worth considering several Key Performance Indicators measurements commonly used to quantify quality in university courses. Specifically pass rate, dropout rate, lecture attendance as well as student feedback. As detailed in Table 1 and illustrated in Fig. 1 the PG Cert outperformed the MSc Programming module in overall module score and it also had a 100% pass rate. However, it had an almost 30%

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 dropout rate. Table 1. Detailed module performances

Overall score Pass rate (fail to pass module)

Dropout rate (withdraw from the module) PG Cert 69.1% 100% 29.9% MSc SD 63.5% 95.6% 4.3%

lecture attendance

drop out rate

pass rate

overall score

0% 20% 40% 60% 80% 100% 120%

MSc SD PG Cert

Figure. 1. General module performances comparing the PG Cert and MSc

programming module

The student evaluations rated the Cert at 4.9/5.0 for overall quality (teaching, assessment, feedback, resources, organisation) which is similar to the historic score for the MSc. The attendance at the online lectures was on average 83.5%. This is better than MSc SD module with historical lecture and labs attendance being recorded at 71.2%.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Except for the dropout rate, the Cert either compared favourably or outperformed the MSc. It is not uncommon for fully funded courses to have a high dropout rate. Overall based on these indicators it is therefore unlikely that the online delivery of the Cert adversely affected the outcomes. However, this conclusion must also be weighed against the cohort intake standard. The oversubscription of the course enabled higher entry criteria including aptitude test score, degree classification, and resulted in a skew towards a cohort with STEM backgrounds. The following sections provide an analysis of the Cert concerning the performance of these group factors.

Aptitude tests, degree classifications, and previous degree types Previously published research on the MSc SD programming outcomes showed that students with a STEM degree outperformed Non-STEM by +7.1% [7]. It also demonstrated that students that hold higher class degree classifications (1st) outperform others by +4.5%. In addition McGowan et al (2021) has also shown a clear and positive correlation between Aptitude test scores and programming scores in the MSc SD. However, Fig. 2 illustrates that there was little to separate the STEM and Non-STEM groups in the Cert for programming scores, with no correlation, found between aptitude test scores and programming scores [Fig. 3]. Nor did previous degree classification appear to influence programming scores.

80

70

60

50

40

30

20

10

0

aptitude test score % programming score % 1st (programming score %)

2:1 (programming score %)

STEM Non STEM

Figure. 1. Comparisons of STEM vs. NON STEM, degree classifications and aptitude test score averages and scores.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Figure. 2. Programming score for Cert vs Aptitude test score

DISCUSSION

The results section established that in general, the PG Cert outperformed the MSc SD, but degree type and classification within the Cert cohort had little bearing on the Programming score. However, the relevance of previous degree and degree classification in determining the outcome is likely to have still been a significant influence on the positive overall outcomes. Entrance to the MSc SD is normally on a rolling first-come-first severed basis, subject to the baseline course entry requirements. Consequently, the MSc cohort is normally an approximate 50:50 mix of STEM and Non-STEM, with a predominant number of 2:2 degrees. The average historical scores in the aptitude test for MSc cohorts range from 48% to 52%. The enhanced entry requirements shifted this normal demographic profile for the Cert.

510 applicants of all degree classifications attempted the aptitude test, with the following outcomes: STEM average=51.58%, pass rate=81.9% and Non-STEM average=48.4%, 71.5% pass rate. The STEM degrees holders did better in the aptitude test. This was especially true for the higher classifications, for example, almost all the STEM group with a 1st passed the aptitude test whereas the same Non-STEM group had a 27% failure rate. It, therefore, followed that with more of the STEM applicants met the selection criteria with the breakdown of the Cert cohort as STEM (n=102 and Non-STEM (n=61).

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

It would appear that the higher proportion of STEM and higher degree classifications that predominated the PG Cert positively influenced the outcomes. Although the results would suggest that there is little to separate higher degree levels at STEM and Non-STEM for module outcomes. Although, as consistently demonstrated with the MSc cohort, lower-order degrees and degree background show a statistically significant negative influence on module and course outcomes.

These findings are reflective of the general body of research that a STEM background is better preparation for learning how to program and that higher previous degree holders outperform those with lesser qualifications in MSc computing con-version courses. It does also uniquely suggests that high-performing students based on previous degree classifications will generally continue to perform highly regardless of degree background. The paper identifies that aptitude tests have a role to play in helping determine which students have an aptitude for programming and will likely succeed in programming-based courses. Given the high demand for computing courses, this research sets a benchmark for performance reflective of intake quality and demographic mix that may be used to determine future course entry criteria.

The online delivery of the PG Cert did not appear to adversely affect outcomes. However, it may have positively affected the gender balance in the cohort. Almost 43% of the cohort were female, which compares favourably with a normal average of 24% in the MSc SD and broadly accord computing courses worldwide (McGowan et al (2017). Qualitative feedback from the students on the course was very positive with the flexibility of online delivery being one of the main positives. There was anecdotal evidence it was this flexibility that attracted and supported females to the course. This is a complex area but worthy of future research scrutiny.

In terms of outcomes for the funders of the course, the attainment of the overall qualification of 70% of the cohort is a notable achievement. The PG Cert became a role model for other similar funded courses in the region. Many of the students, around 50-60% proceeded onto the full MSc SD. The investment from the funders was around EUR 2,000. The students that complete the MSc SD are highly employable with on average each graduate providing on average 1.2 to 1.5 EUR million in lifetime salary to the local economy.

CONCLUSIONS

Despite the general reservations of online delivery, this paper concludes that the Cert cohort outperformed all other previous MSc SD cohorts in the programming module. The attendance at online lectures for the Cert was higher than traditional average attendances at face-to-face MSc SD lectures. The Cert attracted a large number of females (43%). Considering the evolving nature of COVID on university education and the policy

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 decisions that university and government bodies are now being forced to consider concerning face-to-face, blended, and fully online learning this paper presents a timely benchmark for education delivery policy and practices.

REFERENCES

Denning, P., Tedre, M.: Computational thinking. Cambridge (2019) ISBN: 9780262536561 Foster, I., Kesselman, C.: (1999) The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufmann, San Francisco Kuncel, N.. & Hezlett, S. A (2007) “Assessment: Standardized Tests Predict

Graduate Students’ Success”, in Science (American Journal for the Advancement of Science) Partington, R. (2011) “Predictive Effectiveness of Metrics in Admission to the University of Cambridge”, Internal Report, Cambridge University Admissions Research Working Party, 2011. Retrieved from https://openclimb.io/csat/#csat-explained. McGowan, A, Hanna, P, Greer, D & Busch, J (2017) “Learning to program – choose your lecture seat carefully?”, Paper presented at Innovation and Technology in Computer Science Education, Bologna, Italy. McGowan, A. Sage, P. & Adhikari, J. (2021) “Pre-coures prediction of outcomes for conversion programming courses” Paper presented at the ICERI 2021. McGowan, A, Hanna, P & Anderson, N (2017), “Computing Gender Wars - A New Hope”, Paper presented at IEEE Frontiers in Education Conference (FIE 2017). , Indianapolis, United States, 16/10/2017 - 22/10/2017 Rahman, N. Reese, L. M. & Lauth. L. A (2017) “The Validity of Law School Admission Test Scores for Repeat Test Takers: 2010 Through 2014 Fall-Entering Law School Classes (TR 18-02)", The Law School Admission Council. lsac.org, 2017. Retrieved from https://www.lsac.org/data-research/research/validity-law- school-admission-test-scores-repeat-test-takers-2010-through Stackoverflow. (2018) The role of STEM in learning to program (2018) Explore Group, https://www.explore-group.com/blog/do-you-need-a-cs-degree-to-be-a- developer/bp78/

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Cover of the proceedings volume Human Interaction & Emerging Technologies (IHIET-AI 2022): Artificial Intelligence & Future Applications

Explore the complete digital volume

Human Interaction & Emerging Technologies (IHIET-AI 2022): Artificial Intelligence & Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-7923-8989-4

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →