Are STEM subjects really difficult?

February saw the publication of Inter-subject comparability studies by QCA. Close on its heels (albeit in draft form) was Relative difficulty of examinations in different subjects by the Durham University research group. Both reports were commissioned broadly in response to concerns about whether standards required to achieve success in GCSEs and A-levels are the same across different subjects. (The latter report was commissioned by SCORE, an alliance of science educationalists from the Royal Society, the Association for Science Education, the Biosciences Federation, the Institute of Biology, the Institute of Physics, the Royal Society of Chemistry, and the Science Council.) 

Not a new problem

For decades, chemistry along with the other sciences and mathematics has suffered the ignominy of being a 'hard' or 'difficult' subject. While such perceptions didn't seem to put off youngsters from studying these subjects post-16 up to the mid-1970s, the same cannot be said today. True, there has been a recent upturn in the numbers of students choosing to study chemistry at degree level in the past couple of years, but overall the popularity of STEM subjects (science, technology, engineering and mathematics) in HE is in decline. 

Increasingly students are studying unusual combinations at A-level. Many apparently opt for 'easy' or 'soft' subjects because they think they are more likely to get better grades in them and thus more UCAS points for university entry. If this is the case then students may not be taking science A-levels for the wrong reasons. 

With the Government recognising the importance of people with STEM skills to the UK economy, there is renewed interest in re-opening the debate over subject difficulties. If STEM subjects are more difficult at A-level, this would give the science community a reason to lobby the Government to differentiate the 'UCAS' scores for different subjects, allocating a higher tariff to STEM A-levels. (A similar scheme operates in Australia, for example.) Moreover, the recent announcement that QCA is to be split into separate regulatory and curriculum bodies (see Educ. Chem., 2008, 45 (2), 37), coupled with the fact that there will be new A-levels this September, adds to the timeliness of the debate.  

The QCA report

The QCA report is based on a judgement approach, done by subject experts, with a background in assessment. Such an approach tends to rule out comparing disparate subjects, such as media studies and chemistry. Instead comparisons are made between related subjects, where there are teachers and other experts with sufficient expertise in them.  

Two studies compared selected specifications at GCSE, AS and A2 for: geography and history; and biology, chemistry, and physics. Two other studies compared selected A-level specifications across biology, psychology and sociology; and across English literature, history and media studies. The experts evaluated the demands implied by the syllabus materials for each subject at each level, and compared candidates' work across each subject.  

The report finds that while chemistry was the most demanding of all three sciences at all qualification levels, the performance standards revealed no substantial or consistent differences between the sciences at any level. Overall, QCA concludes that subjects at the various levels are generally in line, and no immediate action is required to even things out. In particular, the report states, 'there was little evidence that the A-level subjects sometimes described as "soft" were any less demanding than their more established counterparts'. 

The Durham report

The analyses done by the Durham researchers paint a different picture. The remit from SCORE was to do a rigorous review of available evidence on the relative difficulties of different subjects as measured through external assessment across the UK. The researchers reviewed five statistical methods, as well as the judgement method, that have been used to assess the difficulty of subjects over the past 30 years. They also looked at national examination results for England in 34 subjects at GCSE and in 33 A-level subjects taken in 2006 to find out the extent to which the five statistical methods are consistent with each other, and investigated whether STEM subjects are more difficult than others. 

The statistical methods either compare the results achieved by the same candidates in different examinations, or they compare the grades achieved in different exams by students judged to be similar on the basis of their performance in a particular test. The results from these studies consistently point to the STEM subjects and languages being more difficult than other subjects at A-level and to a lesser extent at GCSE, and a divide between 'soft' and 'hard' subjects. Specifically, most methods put the range between the easiest and hardest subjects at A-level around two grades and at GCSE around 1.5 grades.  

As to the differences emerging from the two reports, Dr Robert Coe, one of the authors of the Durham report, told Education in Chemistry, 'It comes down to the fact that the different methods define "difficulty" slightly differently. There is a lot of literature which criticises the statistical approach to assessing the relative difficulties of different subjects, some of which we have referred to and addressed in the report. There tends to be less criticism of the judgement approach, which I think has serious limitations. The main one is that people can't actually judge how difficult a task is for someone else. Even experts cannot look at an exam question and its mark scheme and make a judgement about how difficult it is to get different marks in one exam compared to another from one year to the next, even if it's in the same subject and the same specification. In the end it comes down to a statistical comparison based on their experience. And how difficult something is can also be influenced by quite subtle factors that the experts will have no idea about and even if they did they wouldn't be able to make judgements on - for example, the conditions in which the examination was taken, how well the students were prepared for that specific exam, if the candidates were feeling unwell etc. When you take these factors into account the judgement method looks more problematic and unreliable'.  

Whether SCORE acts on the findings of the Durham report, however, remains to be seen. The science community could lobby the Government to scale the grades for specific purposes such as the UCAS tariff, or make the grades statistically comparable, which would, according to the report, compromise standards in the different subjects in time, or leave well alone. 

The science community will need to decide whether the strong evidence presented by the statistical methods is robust enough to support change, and the implications of any changes. And, importantly, would addressing the problem of difficulty through the assessment system encourage more people to take STEM subjects, or would it lead to a new set of different problems?