Multiple-choice tests - are they fit for purpose?

Until the 1950s and early 1960s, only a minority of young people took O-levels in science subjects and there was no serious ambition to provide a worthwhile science education for all 14-16-year-olds. Typical O-level chemistry questions emphasised recall, see, for example, Box 1. To encourage more students in grammar schools to take the sciences post-16, a new curriculum was developed with the support of the Nuffield Foundation in the mid-1960s. 

The Nuffield O-levels: a radical approach 

The Nuffield O-level courses in biology, chemistry and physics were characterised by a reliance on practical work carried out by students and a spirit of inquiry that infused the teaching. The aim of these courses was to encourage conceptual understanding and to reduce the emphasis on factual recall. However, if they were to be successful, then new approaches to assessment, which reflected the aims of the new courses, were essential and need to be developed.  

Project teams for the Nuffield-funded O-level programmes, working in partnership with the three leading examining boards (now called awarding bodies), took up the challenge. The Nuffield chemistry team collaborated with John Matthews and other leading members of the London Schools Examination Board (one of the precursors of Edexcel) to develop new models of assessment for the chemistry course.¹ 

Aims and impact of assessment 

At the time all exams were taken at the end of the course. Matthews and his team, concerned about the impact of styles of examining on teaching, wanted to do as much as they could to ensure that the new examinations would encourage teachers to adopt the aims and methods of the Nuffield programme. To provide a structure to their thinking, they adopted a simplified version of Bloom's taxonomy of educational objectives.² They wanted to ensure that the exams not only rewarded knowledge in the form of the recall of facts, procedures, patterns and principles but also tested: 

• comprehension - the ability to translate information as well as routine problem-solving in familiar situations; 
• application - testing the skills needed to apply knowledge in unfamiliar situations; and 
• higher skills - ie analysis, synthesis and evaluation. 

By adding these dimensions to the test grids used to monitor and evaluate the examinations, the team ensured that the test papers were educationally valid and true to the spirit of the Nuffield approach. Their work led to innovations in examining which have stood the test of time. Remarkably, perhaps, the specimen written papers for Edexcel's new 2008 AS-level chemistry course use essentially the same types of assessment that were used for Nuffield O-level chemistry in the 1960s - fixed-response (mainly multiple-choice) items,³ structured questions and free-response items. 

Fixed-response items in chemistry examinations 

Chemistry examiners have found that fixed-response items of various kinds have a valuable and valid contribution to make to testing. 

A collection of multiple-choice items, for example, makes it possible to sample student learning over the full range of a course. The broad coverage can enhance the reliability of a test.  

Multiple-choice items of various format have been tried out. Originally, the examination boards pre-tested such questions to ensure that only those of appropriate difficulty and discrimination were used in test papers. The most common type is the standard item which requires students to pick the best answers from four or five options. The multiple-completion format is also appropriate in chemistry because there are many instances where a full answer cannot be reduced to a single statement (Box 2).  

It is relatively easy to devise fixed-response items to test knowledge, comprehension and the application of learning in unfamiliar contexts. Generally, however, attempts to devise items to assess the higher skills of analysis, synthesis and evaluation have led to contrived formats that have not stood the test of time.  

Fixed-response items in teaching and learning 

Fixed-response items also have a place in teaching and learning. Evidence from the Assessment Reform Group (a group of researchers set up by the British Educational Research Association (BERA) as a Policy Task group on assessment), for example, has highlighted the benefits of assessment to inform teaching ⁴ and enhance learning.⁵ A project carried out by the Evidence-based Practice in Science Education (EPSE) Research Network explored the use of fixed-response items for these purposes in science education.⁶ The team, led by Robin Millar at the University of York, worked with teachers to develop a bank of diagnostic questions (see, for example, Box 3) to monitor and evaluate students' learning in four areas, one of which was chemical change.⁷ They explored how such questions can be used to support teaching and learning of the central scientific concepts that feature in the Key Stage 3 programme of study. 

The EPSE research team identified several effective ways for teachers to use the questions in their classes. The findings suggest that fixed-response items can help to provoke discussion in small groups of learners who can then be challenged to explain and defend their answers. At the beginning of a lesson, a single item can be used to get students thinking. Alternatively, one or two of the questions can be the focus of the final lesson to check their understanding of the key ideas introduced in the lesson. Another approach that some teachers have tried successfully, is to use a few diagnostic questions at the start of a new topic, to find out what students can recall of previous learning. Monitoring attainment in this way helps to stress the importance of understanding fundamental ideas rather than simple recall of facts, rules and conventions that all too often dominate in external tests and examinations. 

2006 GCSE assessment 

Since the 1960s, multiple-choice items have never completely disappeared from public examinations, and have been used in the main as short, end-of-module tests. The cost of administering pre-tested multiple-choice items before they are operational, however, has meant that this process has largely been dropped. Now the awarding bodies for GCSE and GCE examinations depend on learning from the data from previous tests when compiling new examinations.  

The development of new technologies that incorporate on-screen scoring and the electronic processing of marks means that it is now possible to provide tests with a mixture of fixed-response and other types of test questions relatively easily. From the point of view of the awarding bodies, fixed-response items are attractive because they can be scored by relatively unskilled markers, leaving specialist teachers the task of dealing with responses where judgement is needed to award the marks correctly.  

With the large numbers of young people taking GCSE Science, the provision of end-of-module tests has become popular. So the pressure on awarding bodies to adopt fixed-response items for such short periodic tests in the new GCSE Science courses is understandable. However, they should not have been allowed to exclude opportunities for students to demonstrate the higher skills in Bloom's taxonomy. 

The GCSE Science curriculum reflects the rationale for compulsory school science that was set out in Beyond 2000.⁸ At the core of the new curriculum is the aim of developing scientific literacy for all secondary school students. The Qualifications and Curriculum Authority's (QCA) programme of study⁹ states, among many other things, that students should be taught to: 

• evaluate methods of data collection and consider their validity and reliability as evidence; 
• present information, develop an argument and draw a conclusion, using scientific, technical and mathematical language, conventions and symbols and ICT tools; 
• consider how and why decisions about science and technology are made, including those that raise ethical issues, and about the social, economic and environmental effects of such decisions. 

Educational aims such as these call for the higher skills of analysis, synthesis and evaluation. These are not skills that can be appropriately tested with fixed-response items. Despite this some of the awarding bodies have adopted models of assessment for GCSE Science that rely almost exclusively on multiple-choice and other fixed-response formats, such as matching and sequencing exercises. Such forms of assessment do not reflect the true spirit of the new curriculum. 

The introduction of the 2006 GCSE Science courses is as radical a change as that intended by the Nuffield O-level courses aimed at the grammar school population in the 1960s. Unfortunately, the thought that has gone into devising the courses and to creating new resources for teaching and learning has not been complemented by appropriate investment in new approaches to assessment. This work remains to be done and is urgently needed.  

Andrew Hunt is co-director of the Twenty-first century Science Project. 

Related Links

Nuffield Curriculum Centre
The Nuffield Foundation has supported innovations in teaching and learning for over 40 years

University of York Department of Educational Studies
Information on the Evidence-based Practice in Science Education (EPSE) Research Network, with EPSE teaching resources including diagnostic questions

King's College London Department of Education & Professional Studies
Selected publications of Professor Jonathan Osborne, including link to pdf of 'Beyond 2000: Science Education for the Future'

Qualifications and Cirriculum Authority
National Curriculum Key Stages 3 and 4

Twenty First Century Science
Twenty First Century Science is a set of GCSE science courses developed to give all 14 to 16 year olds a worthwhile and inspiring experience of science