The United Kingdom Open University’s (UKOU) programme of introductory distance education courses, Openings, designed for a widening participation audience, was first presented in the year 2000 (Open University 2003). This practitioner-focused paper describes some factors which support (or hinder) students’ participation and progression on these courses, concentrating upon innovative assessment within the UKOU’s systems. Openings covers a range of subjects. This paper focuses upon two Openings courses in mathematics, science and technology, Breakthrough and Another Breakthrough, and the evolution of their assessment in response to feedback from students and tutors.
The courses are designed for students with low previous educational qualifications. For example, distance education enables students with caring responsibilities, disabilities or those based in geographically remote areas to participate in education. The teaching materials, tutor support and assignments are designed around the assumption that the students have been out of formal education for a number of years, and may lack confidence as well as study skills and subject knowledge. The courses balance an emphasis upon developing organisational, numeracy and writing skills with understanding the subject.
Each course lasts for 14 to 20 weeks, with a flexible timetable that is negotiated between tutor and student. There are currently three formative assignments and a fourth summative assignment that earns a successful student 10 Credit Accumulation and Transfer (CATS) points at level 1. Each student is allocated a personal tutor, who keeps in contact by telephone and marks the three formative assignments. The summative assignment is marked by a pool of script markers recruited from amongst the tutors.
The Openings programme has grown rapidly from an initial intake of 3,500 in the year 2000 to 5,850 in 2001, and 8,000 in 2002, and is expected to reach 10,000 in 2003. About a fifth to a quarter of these students have studied the two Breakthrough courses. Of the Breakthrough students in 2000 and 2001, about a fifth have low previous educational qualifications (lower than A-level) - a higher proportion than for the general UKOU intake. Compared with the general UKOU demographic, there is also a higher proportion of students with disabilities or on low incomes (receiving State benefits)(Sutton 2002).
Evaluation questions from the September 2002 presentation of Another Breakthrough asked why students enrolled - they could choose more than one reason from a list. The majority state ‘to see if I could cope with distance learning’ as the main reason. The next most popular reason is ‘interested in the subject’ followed by ‘wanted help to develop on a personal level’. Relatively few chose Another Breakthrough as ‘preparation for an undergraduate course’, although this particular course has the highest rate of progression onto undergraduate courses, compared with other courses in the Openings Programme.
The assessment strategy is generic to all courses in the Openings Programme – this section discusses aspects which support the Widening Participation aims.
The needs of the Widening Participation agenda have shaped the pedagogy and assessment design. A central assumption is that students bring a rich variety of abilities, aspirations and anxieties to their work. Student motivation is a primary concern, and the role of assessment in motivating students (or otherwise) is recognised. For example, Iphofren (1998: 37), in discussing socio-psychological theories of motivation, argues that motivation is enhanced by clarifying shared goals and encouraging individual responsibility in achieving those goals. To support this, there is clarity about the role of assessment within Breakthrough, and transparency about the requirements of each assessment task. From the start, the assignments are presented as an integral part of the student-tutor dialogue. Whilst acknowledging that many people have negative experiences of assessment, the assignment book sets out several positive reasons for including assignments in the course: to help the student make sense of what they have learnt, to try what they have learned, to give practice in presenting work, to obtain feedback from the tutor, to practise planning and organising their work and to provide evidence of their work.
The process of writing an assignment and posting to the tutor represents a significant commitment by the student. That stage marks the difference between ‘reading about a subject’ and ‘doing a subject’. General evaluation of UKOU courses has revealed an almost ubiquitous pattern of student participation – the first assignment is a watershed, so that those students who send in that assignment are far more likely to complete the course than those who do not. Openings students follow this pattern. In their study of beginning UKOU students, Rickwood and Goodwin (200: 3) report that ‘…the assessment regime was crucial to their progress not only because the students wanted success, but because they saw it as an assurance that they were capable of understanding the materials...’
Holmberg’s (1989) theory of dialogic conversation suggests that effective distance education should establish a dialogue between student and tutor. This dialogue is mediated through text, or via communications media. Openings courses incorporate the assignments into this dialogue by encouraging discussion of assignment feedback with the tutor, and requiring students to reflect upon their skills development since previous assignments. Aalto and Jalava (1998: 263) note that this type of practice reinforces the learning dialogue. Potential barriers to effective communication include the student’s possible perception of the tutor or university as a judge. Also, many students have memories of past difficulties, or specific problems ‘I can’t write properly’ ‘I can’t do percentages.’ Assessment could easily become a barrier to these students, reinforcing their fears and pre-conceptions. By writing the teaching materials around everyday examples, and developing these in the assignments, the students can find a new perspective, and have a basis for effective dialogue.
The courses have an unusually high proportion of tutor-assessed formative assessment. As Lockwood (1995) comments, in distance education, formative assessment often takes the form of self-assessment questions embedded within the course text, whilst the tutor is only involved for the summative tasks. For Openings courses, this type of activity is supplemented by tutor marked assignments which, although assessed by the tutor, do not affect the final result.
The pattern of three formative assignments followed by a summative assignment gives students a chance to practise their skills before attempting the assessment for credit. This is a vital part of the assessment strategy, as it frees the students to try out their abilities, with a second chance to improve. The relationship between student and tutor is important here – the tutor has about a month to establish a friendly and supportive relationship with the student over the telephone before the first assignment is due. The comments from both students and tutors indicate that the first barrier – handing in the assignment – is overcome by establishing this relationship.
These courses are innovative in using criterion-referenced assessment within the UKOU’s systems - the first time that this has been implemented on this scale within the university. There are several reasons for this choice:
· In Further Education, criterion-referencing is well-established for kite-marked Access courses at pre-degree level
· The alternative, norm-referencing is established within the university’s systems as percentage marking, with the implication for the student that there is an ideal ‘100%’ answer, contrary to the ethos that the course team wanted to develop.
· Criterion-referencing supports the tutor-student dialogue, emphasising the individual’s progress against a set of performance indicators, rather than judging each student against the cohort as a whole.
The last of these reasons is the most compelling. As Morgan and O’Reilly (1999: 19) suggest, ‘It would be difficult to argue for the place of norm-referenced assessment in open learning contexts, as it is very much the antithesis of open and student-centred learning.’ Criterion-referencing has the advantages of transparency, precision and clarity about the focus of the assessment. Implementing this on a large scale within university systems designed for norm-referencing has been a challenge. The course designers have resisted all attempts to apply numerical measures to the student results - all feedback, even upon the summative assignment, takes the form of a skills profile, using, for example, ‘just achieved’ or ‘well achieved’ as performance indicators against each of the pre-defined learning outcomes.
The students and tutors are given a table of learning outcomes, with a clear indication of the evidence required to meet the various performance indicators within each outcome. In the first year of presentation, there were ten learning outcomes, and only three levels of achievement. Feedback from tutors made it clear that there were too many outcomes. When marking work, the tutors found it difficult to decide which outcome to apply in a given situation, and students were confused by the detailed lists of feedback. Tutors also requested more levels of achievement, allowing them to give more nuanced feedback - some tutors had even invented their own performance indicators in commenting upon the formative tasks, to make the apparently rigid performance criteria more flexible. In response to these comments, the course team reduced the number of outcomes to four, which covered the main skills and knowledge that the students would need for any subsequent work. The reduced number of outcomes also meant that all outcomes were assessed in every piece of work, giving students an opportunity to practise their skills and improve upon earlier achievements. The number of performance indicators was increased from three to four, allowing tutors more scope in profiling the students’ work.
Reflection upon your own progress is a relatively high level skill – meta-cognitive – and some Breakthrough students fall into the trap of recording their feelings and gratitude to the tutor rather than analysing their strengths and weaknesses with regard to the skills being assessed. Experience has shown that this type of reflection needs to be clearly targeted at specific aspects of the student experience, with detailed guidance about how to go about the task. Encouraging students to cite evidence of achievement from their earlier work has proved fruitful. Very general questions only served to confuse the students.
Requiring students to reflect upon their progress and performance in previous assignments does bring their attention to areas that need more work. For a student who has been out of formal education for many years, realising that they have understood a concept, but need more practice in expressing their ideas in writing can be a revelation. Reflection allows students to recognise unexpected abilities or weaknesses – for example, students who found questions about calculating mean, median and mode straightforward in terms of numerical manipulation may have made slips in their arithmetic, or not explained their working clearly. Fear of mathematics is one area students refer to repeatedly - students report progress with their confidence as well as abilities, although needing to refer back to course materials for methods.
The assignment tasks on Breakthrough are designed so that students have to explain or justify their answers, as well as carry out numerical manipulation. They combine numeracy with subject content, mathematical problem-solving and written reports. This leads students away from the preconception that there is a ‘right’ answer, giving more scope for interpretation, and more evidence for the tutor to see how well the student has understood the concepts behind the numbers. The emphasis is upon process and skills, including study skills, rather than inculcating extensive factual knowledge.
Presenting work for others is a vital skill for any student, especially those in distance education, for whom written assessment is the main evidence of progress. Students can find that found that their presentation, whether in words or symbols, needed more work during the course. For numerical work, a relatively large proportion of students are able to carry out calculations, but have more difficulty in presenting their working using conventional notation. In extreme cases, some students invent their own notation. These students benefit from the formative assignments, and recognise their progress in the reflective task at the end of the course.
As a result of feedback from tutors, the assessment tasks are now more integrated around a theme, rather than a series of exercises on individual techniques. There is a greater emphasis upon students explaining why and how they have used techniques and information. This allows similar skills to be revisited in later assignments building a more coherent skills ramp. The reflection tasks have more guidance, and more meaningful questions which focus upon how the student has improved specific skills and understanding as a result of studying the course materials and comments from their tutor. Overall, there is more emphasis upon process, both within the questions and within the tutor-student dialogue created by the assignments. In giving comments, tutors explain their decisions, and place more emphasis upon next steps for the student, rather than seeing the completed assignment as a product.
In the reflective sections of their assignments, many students report that their confidence has increased as a result of studying Breakthrough, and evaluation data show that students with low previous educational qualifications perform better on subsequent courses than those who have not studied Openings. The assignments themselves give evidence of improved written communication, and more successful attempts to engage with the academic discourse.
Students also report changes in the way in which they approach their studies, such as spending more time planning and reading the course materials, rather than skimming through. Tutors report that students’ organisational skills improve as the course progresses. Deeper barriers may underlie problems with scheduling work, such as lack of confidence, or life circumstances that inhibit regular study. By allowing students and tutors to negotiate individual deadlines, rather than setting a fixed date for the whole cohort, there is increased motivation for students to keep to their personal schedules and send in assignments at agreed times. Some students have recorded changes in their habits, such as reading a book rather than watching television.
Tutors on Openings courses have more support with the students individually than on other UKOU courses, where face-to-face group tutorials are the norm. As a result, tutors report greater understanding of the individual needs and circumstances of the students. This increased communication also emphasises the process within assessment and support, reducing the sense of an assignment as a product of learning. Tutors are encouraged to use a friendly style of communication, both over the telephone and in writing, and to be aware of the range of students’ circumstances, and many see their work on the course as integral to their own staff development. Although some tutors have previous experience of outcomes-based assessment, for many this is the first time they have used anything other than percentage marking. For some, the assessment tasks are novel, combining several disciplines, so they need to step beyond their specialist subject. This is unusual in a short course, and has stimulated peer support between tutors from different Faculties, fostering a distinctive multidisciplinary approach.
The assessment has become more responsive to student and tutor needs, building upon the experiences of both groups to refine the tasks and overall strategy. The learning outcomes are more focussed on the most relevant areas for student development, simplifying the use of criterion-based assessment. Reflection has been built into the assessment tasks, with more guidance giving a clearer purpose for the students. The course designers have a deeper understanding of the needs of the widening participation audience within a distance learning environment.
The UKOU’s record-keeping, grading and quality assurance systems are all based upon percentage norm-referencing, and have limited flexibility due to the standardisation and automation required to manage distance education on a large scale. Colleagues across the university have been very supportive in adapting their work to the needs of these criterion-referencing, but there are several areas where the course designers have had to compromise due to inflexibility of current processes, which are too complex to adapt easily. The rapid evolution of the assessment has been a factor here – in a university where planning a course three years before student use is a norm, changing the assessment strategy on a yearly basis requires particular energy and determination.
Assessment is central to many students’ experience of distance education. This paper has outlined some factors which can transform it from an obstacle into a gateway. The most significant are to make the assessment integral to the learning process, use transparent criterion-referenced assessment, and encourage communication about assignment feedback as part of the student-tutor dialogue.
Aalto P and Jalava M (1995) ‘Implementing experiences from small-scale courses to large educational systems’ in Lockwood F (ed) Open and distance learning today, London, Routledge.
Holmberg B (1989) Theory and practice of distance education, Routledge, London
Iphofen R (1998) ‘Understanding motives in learning: mature students and learner responsibility’, in Brown S, Armstrong S, Thompson G (eds) Motivating students, Kogan Page, London
Lockwood F (1995) ‘Students’ perceptions of, and response to, formative and summative assessment material’ in Lockwood F (ed) Open and distance learning today, London, Routledge.
Morgan C and O’Reilly M (1999) Assessing open and distance learning , London, Kogan Page.
Open University (2003) Openings course website http//www.open.ac.uk/openings (accessed 20/4/03)
Rickwood P W and Goodwin V (2001) ‘A year as a student: a study of early learning experiences and their significance in the context of the future of supported open learning’, in The future of learning - learning for the future. Abstracts and conference proceedings of the 20th ICDE world conference April 1-5 2001, Düsseldorf, Germany
Sutton H (2002) Comparison of openings programmes 2000 and 2001 - internal report, The Open University, Milton Keynes