Goldsmiths - University of London

Context

Transferability (science, geography) and scalability (d&t)

In e-scape phase 2 we had successfully built a prototype system that enables us to run and design & technology test activities in 11 schools across England.  This had resulted in 250 performance portfolios in the website and these had been assessed using a Thurstone Pairs methodology.  Learners were enthusiastic about using the system in schools and the reliability of the subsequent assessments was significantly higher than is possible using conventional approaches.

However, there were two limitations with the phase 2 system:

Firstly, it operated only in design & technology, and this raised the question of its transferable value into other subjects.

Secondly, the phase 2 tests had been run as a research project - with the research team operating the system in schools.  This was not a scalable model for national assessment. It was necessary for such a national system to be operable by teachers in their own classrooms.

The brief for phase 3 derived from these two imperatives.

Brief

The brief for phase 3 has two elements:

• Concerning the transferability of the e-scape system in the curriculum.  After a good deal of discussion with QCA, Becta and Awarding Bodies, it was agreed that we would focus additionally on science and geography.
• Concerning the scalability of the system.  For the system to be operable for national assessment purposes it is not sufficient to have the system run by the research team (as we had in phases 1 and 2).  It would be necessary to prove that it could be operated by teachers working in their own classrooms, studios, laboratories and workshops.  And then we would also have to show that having created web-portfolios, those same teachers could undertake the necessary assessments of the work.

Methodology

The transition from 'timeless' portfolios is significant in several ways, but centrally the issue concerns direct evidence of process-centred capability.  Real-time approaches for the collection of performance data have been used for some years in the worlds of management and computer science, where system evaluation has entailed the forensic analysis (step-by-step / minute-by-minute) of sub-elements of the performance of systems.  Our approach has been to develop this approach withing the learning and assessment context of classrooms.

Phase 3 work evolved through several steps:

• creating subjects teams in geography and science
• development and trialling of tasks (science, geography and d&t)
• development of the technology to facilitate task evolution (authoring tool); to enable teachers to run activities in schools (EMS); and to manage the pairs judging (pairs engine)
• running test activities (geography, science and d&t) in schools across England and Wales
• conducting the judging and analysing the outcome
  

Outcomes

The phases of the project outlined above were successfully concluded.  The software worked as specified, the teachers coped very effectively with managing the activites and the testing in 19 schools established 350 d&t, 60 science and 60 geography portfolios in the website.  The judging was then completed with reliability coefficients of around 0.95.

The key question of whether e-scape methodology provides a sustainable national system of assessment for schools.  In relation to this over-arching question, plans are advanced with Awarding Bodies to running parts of existing GCSE and Diploma qualifications in e-scape format, starting in September 2009.  With each Awarding Body we have identified a qualification and the sections/modules of it that lend themselves to e-scape methodology.  We have identified an approximate cohort of schools/centres in each case and are now planning what will be an Awarding body pilot programme monitored by Ofqual.

This will become e-scape phase 4 through which learners in schools and colleges will receive live awards on the basis of their work undertaken and captured into web-portfolios using the e-scape approaches developed over the last four years of research and development.