Thursday, September 25, 2014

ACARA Update on Tailored Testing

ACARA has just released the following report:

National Assessment and Surveys Online Program: tailored test design 2013 study

Tailored test design is sound, feasible and more engaging
Today ACARA is releasing the National Assessment and Surveys Online Program: tailored test design 2013 study.
Key findings from completed research by ACARA, which was funded by the Australian Government Department of Education, have revealed that the tailored test design – a key concept in the move to online assessment – is sound, feasible and more engaging for students.                                    
The research also found that as many as 50 per cent of students will experience an enhanced assessment experience, particularly high- and low-achieving students.
ACARA General Manager, Assessment and Reporting, Dr Stanley Rabinowitz, said today, ‘The tailored test design and online assessment in general have been proven effective in the NAPLAN context.

‘We will get better quality results that are returned quicker, which will benefit governments, schools, parents, teachers and students. We expect to see more engaged students and better informed teachers and parents.’

As students concluded the trials of online tailored test design, they reported feeling more positive and accomplished, regardless of their abilities and educational background. The trials also provided an opportunity for students to be assessed by tests catering to their needs, as well as more accurate and timely diagnostic information about student learning needs.
‘ACARA is continuing the research into moving assessment online, and further research results will be available in 2015. The research enables ACARA to continue to fine tune our approach to moving assessment online. We need to keep investing in the research so that when the Education Council makes its decision about the country’s move to online assessment, ACARA, along with our partners, is ready to go,’ said Dr Rabinowitz.
‘The sooner we get online, the quicker the benefits can appear for governments, schools, parents, teachers and students. This research shows us the future of assessment for Australian school students, and we know it works. We now need to get on and implement it.‘
Read the research paper and a summary of the research paper on the National Assessment Program (NAP) website.
Background to the research program
In 2012 ACARA was funded by the Australian Government Department of Education to conduct research to inform decisions on the transition of the National Assessment Program – Literacy and Numeracy (NAPLAN) from paper-based to computer (online) assessments.                                    
The initial phase of research, undertaken in 2012, investigated how the test delivery mode (paper versus computer) affected student performance in and engagement with the current NAPLAN tests. The second phase of research, conducted in 2013 and summarised here, investigated the feasibility of the enhanced adaptive test design for NAPLAN online assessments.
Learn more about the research program on the NAP website. 
Multistage adaptive test design (the ‘tailored test design’)
Multistage adaptive test design (the ‘tailored test design’) is where the test difficulty is adjusted to students’ needs. After students respond to a set of test items, they are provided with a subsequent set of items that best suits their observed ability (that is, achievement level). In the proposed tailored test design, students go through three stages containing item sets (testlets) of varying difficulties to complete the test as illustrated in Figure 1: 

Figure 1: Tailored test design (TTD). This figure illustrates testlets and six test pathways available in TTD
2013 main tailored test design study
During 2013, more than 250 schools of satisfactory diversity participated voluntarily in the main tailored test design study. Over 2500 students in Years 3 and 5, and 1500 students in Years 7 and 9 completed either numeracy or reading online tests. Most of the testlets used in this study were created from existing NAPLAN test items that had been rendered to suit the online delivery mode.
Learn more about tailored test design on the NAP website.

Related studies
In a related study, students from 16 mainstream schools participated in structured interviews to ascertain how they reacted to the rising and falling pattern of item difficulty, a crucial feature of the tailored test design.
The same methodology was used in a separate study that investigated whether the tailored test design can accommodate the assessment needs of students with socio-educational disadvantage.                                                                          
In another study, ACARA collaborated with the Northern Territory Department of Education to collect information about the extent to which the proposed tailored test design provides a better testing experience for Indigenous students and students in remote communities. In this study, online tests were administered in eight Northern Territory schools, including two very remote schools.
Key findings
The key findings from these studies are as follows:
  • Results of the tailored test design studies show that the delivery of multistage branching tests for NAPLAN online is sound and feasible, and that these tests offer better measurements of student performance, particularly for high- and low-achieving students. The results show that the current measurement model can be used to construct a NAPLAN online measurement scale.
  • The psychometric analyses also show that further work is required to finalise the measurement aspects of the tailored test design; in particular, testlet boundaries require further refinement.
  • The tailored test design and the proposed branching mechanism work effectively to adapt to the different ability groups. Consequently, well-targeted tests can be administered to different ability groups, thus increasing measurement precision.
  • The investigation of cognitive and behavioural engagement of students with the tailored test design showed that multistage testing will provide an opportunity for all students to be assessed by tests catering more fully for their assessment and learning needs.
Third phase of the research program
In August 2014 ACARA began the third phase of its research program with the aim to further refine the measurement aspects of the tailored test design for reading and numeracy tests. This study also includes trialling of grammar, punctuation and spelling test items delivered online, including students listening to spelling items through headphones. The results of this program will be available in 2015.