ACARA has just released the following report:
National Assessment and Surveys Online Program:
tailored test design 2013 study
Tailored test design is sound, feasible and more engaging
Today ACARA is releasing the National Assessment and Surveys Online Program: tailored test design 2013 study.
Key
findings from completed research by ACARA, which was funded by the
Australian Government Department of Education, have revealed that the
tailored test design – a key concept in the move to online assessment –
is sound, feasible and more engaging for students.
The
research also found that as many as 50 per cent of students will
experience an enhanced assessment experience, particularly high- and
low-achieving students.
ACARA
General Manager, Assessment and Reporting, Dr Stanley Rabinowitz, said
today, ‘The tailored test design and online assessment in general have
been proven effective in the NAPLAN context.
‘We will get better quality results
that are returned quicker, which will benefit governments, schools,
parents, teachers and students. We expect to see more engaged students
and better informed teachers and parents.’
As students concluded the trials of
online tailored test design, they reported feeling more positive and
accomplished, regardless of their abilities and educational background.
The trials also provided an opportunity for students to be assessed by
tests catering to their needs, as well as more accurate and timely
diagnostic information about student learning needs.
‘ACARA
is continuing the research into moving assessment online, and further
research results will be available in 2015. The research enables ACARA
to continue to fine tune our approach to moving assessment online. We
need to keep investing in the research so that when the Education
Council makes its decision about the country’s move to online
assessment, ACARA, along with our partners, is ready to go,’ said Dr
Rabinowitz.
‘The
sooner we get online, the quicker the benefits can appear for
governments, schools, parents, teachers and students. This research
shows us the future of assessment for Australian school students, and we
know it works. We now need to get on and implement it.‘
Background to the research program
In
2012 ACARA was funded by the Australian Government Department of
Education to conduct research to inform decisions on the transition of
the National Assessment Program – Literacy and Numeracy (NAPLAN) from
paper-based to computer (online) assessments.
The
initial phase of research, undertaken in 2012, investigated how the
test delivery mode (paper versus computer) affected student performance
in and engagement with the current NAPLAN tests. The second phase of
research, conducted in 2013 and summarised here, investigated the
feasibility of the enhanced adaptive test design for NAPLAN online
assessments.
Learn more about the research program on the NAP website.
Multistage adaptive test design (the ‘tailored test design’)
Multistage
adaptive test design (the ‘tailored test design’) is where the test
difficulty is adjusted to students’ needs. After students respond to a
set of test items, they are provided with a subsequent set of items that
best suits their observed ability (that is, achievement level). In the
proposed tailored test design, students go through three stages
containing item sets (testlets) of varying difficulties to complete the
test as illustrated in Figure 1:
Figure 1: Tailored test design (TTD).
This figure illustrates testlets and six test pathways available in TTD
2013 main tailored test design study
During
2013, more than 250 schools of satisfactory diversity participated
voluntarily in the main tailored test design study. Over 2500 students
in Years 3 and 5, and 1500 students in Years 7 and 9 completed either
numeracy or reading online tests. Most of the testlets used in this
study were created from existing NAPLAN test items that had been
rendered to suit the online delivery mode.
Learn more about tailored test design on the NAP website.
Related studies
In a
related study, students from 16 mainstream schools participated in
structured interviews to ascertain how they reacted to the rising and
falling pattern of item difficulty, a crucial feature of the tailored
test design.
The
same methodology was used in a separate study that investigated whether
the tailored test design can accommodate the assessment needs of
students with socio-educational disadvantage.
In
another study, ACARA collaborated with the Northern Territory
Department of Education to collect information about the extent to which
the proposed tailored test design provides a better testing experience
for Indigenous students and students in remote communities. In this
study, online tests were administered in eight Northern Territory
schools, including two very remote schools.
Key findings
The key findings from these studies are as follows:
- Results
of the tailored test design studies show that the delivery of
multistage branching tests for NAPLAN online is sound and feasible, and
that these tests offer better measurements of student performance,
particularly for high- and low-achieving students. The results show that
the current measurement model can be used to construct a NAPLAN online
measurement scale.
- The
psychometric analyses also show that further work is required to
finalise the measurement aspects of the tailored test design; in
particular, testlet boundaries require further refinement.
- The
tailored test design and the proposed branching mechanism work
effectively to adapt to the different ability groups. Consequently,
well-targeted tests can be administered to different ability groups,
thus increasing measurement precision.
- The
investigation of cognitive and behavioural engagement of students with
the tailored test design showed that multistage testing will provide an
opportunity for all students to be assessed by tests catering more fully
for their assessment and learning needs.
Third phase of the research program
In
August 2014 ACARA began the third phase of its research program with the
aim to further refine the measurement aspects of the tailored test
design for reading and numeracy tests. This study also includes
trialling of grammar, punctuation and spelling test items delivered
online, including students listening to spelling items through
headphones. The results of this program will be available in 2015.
