Assessment Literacy Self-Study Quiz #8
by Tim Newfields
This ongoing column features questions about testing, statistics, and assessment in a quiz format to promote greater assessment literacy.
Suggested answers to the problems below are online at http://jalt.org/test/SSA8.htm.
Part I: Open Questions
An article by Higginbotham (2009, pp. 15-18) in a recent issue of The Language Teacher attempted to highlight
the value of extended task based instruction. Support for this instructional approach was based on a single questionnaire about a
carol singing activity using that instructional framework. One month after the activity, 69 of of the 80 students who participated
in that activity completed a Likert questionnaire and vocabulary test with merely five words from the Christmas carols they had
supposedly memorized. Based on those results, the author suggests activities such as this lead to "deep processing of the language" (p. 15).
What confounding issues are present in this study? How could the validity and reliability of studies such as this be enhanced?
Also, how should the graph to the right probably have been revised for academic publication?
Source: Higginbotham, G. (2009). Event-based learning: The benefits of positive pressure. The Language Teacher. 33 (1) 15-18.
[ p. 26 ]
At most (if not all) universities in Asia, multiple entrance pathways exist. One widely used pathway is the entrance by recommendation system
In this system, a limited number of students from a select number high schools are allowed to recommend canditates for admission to a specific university.
Those select schools are often, but not always, affiliated with each given university in some way. In many cases, admission decisions are based on a
submitted essay and a very short screening interview. Applicants generally have a month or so to compose their essays and they usually solicit the
help of parents and/or teachers when composing and revising their essays. Moreover, most of the oral interview questions are known in advance and many
students are able to churn out memorized answers to the most common questions.
What specific test validity issues are compromised through this process? What viable alternatives to the current entrance by recommendation (suisen nyuushi) system exist?
What are the pros and cons of balancing the correct answer choice sequences in the answer sheets of fixed-response exams? What alternative ways of organizing multiple-choice key answers currently exist?
Source: Harrell, F. & Banfield, J. (2005, March 8). R Graph Gallery Database: Box-Percentile plot.
Retrieved April 5, 2009 from http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=77
In an ETS-funded research report, the TOEIC Bridge (Reading Section) was found to have a .84 correlation with a government run test of mathematics ability among 198 high school students in Columbia as the graph below suggests. What would be an appropriate interpretation of these results?
Source: Sinharay, S. et al. (2008). Establishing the Validity of TOEIC Bridge Test Scores for Students in Colombia, Chile, and Ecuador.
ETS Research Report 08-58. Retrieved on November 12, 2009 from http://www.ets.org/Media/Research/pdf/RR-08-58.pdf