Shiken: JALT Testing & Evaluation SIG Newsletter
Vol. 13 No. 2. May 2009. (p. 24 - 34) [ISSN 1881-5537]
PDF PDF Version

Assessment Literacy Self-Study Quiz #7
by Tim Newfields

South Korean English Test Sample Reading Problem This ongoing column features questions about testing, statistics, and assessment in a quiz format to promote greater assessment literacy. Suggested answers to the problems below are online at

Part I: Open Questions

  1. Please examine the following test item and then answer these questions:(1) What skills is this task likely tapping into? (2) In what ways (if any) might this item be inappropriate in a general university undergraduate EFL entrance exam?

    This item was taken from South Korea’s 2008 College Scholastic Ability Test [2008 Daehak suhakneungnyeok siheom] – one of the three types of exams that high school applicants seeking university admission must take.

    The task is to determine the correct sequence of three passages (A-C) that follow a 41-word paragraph opening.

    Source: Korea Institute of Curriculum and Evaluation. (2008). 2008 Haknyeondo Daehaksuhakneungryeoksihum Munjeji Oegukeo (Yeongeo) Yeongyeok – Holsuhyeong Oegukeo (Yeongeo) Yeongyeok 7. [2008 College Scholastic Ability Test Foreign Language (English) Field 7]. Retrieved April 7, 2009 from

  2. [ p. 24 ]

  3. Explain the difference between these concepts: (1) local test item independence and (2) test item unidimensionality.

  4. Look at the box-percentile plots below, then briefly describe how the two groups differ. Also, what advantages do these projections have over common boxplots? Finally, how can such box-percentile plots such be drawn?
    Two box-percentile plots
    Source: Harrell, F. & Banfield, J. (2005, March 8). R Graph Gallery Database: Box-Percentile plot. Retrieved April 5, 2009 from

  5. In 2005 Hirsch proposed a simple index as one way of quantifying faculty research output and granting promotions. This index, known as the h-index, combines two metrics: (1) the number of papers one has written [k] and (2) the number of times those papers have written have been cited [d]. The precise formula is:
    H index formula

    Hence an author with a h-index of 12 has at least 12 publications, all of which have each been cited at least 12 times. That author might have a few publications with more citations, and many with less – but 12 is the largest number of articles-with-equal-number-of-citations for this person.

    What are some advantages and disadvantages of Hirsch’s ranking proposal?

    Source: Hirsch, J. E. (2005, September 1). An index to quantify an individual's scientific research output. Physics and Society, (DOI: 10.1073/pnas.0507655102) Retrieved April 3, 2009 from


    [ p. 25 ]

    Part II: Multiple Choice Questions

    1. Which of the following statements is true about Poisson distributions?

    2. The graph below is an example of a

      Graph for Question 2

    3. Which of the following is least likely to improve the reliability of an examination?

      [ p. 26 ]

    4. One difference between the KR-20 and KR-21 is:

    5. What is an ideal item facility index (a.k.a. “p-value” or “item difficulty index”) for a 4-option, multiple-choice norm-referenced test item? What about a true-false norm-referenced test item? What if both these items were designed for a criterion-reference test?

      NOTE: At least two answers for each sub-question are possible.


    NEWSLETTER: Topic IndexAuthor IndexTitle IndexDate Index
    TEVAL SIG: Main Page Background Links Network Join
    last Main Page next
    HTML:   /   PDF:

    Quiz #1: Qs   / Qs   / As   *    2: Qs   / As   *    3: Qs   / As   *    4: Qs   / As   *    5: Qs   / As   *    6: Qs   / As   *    7: Qs   / As   *    8: Qs   / As   *    9: Qs   / As   *    10: Qs   / As   *    11: A HREF="SSQ11.htm">Qs   / As  

    [ p. 27 ]