How do you Japanese teachers of English feel about guns in America? Do you
have definite opinions? Perhaps strong negative views? I'm sure some of you do. Well,
the reverse is true too: at least some foreigners have strong negative views about
university entrance exams in Japan. Is that "cultural imperialism"? Of course not. It is
perfectly alright for people to have opinions about aspects of other countries without
them becoming cultural imperialists, unless, of course they try to impose their ideas on the other country.
To tell you the truth, despite the fact that I have become known as an outspoken
critic of the university entrance examinations in Japan, I must admit that I started out
thinking I should just ignore the entrance exams because they are basically a Japanese
problem and therefore none of my business. What changed my point of view? Well,
I've been coming to Japan for about fourteen years now, and everywhere I've given lectures
or workshops, no matter what the topic was, the audience in Japan always seems to
steer the discussion around to the university entrance examinations. This went on for a
numbers of years during my frequent visits to Japan. People would ask my opinion of
the entrance exams, but what they really wanted me to do was make statements about
how bad the exams were for communicative language teaching, or how dreadful they
were for the youth of the country, or how expensive or unfair they were, and so forth.
Through all this, I staunchly argued that the entrance exams are not a problem for an
outsider to deal with. This is a Japanese problem, one that only Japanese will ever be
able to solve. People argued back that maybe I should speak out on the issue because
I can't be fired. In other words, I could say whatever I liked, perhaps stirring up useful
discussion of the issues, and then leave, without suffering the consequences that teachers in Japan might face.
Somewhere along the line, another factor began to play on my attitudes toward the
entrance examinations. I was given an opportunity over several years to observe the
effects of the entrance examinations on the two sons of a colleague here in Japan. I'll call
them Toshi and Yuki. Yuki was the older brother, so he was the first to face the problem. I
watched as Yuki, a very bright kid, took eight entrance exams that cost his parents 30,000
to 40,000 yen each over the course of one examination hell season. He failed them all.
He was crushed. Naturally, he did not give up. He became a ronin and spent a year at
an expensive yobiko cram school (another expense for his poor parents) preparing to do
better the second year. This time he picked a smaller number of universities of varying
levels of prestige. Ultimately, he succeeded in passing the exam for the international
studies department at a well-known university that I will call University A. Yuki thought
he passed that exam for that particular department because he was a returnee and the
English part of the exam was double weighted. Yuki ended up going to University A and
studying a major that didn't interest him at all, simply because that was the entrance
examination that he managed to pass. According to him, he didn't care about learning
anything; he just needed a degree from University A. He just needed that piece of paper
in order to go on to study law, his real interest.
". . . a great deal has been written about the quality and appropriateness of Japanese university English language entrance examinations."
His brother Toshi was also a bright kid. In fact, he was so bright that he refused to
go through the same nonsense that his brother had suffered. Instead, he went straight
to a community college in Canada and side-stepped the Japanese university entrance
examinations altogether. He is now studying at the University of British Columbia and
making plans to go to grad school at a prestigious university in the United States. Seeing
all this, his older brother is now also making plans to go the United States to study. Thank
you Japan for devising an entrance exam system that is driving some of your best,
brightest, and most creative people to leave for North America. We will welcome them
and give them plenty of opportunities to contribute to our society.
There is a part of me that says, maybe it would be better to keep the entrance exam
system the way it is in Japan, so that bright Japanese kids keep coming to my country.
But another part of me realizes that I am watching a terrible injustice. Not just one kid, but
many wasted a great deal of time and effort, suffering through many exams, failing, and
finally passing only to major in something that wasn't at all interesting to them. Perhaps
I should speak out. Maybe I can serve as a catalyst, and in any case, I cannot be fired.
I can just say what's on my mind and go back to Hawaii to my regular life.
That is how I decided to speak out on the issue of the English language entrance
examinations in Japan, and I will continue to do so here by (a) briefly reviewing the
literature on English language entrance examinations, (b) exploring some of the most
prominent problems with the entrance examinations, and (c) suggesting some potential
solutions to those problems.
The literature on English language entrance exams
During the last fifteen years, a great deal has been written about the quality and
appropriateness of Japanese university English language entrance examinations. For
any readers interested in the topic, I will simply list some of the key articles on the topic
and add a brief note to explain what they were about:
- Brown and Christensen (1987) discussed the possibility of including listening tests on Japanese university English language entrance examinations.
- Brown (1987) offered some ways to improve the testing of false beginners.
- Brown (1990) explained how to improve the fit of tests in language programs.
- Watanabe (1992) examined the effects of the university entrance examinations on classroom language learning strategies.
- Brown (1993) discussed one Japanese newspaper article's misinterpretation of TOEFL results.
- Brown (1995a) explored the nature of examination hell, and the social and psychological consequences of the Japanese exam system.
- Brown (1995b) furnished definitions for some of the key Japanese terminology used by students and others to describe various aspects of examination hell.
- Brown and Kay (1995) discussed the washback effect of university entrance examinations on English language teaching in Japanese high schools.
- Brown and Gorsuch (1995) brought a number of entrance examination issues to public attention in The Daily Yomiuri.
- Brown and Yamashita (1995a) analyzed the 1993 entrance examinations at 21 universities (including 10 private, 10 public, and the national Center exam).
- O'Sullivan, B. (1995) criticized Brown and Yamashita (1995a) for having a weak design and not being very useful for teachers, test users, or researchers.
- Brown and Yamashita (1995b) responded to the criticisms leveled by O'Sullivan (1995) and expanded the list of research questions which should be investigated.
- Brown and Yamashita (1995c) compared 21 of the 1994 university examinations to the 1993 equivalents.
- Brown (1996a, in a plenary speech at the 1995 JALT Conference) addressed problems with the entrance examinations and ways to solve them.
- Stapleton (1996) offered a thoughtful reaction to some of Brown's articles pointing out not so much that they were wrong, but rather that Brown was ignorant of the "Japanese" perspective.
- Yoshida (1996a) criticized Brown's (1996a) JALT plenary arguing that Brown ignored cultural differences between Japan and the United States.
- Brown (1996b) answered Yoshida's (1996a) arguments, and Yoshida (1996b) had another go at discrediting Brown's views.
- (already in the line above) Watanabe (1996a) provided a model of how the washback effects of university entrance examinations can be studied.
- Brown (1998) discussed the general effects of university entrance examinations on English language teaching in Japan.
- Brown (1999) addressed the roles and responsibilities of assessment in foreign language education in terms of the purposes, effects, roles, and responsibilities of assessment.
- Mulvey (1999) cast some doubts on the degree of influence of entrance exams on junior and senior high school reading pedagogy.
- Brown (2000a) suggested strategies for using the entrance exams to create positive washback on English language teaching in Japan.
- Brown (2000b) argued for a more flexible educational system that gives "second chances".
- Murphy (2000) talked about the "International TESOL Resolution on English Entrance Exams at Schools and Universities" which argues for sound testing practices.
- Murphey (2001) discussed some of the "non-meritorious" characteristics of the entrance examination system.
- Mulvey (2001) showed how the influence of the entrance exams has changed in recent years in terms of reforms that have resulted from demographic and economic forces in Japan.
- Cheng and Watanabe (in press) collected articles from around the world showing that washback exists in a number of contexts.
- Irinoda (forthcoming MA thesis) greatly expands on Brown and Yamashita's two studies by examining the same 21 entrance exams in each of the years between 1993 and 2001 (exactly the sort of study we hoped to inspire).
In the articles above, a number of criticisms were leveled and problems were
identified, but also, a few solutions were offered. Next, expand on the problems with
the entrance examinations and then provide my suggestions for solving some of those problems.
Problems with English language entrance exams
The questions raised about the Japanese entrance examinations were first discussed
in Brown (1999). Here, I will offer an expanded list of those questions organized into
five main topics (purposes, effects, roles, responsibilities, and perspectives) within the
larger framework of Japanese education and society as a whole.
A. Purposes of assessment
B. Effects of assessment: Watanabe (1996a & b) made a good start on researching washback effects in Japan, but clearly, many questions remain to be answered.
- Is the distinction between norm-referenced testing and criterion-referenced testing recognized in Japan?
- Which type of test are the university entrance examinations, norm-referenced or criterion-referenced?
- If the entrance examinations are norm-referenced, are they aptitude tests, or proficiency tests? Or do they serve entirely different purposes?
- Instead of language abilities, do they measure 'test-wiseness', toughness, endurance, ability to stick to a task, mindless ability to follow directions, or some other traits?
- In short, what is the purpose of the university entrance examinations? As far as I can tell the purpose is as yet unclear.
- How can any test be valid if the purpose is unclear? In other words, the first issue that needs to be resolved is the purpose of the entrance examinations.
- What are the negative effects of entrance examination washback on English teaching, course content, course characteristics, and class time?
- What test design issues could be used to affect positive washback from the university entrance examinations?
- What test content, logistic factors, and test interpretation factors could also be changed to create more positive washback effects from the entrance exams?
- Why are curriculum and testing factors not related for the most part to the university entrance examinations?
- And, how can those factors be aligned with the entrance exams?
C. Roles of assessment
D. Responsibilities of assessment [Questions raised by Messick's (1996) validity ideas]:
- How are the entrance examinations used in analyzing students' needs (if at all)? And, how are the tests meeting the students' needs? Teachers' needs? Administrators' needs? Prefectural needs? National needs?
- How can the entrance examinations be improved to better meet language program needs in the high schools of Japan?
- What can tests tell us about the appropriateness of our goals and objectives? And, what can tests tell us about how well we are meeting those goals and objectives?
- How can the entrance examinations be improved to foster better goals and objectives in the high schools of Japan?
- How sound are the entrance examinations in terms of reliability and validity?
- How can we develop better entrance exams?
- How do the entrance exams affect classroom testing? How can the entrance examinations be improved to foster better testing in the high schools of Japan?
- How do the entrance exams affect the types of high school teaching materials being used? Are the entrance examinations testing the same things that are covered in high school materials? What do the entrance examinations indicate about the effectiveness of the teaching materials?
- How can the entrance examinations be improved to foster better teaching materials in the high schools of Japan? How are the entrance exams affecting the ways teachers deliver instruction? What do the entrance examinations indicate about the effectiveness of high school teaching? How can the entrance examinations be improved to foster better teaching in the high schools of Japan?
- What do the entrance examinations indicate about overall program effectiveness? How can entrance examinations be changed to better evaluate the overall effectiveness of the junior and senior high school language programs? And, how can the entrance examinations be improved to foster better language program evaluation in the high schools of Japan?
- What evidential and consequential bases (in terms of test interpretations and test uses) should we be considering in assessing the validity of the university exams?
- What are the value implications that are important for the entrance examinations in Japan? And, how shall we deal with them?
- What are the social consequences in Japan for the entrance examinations? And, how shall we deal with them?
E. Perspectives on assessment [Based on Cronbach's (1988) perspectives on
- From a functional perspective, what are the content, truth, and worth concerns with regard to the entrance examinations in Japan?
- From a political perspective, what local and general political concerns must be considered with regard to the university entrance exams in Japan?
- How can we better explain the quality of the entrance exams, the meaning of their results, as well as the values and consequences involved in decision making so the non-specialists who use the entrance exams will make better decisions?
- Should the university entrance examinations continue to be developed separately at the hundreds of universities across Japan, indeed in the many hundreds of departments in universities all over Japan?
- Or should the testing be done using a single centralized test?
- Should the entrance exams continue to be developed by amateurs who know nothing about test design? Or rather, should the entrance exams be developed by a team of specialists who know what they are doing?
- From an economic perspective, what are the overt and hidden costs of the university entrance examinations to students and parents?
- Why are the entrance examinations so very expensive?
- What are the implications of the fact that students who fail the exam are helping to support universities from which they will never benefit?
- What are the impacts of economic factors on test design and administration?
- What would be the economic effects if the entrance exams were all canceled and replaced by one cheaper national center exam developed and maintained by experts and administered once a month throughout the year?
- Where would the universities then get the money they raise with entrance exams?
- Could alumni associations and endowment funds focus on raising money from the alumni who benefited from the university (rather than from students who fail the entrance examinations as in the present system)?
- Could the money now earned through the entrance examinations be replaced by simply having higher application fees?
- Wouldn't corporate funding be a more efficient, more lucrative alternative source of much greater funding for universities?
These and many other questions need answers in the coming years. They need to
be answered not only for the entrance examinations, but also for the many other types
of language tests used in Japan if these tests are to be productively reformed along with
the rest of the educational system in Japan (as suggested in Mombusho, 2000).
Some solutions to the problems
The literature offers a number of possible strategies for addressing the questions
listed in the previous section. These strategies are summarized here in four sets (test
design, test content, logistical, and interpretation strategies) in the hope that some or all
of these ideas will be used to help ameliorate the current negative effects of the university
entrance examinations (adapted from Brown 2000a):
A. Test design strategies
B. Test content strategies
- Base achievement tests on objectives (Hughes, 1989)
- Base the test on sound theoretical principles (Bailey, 1996)
- Design the test to be criterion-referenced, if appropriate (Hughes, 1989; Wall, 1996)
- Design the test to measure what the programs intend to teach (Bailey, 1996)
- Foster learner autonomy and self-assessment (Bailey, 1996)
- Sample test items widely and unpredictably (Hughes, 1989)
- Use more direct testing (Hughes, 1989; Wall, 1996)
C. Logistical strategies
- Assess higher-order cognitive skills to ensure they are taught (Heyneman and Ransom, 1990; Kellaghan and Greaney, 1992)
- Do not limit skills to be tested to academic areas (they should also relate to out-of-school tasks) (Kellaghan and Greaney, 1992)
- Make examinations reflect the full curriculum, not merely a limited aspect of it (Kellaghan & Greaney, 1992)
- Test the abilities whose development you want to encourage (Hughes, 1989)
- Use a variety of examination formats, including written, oral, aural, and practical (Kellaghan and Greaney, 1992)
- Use authentic tasks and texts (Bailey, 1996; Wall, 1996)
- Use more open-ended items (as opposed to selected-response items like multiple-choice) (Heyneman and Ransom, 1990)
D. Interpretation strategies
- Ensure that test-takers, teachers, administrators, curriculum designers understand the purpose of the test (Bailey, 1996; Hughes, 1989)
- Make sure language learning goals are clear (Bailey, 1996)
- Make sure teachers and administrators are involved in different phases of the testing process because they are the people who will have to make changes (Shohamy, 1992)
- Provide detailed and timely feedback to schools on levels of pupils' performance and areas of difficulty in public examinations (Kellaghan and Greaney, 1992)
- Provide detailed score reporting (Bailey, 1996)
- Provide feedback to teachers and others so that meaningful change can be effected (Heyneman & Ransom, 1990; Shohamy, 1992)
- Where necessary, provide assistance to teachers to help them understand the tests (Hughes, 1989)
- Conduct predictive validity studies of examinations (Kellaghan & Greaney, 1992)
- Consider factors other than teaching effort in evaluating published examination results and national rankings (Kellaghan & Greaney, 1992)
- Develop regional professional networks to initiate exchange programs and to share common interests and concerns (Kellaghan & Greaney, 1992)
- Have testing authorities work closely with curriculum organizations and with educational administrators (Kellaghan & Greaney, 1992)
- Improve the professional competence of examination authorities, especially in test design (Kellaghan & Greaney, 1992)
- Ensure that each examination board has a research capacity (Kellaghan and Greaney, 1992)
- Make sure exam results are believable, credible, and fair to test takers and score users (Bailey, 1996)
". . . admission by entrance exam is far from being the only way to get into a Japanese university. Increasingly, there are other possibilities . . ."
I have explored some of the problems with the university entrance examinations in
Japan and have also suggested some potential solutions to those problems. However, it
is important to recognize that to some degree these issues have become less important
in recent years. Various demographic (e.g. the plummeting numbers of applications
to Japanese universities), economic (e.g. the dwindling employment opportunities
after graduation), and social forces (e.g. the need to accommodate returnees, the
emigration of bright students like Toshi and Yuki, and so forth) are leading to changes in
admissions policies. In fact, admission by entrance exam is far from being the only way
to get into a Japanese university. Increasingly, there are other possibilities: entrance 'by
recommendation'; special policies and exams for returnees; automatic admissions for
students who attended feeder high schools; the use of multiple sources of information in
admitting students; and so forth. Clearly exams are no longer the only way to get into a
Japanese university nor are they any longer an isolated issue. The entrance examinations
that were originally designed to create a 'level playing field' for all applicants to universities
can now be circumvented in a number of ways. In other words, the situation has become
considerably more complex and perhaps even less fair than how they were originally conceived.
I would also like to make it clear that I recognize that problems in admissions decisions
are not just an issue in Japan. We have persistent problems in the American admissions
systems as well. However, using hundreds of different university entrance examinations
developed by amateurs to help raise money for the universities is not on any list of our
problems. Naturally, admissions problems also exist elsewhere in the world, especially in
Asia. Indeed the stakes are particularly high in Asia because the country that first figures
out how to test and teach English effectively may have a distinct economic advantage
over other countries. In recent years, both China and Korea have developed unified
nationwide professionally developed English entrance exam systems, and thus, they
may have developed an important advantage over Japan with its hundreds of entrance
exams developed by amateurs. It will be interesting to see if the simpler, more cogent
admissions systems in China and Korea do indeed confer an economic advantage during
the next couple of decades.
So let me end at the beginning by asking you once again: What do I think about
gun control in the United States? And more importantly, what do you think about the
university entrance examination system in Japan? I hope you have found something in
this paper that has made you think more deeply about the issues involved. The future
of Japanese young people is far too important to be left in the hands of the hundreds of
teams of amateur test developers in the many universities all over Japan. Think about
it, and don't be afraid to express your opinion when the need arises. Maybe I can get
you a job in Hawaii.
Bailey, K. M. (1996). Working for washback: A review of the washback concept in
language testing. Language Testing, 13, 257-279.
Brown, J. D. (1987). False beginners and false starters: How can we identify them? JALT Journal, 11, 14, 9-11.
Brown, J. D. (1990). Where do tests fit into language programs? JALT Journal, 12, 1, 121-140.
Brown, J. D. (1993). Language test hysteria in Japan? The Language Teacher, 17, 12, 41-43.
Brown, J. D. (1995a). English language entrance examinations in Japan: Myths and facts. The Language Teacher, 19, 10, 21-26.
Brown, J. D. (1995b). A gaijin teacher's guide to the vocabulary of entrance examinations. The Language Teacher, 19, 12, 25.
Brown, J. D. (1996a). English language entrance examinations in Japan: Problems
and solutions. In G. van Troyer (Ed.) JALT '95: Curriculum and evaluation (pp. 272-283). Tokyo: Japan Association for Language Teaching.
Brown, J. D. (1996b). Japanese entrance exams: A measurement problem? The Daily Yomiuri (Educational Supplement, February 5, 1996), 15.
Brown, J. D. (1997). Do tests washback on the language classroom? TESOLANZ Journal, 5, 63-80.
Brown, J. D. (1998). University entrance examinations and their effect on English
language teaching in Japan. In J. Kahny & M. James (Eds.), Perspectives on
secondary EFL education: A publication in commemoration of the 30th anniversary
of the Language Institute of Japan (pp. 20-27). Odawara, Japan: LIOJ.
Brown, J. D. (1999). The roles and responsibilities of assessment in foreign language
education, JLTA Journal, 2, 1-21.
Brown, J. D. (2000a). University Entrance Examinations: Strategies for creating
positive washback on English language teaching in Japan. Shiken: JALT Testing &
Evaluation SIG Newsletter, 3, 2, 4-8. Retrieved October 10, 2002 from the World
Wide Web: http://jalt.org/test/bro_5.htm.
Brown, J. D. (2000b). Give second chances in education. The Daily Yomiuri (Educational Supplement, January 3, 2000), 6.
Brown, J. D., & Christensen, T. (1987, November). Interview: James D. Brown. The Language Teacher, 11, 7, 6-10.
Brown, J. D., & Gorsuch, G. (1995). An inteview with J.D. Brown: Analyzing the value,
meaning of entrance examinations. The Daily Yomiuri (Educational Supplement, October, 30, 1995), 15.
Brown, J. D., & Kay, G. S. (1995, November). English language entrance examinations at
Japanese universities: An interview with James Dean Brown. The Language Teacher, 19, 11, 7-11.
Brown, J. D., & Yamashita, S. O. (1995a). English language entrance examinations at
Japanese universities: What do we know about them? JALT Journal, 17, 1, 7-30.
Brown, J.D., & Yamashita, S. O. (1995b). The authors respond to O'Sullivan's letter to
JALT Journal: Out of criticism comes knowledge. JALT Journal, 17, 2, 257-260.
Brown, J. D., & Yamashita, S. O. (1995c). English language entrance examinations
at Japanese universities: 1993 and 1994. In J. D. Brown & S. O. Yamashita (Eds.)
Language Testing in Japan (pp. 86-100). Tokyo: Japan Association for Language Teaching.
Cheng, L., & Watanabe, Y. (In press). Context and method in washback research: The
influence of language testing on teaching and learning. Mahwah, NJ: Lawrence Earlbaum Associates.
Cronbach, L. J. (1988). Five perspectives on validity argument. In H. Wainer & H. I.
Braun (Eds.), Test validity (pp. 3-17). Hillsdale, NJ: Lawrence Erlbaum Associates.
Heyneman, S. P., & Ransom, A. W. (1990). Using examinations and testing to improve
educational quality. Educational Policy, 177-192.
Hughes, A. (1989). Testing for language teachers. Cambridge: Cambridge University Press.
Irinoda (forthcoming MA thesis).
Kellaghan, T. & Greaney, V. (1992). Using examinations to improve education: A study
of fourteen African countries. Washington, DC: The World Bank.
Messick, S. (1996). Validity and washback in language testing. Language Testing, 13, 241-256.
Mombusho (2000). Daigaku nyuushi no kaizen ni tsuite. [About improving university entrance examinations]. Retrieved October 10, 2002 from the World
Wide Web: http://www.mext.go.jp/b_menu/shingi/12/daigaku/toushin/001102.htm.
Mulvey, B. (1999). A myth of influence: Japanese university entrance exams and their
effect on junior and senior high school reading pedagogy. JALT Journal, 21, 1, 125-142.
Mulvey, B. (2001, July). The role and influence of Japan's university entrance exams: A
reassessment. The Language Teacher, 25, 7, 11-17.
Murphey, T. (2000, June). International TESOL encourages assessment literacy among test
makers. The Language Teacher, 24, 6, 1-3.
Murphey, T. (2001, October). Nonmeritorious features of the entrance exam system in Japan.
The Language Teacher, 25, 10, 1-4.
O'Sullivan, B. (1995). A reaction to Brown and Yamashita "English language entrance
exams at Japanese univerities: What do we know about them?" JALT Journal, 17, 2, 255-257.
Shohamy, E. (1992). Beyond performance testing: A diagnostic feedback testing
model for assessing foreign language learning. Modern Language Journal, 76, 4, 513-521.
Stapleton, P. (1996, March). A reaction to J. D. Brown's recent inquiry on the English entrance exam. The Language Teacher, 20, 3, 29-32.
Wall, D. (1996). Introducing new tests into traditional systems: Insights from general
education and from innovation theory. Language Testing, 13, 234-354.
Watanabe, Y. (1992). Washback effects of college entrance examinations on language
learning strategies. JACET Bulletin, 23, 175-194.
Watanabe, Y. (1996a). Investigating washback in Japanese EFL classrooms:
Problems of methodology. The language testing cycle: From inception to washback
(Series S, Number 13) (pp. 208-239). Canberra, Australia: Applied Linguistics Association of Australia.
Watanabe, Y. (1996b). Does grammar translation come from the entrance
examination? Preliminary findings from classroom-based research. Language Testing, 13, 318-333.
Yoshida, K. (1996a). Language testing in Japan: A cultural problem? The Daily Yomiuri (Educational Supplement, January 15, 1996), 15.
Yoshida, K. (1996b). Testing the bounds of culture. The Daily Yomiuri (Educational Supplement, February 12, 1996), 15.