The Interface Between Interlanguage, Pragmatics and Assessment: Proceedings of the 3rd Annual JALT Pan-SIG Conference.
May 22-23, 2004. Tokyo, Japan: Tokyo Keizai University.
The interface between interlanguage pragmatics
by Andrew D. Cohen (University of Minnesota)
This article offers in written form most of the points that I made in my plenary at the JALT Pan-SIG Conference in the
Spring of 2004. The article will first speak to the importance of pragmatics, and especially speech acts,
in the teaching and learning of a second language (L2). Second, it will speak to the challenges of testing pragmatic behavior.
Finally, suggestions for action are offered.|
There is a basic premise in interlanguage pragmatics — that it is not enough just to know the equivalent words and
phrases in a second language (L2). Learners need to determine the situationally-appropriate utterances, namely:
Yet a powerful influence is working against the appropriate application of the L2 forms —
namely, how we do it in the native language and in other languages we know. The extent to which a person is able to cope with
pragmatic challenges has been referred to as their pragmatic ability, namely:
- what can be said,
- where it can be said,
- when it can be said,
- how to say it most effectively.
the ability to deal with "meaning as communicated by a speaker (or writer) and interpreted by a listener (or reader)
...[and to interpret] people's intended meanings, their assumptions, their purposes or goals, and the kinds of actions
(for example, requests) that they are performing when they speak. (Yule, 1996, p. 3-4)
Continuing with basic definitions, a speech act is an utterance which serves as a functional unit in communication.
Utterances have a literal or propositional meaning (e.g., "Where was I when that cell phone rudely interrupted me?"
as uttered by a speaker who was just distracted away from his talk). Utterances also have a functional or illocutionary meaning
(i.e., the effect that the utterance or written text has on the reader or listener, in the cell phone instance serving as a
complaint with the remedy that the participant turn it off so there will not be another similar interruption).
Another useful distinction is that of a speech act set, which has been defined as the set of functional strategies typically
used by native speakers of the target language to perform a given speech act (Olshtain and Cohen, 1983). Any of the speech
act-specific strategies in a speech act set might be recognized as constituting the speech act in question,
depending on the situation and the cultural group involved. So, the indirect request to shut off the phone —
"That cell phone ring just distracted me" — could function on its own as a complaint or coupled with other
strategies within the same speech act set:
- a threat: "If you don't turn that cell phone off, I will ask you to leave the talk,"
- a justification: "It's unfair to the others."
Pragmatic behavior in SLA
Pragmatic behavior, especially as reflected in speech acts, is clearly at the intersection of language and culture since
it is often not enough to string a series of words together grammatically; they must be in a meaningful sociocultural context as well.
For example, learners of Japanese need to know that when the proprietor of a restaurant in Japan thanks them as they are leaving the
restaurant for having eaten there, it is inappropriate to say, do iteshimashita [you're welcome],
which is the way that they would do it in most English-speaking cultures.
"While there has been a presumption (or perhaps wish) among language educators that learners will eventually
acquire on their own a sense of how to perform speech acts just from being exposed to the target language,
experience and research as well has proven this view to be false."
Fortunately, descriptions of speech acts keep improving, as more empirical work continues to appear. Despite some reservations about
the broad terms used to describe speech act distinctions (see Beebe and Waring, 2001), speech acts have been described in terms of their
sociopragmatic and pragmalinguistic nature (Thomas, 1983). In other words, researchers have collected empirical evidence as to the
appropriateness of performing speech acts in certain contexts — the sociopragmatic aspect (e.g., whether/when you could ask a
colleague how much he makes a month), and also just what language forms are appropriate for performing these speech acts —
the pragmalinguistic aspect. The classic case of subtle pragmalinguistic distinctions is that of how to intensify an apology in
American English, where "very" is more an indication of etiquette ("I'm very sorry to interrupt your meeting, but...") while "really"
implies regret ("I'm really sorry I bumped into you.").
While there has been a presumption (or perhaps wish) among language educators that learners will eventually acquire on their own a
sense of how to perform speech acts just from being exposed to the target language, experience and research as well has proven this
view to be false. Even learners in an L2, rather than a foreign language environment, may go many years without mastering speech
acts — because these utterances represent such complex speech behavior (Blum-Kulka and Olshtain, 1986). For this reason,
there is a growing consensus that for best results, we may in fact need to explicitly teach L2 pragmatics (see Rose and Kasper, 2001;
Bardovi-Harlig and Mahan-Taylor, 2003).
A Website on speech acts for teachers
At the Center for Advanced Language Acquisition (CARLA), Noriko Ishihara and I have developed over the last three years this
source of information for language teachers, materials developers, learners, and researchers: http://www.carla.umn.edu/speechacts/teaching.html [inactive link].
This website was created in response to a felt need expressed by a curriculum writer that basic information about speech acts was not
readily available, and that he did not have time to seek it out in research reports.
". . . there is a real problem associated with the explicit teaching of speech acts:
the very act of doing so means that we are extracting them from their rich context,
creating a more artificial situation."
The creation of the website was undoubtedly motivated in part by a desire to encourage the explicit teaching of speech acts. Yet there
is a real problem associated with the explicit teaching of speech acts: the very act of doing so means that we are extracting them
from their rich context, creating a more artificial situation. In addition, courses of instruction have a finite time period,
with much to cover and teachers are usually untrained with regard to speech acts. Thus, teachers would be hard pressed to cover
even the basics of the host of relatively essential speech acts found in a given language. In addition, the target language is often
taught as a foreign language, and the students may not be likely to visit or live in a country where the target language is spoken.
So, challenges abound.
If we explicitly teach speech acts, are we assuming a level of awareness on the part of students which is out of proportion with the
expectations of them in other spheres of language learning? This may well be the case. In addition, we may well ask whether learners
strive to be that native-like in their performance (Ishihara, 2003). What if they do not? And what would be the stakes if they
stick to their native-language means of performing the given speech acts? These are all issues that arise when efforts are made
to have learners be more native-like in their pragmatic behavior.
Self-access materials on speech acts for learners
Again, at the University of Minnesota, several efforts have been made to support intermediate and advanced language learners in the
learning and use of speech acts. For example, a team of authors has generated a students' guide for study abroad with a speech acts
component for self-access learning (Paige, Paige, Kappler, Chi, and Lassegard, 2002). In addition, Ishihara and I have constructed a
website for learning speech acts in Japanese (apologizing, complimenting, thanking, refusals, and requests) and strategies for appropriate
What characterizes this website is that it combines both content about the speech acts, which is largely based on empirical studies
of what native speakers of Japanese actually do, as well as information as to strategies suggested for performing the speech acts effectively:
These strategies are color-coded red, blue, black, and green respectively. Evaluational data we have collected from student users of
the website would suggest that this color-coding is of help to some learners and not to others. In any event, the following are some
examples of strategies for performing speech acts in Japanese:
- Core (i.e., speech act-specific) strategies without which the utterance would not be recognized as the given speech act,
- Support strategies that could enhance or expand upon the use of the speech act,
- Additional strategies that could also help in the performance of that speech act or related speech acts, but which are not essential,
- General strategies that cut across speech acts.
- Speaking hesitantly to appear humble when the speech act calls for it.
- Purposely leaving the utterance incomplete when the speech act calls for it.
- Using repetition of one or more elements in the utterance in order to achieve the appropriate effect in Japanese speech act performance.
- Telling a white lie (in refusals, apologies).1
Numerous unresolved issues
As to be expected with new ventures such as this one many issues are still unresolved. Let us consider three of them.
First, what role does the teacher need to have (if any) if self-access materials are utilized?
In particular, is it imperative that the teachers be well-versed in the pragmatics of their language?
Secondly, does a self-access site ensure that learners will have an adequate sense of both how to perceive
and to produce pragmatic tone (e.g., contemptuous, mocking, false sincerity, playful, supportive; Beebe and Waring, 2002)?
While pre-recorded snippets of language can certainly assist learners in getting a sense of how to use tone,
there is no substitute for live interaction to reinforce perceptions.
Finally, how detailed should such a website be?
In other words, just how valuable are the insights we gain from sub-sub-categorizing, for example, different ways of
showing deference in making requests? Might we not run the risk of discouraging the learners by overloading them
with more detail than they can handle?
Assessing speech acts
Now that we have discussed the teaching of speech acts, let us turn to an even more daunting area, namely,
the assessment of what learners have learned about speech acts and how to perform them. We could start by asking,
"What is our purpose for assessing speech acts?" In general, the focus has tended to be on assessing speech acts for research purposes.
But with the push to teach speech acts in the classroom, what language assessment techniques might we draw on commensurate with this interest?
We will give some examples below.
In addition, how much coverage will speech acts receive in a given course? If the coverage is to be minimal,
then we might expect that their assessment would be integrated into general proficiency. If it is to be extensive,
then presumably assessment would be of achievement with that material.
The following are some of the challenges associated with attempt to assess classroom speech act performance:
- Trying to assess what is highly interactive in a non-interactive way.
- Attempting to create a situational context by means of a single prompt (see Varghese & Bilmyer, 1996).
Concern has, for example, been voiced that lengthy prompts are difficult to construct and difficult for respondents to read through (Roever, 2004).
- Determining what can be scored and how to determine grades.
Some suggested areas for assessment: What to assess
We could start our assessment of speech acts by checking to see the extent to which learners are able to use speech act-specific
core strategies — the set of strategies taught in the material; those that can "easily" be observed and measured (e.g., for
apology — expression of apology, acknowledgment of responsibility, explanation/excuse, offer of repair, promise of non-recurrence).
In addition we could check to see how much control they appear to have over both the sociopragmatic factors
(i.e., whether the speech act can be applied in a given situation) and pragmalinguistic factors (i.e., the language forms that are appropriate in the given situation).
Finally, we could look to see whether the learners are able to deploy appropriate modifications of the speech act, such as:
- showing proper intensity given the seriousness or importance of the situation,
- adjusting the speech act for age,
- making adjustments for gender, and
- taking into account the relative status of the interlocutors.
How to assess speech acts
Much to the benefit of practitioners in the field, researchers at the University of Hawai'i have provided a suggested series of measures of
pragmatic ability (requests, refusals, and apologies) along a continuum from free to cued (Hudson, Detmer, and Brown, 1994). Others have
benefited from these suggestions by constructing assessment batteries making use of them. For example, Yamashita (1996) developed a battery
for assessing speech act ability in Japanese as a foreign language and Yoshitake (1997) did the same for EFL learners. The battery included
an oral discourse completion task (DCT), a written DCT, a video-taped role play (3 speech acts), self-assessment (speculation; rating own
performance on a video), and a multiple-choice task. For greater validity, it is probably best to use multiple measures in order to approximate
the respondents' genuine abilities, as reflected by the Yamashita and Yoshitake batteries.
As with any set of measures, there are trade-offs associated with using more open as opposed to more closed types of assessment.
Open role play, for example, allows for the full operation of turn-taking, sequencing of moves, and negotiation of meaning (Kasper and Dahl,
1991: 228-9). Written response, on the other hand, may foster more thoughtful responses, possibly more indicative of a speaker's competence.
Let us take a look at some actual item types, first looking at the perception of speech acts, and then at the production of them —
first by means of the classical DCT and then by means of a multiple-rejoinder DCT:
Perception of a Speech Act
Rate the following responses according to whether they are "acceptable,"
"more or less acceptable," or "unacceptable" in an American English situation:
(1) A student forgets to return a book to a professor. Student:
a. ____ Oh, damn! I forgot it.
b. ____ Sorry. I forgot.
c. ____ Oh, I'm really sorry. I completely forgot.
d. ____ Oh, well, I've had a lot on my mind lately.
(2) A young woman bumps into your shopping cart at the supermarket and
some of your groceries spill onto the floor.
Aside from helping you pick them up, she says:
a. ____ Sorry.
b. ____ Please forgive me.
c. ____ I'm very sorry.
d. ____ I'm really sorry.
Production of a Speech Act: Classical DCT
(1) You promised you'd buy your neighbor medicine for her sick child
while in town, but you forgot.
Your neighbor: "Did you get the medicine?"
(2) You don't stop in time at a red light and bump into the car
in front of you. The other driver and you get out and see that there is
damage to the other car. The other driver is very upset.
Production of a Speech Act: Multiple-Rejoinder DCT
(1) You find a bargain air ticket to a city where you have great friends.
In order to take advantage of this deal, you need to ask your instructor
for an extension on a paper that you were supposed to hand in after the weekend.
Professor: Well, you know, you had plenty of time to work on this paper
already. There was no need to wait until the last minute to prepare it.
Professor: I'm sorry, but I can't really agree to give you an extension
on this paper. I don't think that going to visit some friends during the
semester is a good enough reason for an extension.
Professor: Well, I'm not so thrilled about doing it. It's not my policy.
Professor: Ok, well, just this time.
Taking a closer look at multiple-rejoinder discourse completion
The multiple-rejoinder approach is meant to get more data than the one-shot approach (see Cohen & Shively, 2002/2003).
What are the reactive effects of the measure? To find out, we could have raters assess how well each rejoinder fits into the discourse
(say, on a scale of 1-5). We could have the raters look at the following factors:
- Level of formality (given the age, status, and familiarity between the interlocutors).
- Degree of politeness/deference (given the importance or severity of the event).
- Degree of directness (given the level of familiarity between the interlocutors).
- Pragmalinguistic control (i.e., appropriateness of language structures used).
- Sociopragmatic control (i.e., is the appropriate speech act performed and at the right time?)
- Overall success of the speech act performance.
So how do you actually rate oral and written speech act production?
The construction of a rating scale and even the calibration of the ratings themselves does not ensure that certain issues of reliability
and validity have been adequately handled. The following are some of the issues that can come up. One is whether we use predetermined
criteria for each speech act — that is to say, those criteria which reflect what was taught in a particular speech act unit. If the
assessment is meant to be criterion-referenced, then this might be a good idea. If it is intended to be more a measure of general proficiency,
then perhaps it would not be so desirable to predetermine appropriate response.
Following up on this issue of predetermined scoring, do the teacher or raters determine their own intuitive coding holistically and then
give their rationale for it? In such a case, for instance, raters would be asked to take notes on what is inappropriate and incorrect for
each rejoinder within a given vignette. Furthermore, would the responses be anchored to baseline data from native speakers for each criterion?
And what if we want to assess gain? For a research study, we could scramble the pre-and posttest data for all learners on each speech act,
so that raters would not know which was which. But in so doing, we would lose the ability to do a fine-tuned analysis of pre- and
post-performance. Might it be better for raters to be comparing the pre- and posttest data learner-by-learner? And are the raters told
in advance which were the responses from pretesting and which were from posttesting? And finally, is each rejoinder scored separately or
are all rejoinders scored together as if they were one single turn? In our work at the University of Minnesota, we have found it preferable
to score all the rejoinders together as one response.
While researchers will continue to fine-tune our descriptions of speech acts, there appears to be sufficient information for teachers and
language assessment specialists to develop measures of speech act ability. As this JALT Pan-SIG plenary indicated, there exist numerous
formats to choose from, for assessing both the perception and the production of speech acts.
". . . the best approach is to collect data from more than one measure and all the same, to view these measures as
an approximation of the respondent's speech act ability."
Since pragmatic behavior by its very nature varies, the best approach is to collect data from more than one measure and all the same,
to view these measures as an approximation of the respondent's speech act ability. In fact, might it not be a good idea to use a portfolio
approach? While pre-existing measures of speech act performance may be appropriate in a given L2 classroom, designing speech act tasks
that really "capture" the characteristics and interests of the given group of L2 students may be preferable. It may be said that language
assessment suffers a bit too much from the "one-size-fits-all" mentality. While it may be expedient to use pre-existing measures of
pragmatic ability, it may be more informative and also more acceptable to the learners themselves if the measures genuinely reflect
the material that they were exposed to and is sensitive to the issues that were covered in their instructional materials.
Bardovi-Harlig, K. & Mahan-Taylor, R. (Eds.). (2003). Teaching pragmatics. Washington, DC: US State Department.
Retrieved on Feb 12, 2004 from
Beebe, L. M. & Waring, H. Z. (2001, February 3). Sociopragmatic vs. pragmalinguistic failure: How useful is the distinction?
Paper presented at the NYSTESOL Applied Linguistics Winter Conference, CUNY Graduate Center.
Beebe, L. M & Waring, H. Z. (2002, April 7). The pragmatics in the interlanguage pragmatics research agenda:
The case of tone. Paper presented at the AAAL Colloquium entitled, "Revisioning Interlanguage Pragmatics Research," Salt Lake City.
Blum-Kulka, S., & Olshtain, E. (1986). Too many words: Length of utterance and pragmatic failure.
Studies in Second Language Acquisition, 8 (1), 47-61.
Cohen, A. D. & Ishihara, N. (2004). A Web-Based Approach to Strategic Learning of Speech Acts.
Report to the Center for Advanced Research on Language Acquisition (CARLA).
Retrieved on Feb 12, 2004 from http://www.carla.umn.edu/speechacts/Speech_Act_Project_Rept.pdf.
Cohen, A. D. & Shively, R. L. (2002/2003). Measuring speech acts with multiple rejoinder DCT's.
Language Testing Update, 32, 39-42.
Paige, R. M., Cohen, A. D., Kappler, B., Chi, J. C., & Lassegard, J. P. (2002).
Maximizing study abroad: A students' guide to strategies for language and culture learning and use.
Minneapolis, MN: Center for Advanced Research on Language Acquisition.
Roever, C. (2004). Difficulty and practicality in tests of interlanguage pragmatics.
In B. Boxer & A. D. Cohen (Eds.), Study speaking to inform second language learning (pp. 283-301). Clevedon, England: Multilingual Matters.
Rose, K. R. & Kasper, G. (Eds.) (2001). Pragmatics in language teaching. Cambridge, UK: Cambridge University Press.
Thomas, J. (1983). Cross-cultural pragmatic failure. Applied Linguistics, 4, 91-112.
Varghese, M. & Billmyer, K. (1996) Investigating the structure of discourse completion tests.
Working Papers in Educational Linguistics 12 (1), 39-58.
Yamashita, S. O. (1996). Six measures of JSL pragmatics. (Technical Report #14).
Honolulu, HI: Second Language Teaching & Curriculum Center, University of Hawaii at Manoa.
Yoshitake, S. (1997). Interlanguage competence of Japanese students of English: A multi-test framework evaluation.
Unpublished doctoral dissertation, Columbia Pacific University, San Rafael, CA.
Yule, G. 1996. Pragmatics. Oxford, UK: Oxford University Press.
See Cohen and Ishihara (2004) for more on the strategies component of the website.