Assessing derivative skills: EFL learners and English word-formation Katja Mäntylä and Ari Huhta
katja.mantyla@jyu.fi, ari.huhta@jyu.fi
Assessing derivative skills: EFL learners and English word-formation Katja Mäntylä and Ari Huhta
katja.mantyla@jyu.fi, ari.huhta@jyu.fi
CEFLING Linguistic Basis of the Common European Framework for L2 English and L2 Finnish
Project funded by the Academy of Finland 2007-2009
Based at the University of Jyväskylä; part of the European SLATE network (Second Language Acquisition and Testing in Europe)
Homepage:http://www.jyu.fi/cefling
Collaboration between SLA and Language Testing researchers in Lancaster and Jyväskylä. We are part of the SLATE network (Belgium, Finland, France, Germany, Italy, the Netherlands, Sweden and the UK)
Questions How to test word formation skills? How do the three methods used in the study function – what are their pros and cons?
What is the relationship between the word-formation skills and overall written proficiency of Finnish school pupils ?
(What kind of knowledge do they have on English word-formation?)
(How do their word-formation skills develop?)
(Is there any difference between Finnish and Swedish speaking participants?)
Word-formation and SLA Word-formation and SLA in general
Mochizuki & Aizawa (2000)
Nyyssönen (2008)
Schmitt & Meara (1997)
Schmitt & Zimmermann (2002)
Role in teaching English in a Finnish school?
Derivation chosen because
Productivity of the method
The participants familiar with it at least implicitly
(though textbook analysis shows that explicit teaching non-existent)
(alongside compounding, the most productive) vain mainittu –LY kieliopin kohdalla
(alongside compounding, the most productive)
Participants 7th - 9th graders
13-16 –year-olds, have studied English as a FL for at least 4-6 years
were administered three short word-formation tests (for practical reasons)
over 300 completed the word formation tests, about 150 of whom also completed four writing tasks as part of the main CEFLING project
Different parts of Finland
Word-formation test 1 (Productive gap-fill test) Three written word-formation tests (revised after piloting)
1. Sentences / sentence pairs in English with a Finnish translation of the target word (from Waystage):
I am ________ (varma) that he will get the job in London.
He will _________ (varmasti) get the job in London.
sure - surely
We also have data from Swedish speakers but haven’t yet looked at that yet
Words from waystage????
Word-formation test 2 (final version) (Non-words based test) 2. Sentences with non-words with explanations in Finnish (gap-filling):
Some of the non-words taken from the DIALANG placement test (English) designed by Paul Meara
Example in Finnish (with Finnish real words)
She could bourble animals very well because she was a good ____ bourble____. (henkilö, joka tekee lihavoidun sanan kuvaamaa toimintaa/työtä)
(a person who does the action described by the bolded word)
We also have data from Swedish speakers but haven’t yet looked at that
Non-words from Dialang???
Prefixes: the numbers didn’t tally actually had to know the answers
Word-formation test 3 (List-based test)
3. A list of prefixes from which the participants were to choose suitable ones to fill in the gaps in sentences He did not follow the instructions. He had
___ understood them. anti-
de-
dis-
in-
im- il-
ir-
inter-
intra-
mega- mini-
mis-
mono-
neo-
non-
poly-
post-
pre-
pro-
re-
trans-
un-
We also have data from Swedish speakers but haven’t yet looked at that
Non-words from Dialang???
Prefixes: the numbers didn’t tally actually had to know the answers
Writing tasks Email to a friend
Email to one’s teacher
Email to a store
Opinion piece
Narrative piece
Each student wrote 4 texts
Each text was assessed by 4 raters
the rating scale was a combination of several writing scales from the CEFR that best suited the writing tasks
Marking word-formation tests Double marking
Scoring:
Productive gap-fill test : 0-1-2-3-4
Non-words based test and List-choice based test: 0-1-2
The respondents were very creative: minigabl
Spelling errors more or less ignored in scoring: unbelievubl,unbelievevabl, unbelievobl; understant, anderstand
(cf. shore sure, deffreno different)
Results of item analyses / characteristics of the tests
One item turned out to be poor (despite piloting):
16. The dog started to ___________ (seurata) the trail of a fox.
17. The hunter caught the fox the ________________ (seuraava) day.
seurata = to follow / seuraava = following
next
Characteristics of the 3 tests Items Mean score (percent) Standard deviat-ion Median Cron-bach’s
Alpha Alpha for 40-item test Average item / total correlation Productive gap-fill test
(n=326) 18 75.1 19.1 77.6 .86 .93 .56 Non-words based test
(n=299) 8 34.3 26.4 25.0 .76 .94 .62 List-choice based test
(n=327) 12 39.3 22.3 37.5 .78 .92 .54 All 3 tests together
(n=327) 38 55.7 21.3 49.6 .90 .91 .48
Characteristics of the 3 tests Items Mean score (percent) Standard Deviat-ion Median Cron-bach’s
Alpha Alpha for 40-item test Average item / total correlation Productive gap-fill test
(n=326) 18 75.1% 19.1 77.6 .86 .93 .56 Non-words based test
(n=299) 8 34.3% 26.4 25.0 .76 .94 .62 List-choice based test
(n=327) 12 39.3% 22.3 37.5 .78 .92 .54 All 3 tests together
(n=327) 38 55.7% 21.3 49.6 .90 .91 .48
Characteristics of the 3 tests (analyses with the TiaPlus programme) Items Mean score (percent) Standard Deviat-ion Median Cron-bach’s
Alpha Alpha for 40-item test Average item / total correlation Productive gap-fill test
(n=326) 18 75.1% 19.1 77.6 .86 .93 .56 Non-words based test
(n=299) 8 34.3% 26.4 25.0 .76 .94 .62 List-choice based test
(n=327) 12 39.3% 22.3 37.5 .78 .92 .54 All 3 tests together
(n=327) 38 55.7% 21.3 49.6 .90 .91 .48
Word-formation test 2 (Non-words based test) Sentences with non-words with explanations in Finnish:
ITEM 1:
She could bourble animals very well because she was a good ____ bourble____. (henkilö, joka tekee lihavoidun sanan kuvaamaa toimintaa/työtä)
(translation of the Finnish text: ”a person who does the action / work described by the bolded word”)
We also have data from Swedish speakers but haven’t yet looked at that
Non-words from Dialang???
Prefixes: the numbers didn’t tally actually had to know the answers
Correlations between word-formation tests N = 281-310 A (total) Productive gap-fill test A1. items tapping the base form A2. items tapping the inflected form B. Non-words based test C. List-based test A. Productive gap-fill test 1.00 (.905) (.971) .567 .618 A1. Items tapping the base form 1.00 .778 .511 .566 A2. Items tapping the inflected form 1.00 .544 .596 B. Non-words based test 1.00 .601 C. List-based test 1.00
Conclusions about test characteristics
Productive gap-fill test was rather easy for these test takers because the words were based on Waystage (A2) for English
Non-words based and list choice based tests were quite difficult (but for different reasons)
Reliable (relative to their length)
Fairly high correlations between the different word formation tests but far from perfect not equivalent
The non-words based test appeared to be at least as good as the other, more traditional word-formation tests
Relationship between word-formation skill and more general language profiency
n = 141-160 WRITING SKILL (on CEFR scale)
Mean rating across 4 raters and 4 tasks Productive gap-fill test .696 Non-words based test .652 List-based test .742 All 3 tests together (raw score) .798 All 3 tests (only derivated forms; IRT theta value) .789 P = .000
Comments