CHAPTER 8: PROGRAMME EVALUATION (Continued)
Previous pages: 1

8.3 Research instrument 1: Self-assessment.
8.3.1 Description
This instrument (tables A-53/4, appendix C-34), was designed to provide data on changing student beliefs and abilities, and was the central self-assessment item in the first two books, providing a benchmark against which students could assess their progress in the first two years of the programme. When this instrument first appeared, in the 1998 version of the books, it was left to teachers and students to decide whether to perform the assessment as part of their normal class-work, since the idea of self-assessment was very new to most teachers, and a notable number chose not to use it. For the purposes of this study, however, it was administered twice (week 1 & week 30 of the academic year) in 1998 by the author, to a randomly selected pilot group of 5 classes (106 Freshmen students). In 1999, use of this instrument was still optional, though (following teacher-training throughout 1998) it received more emphasis as an important part of the syllabus, and all students were encouraged to fill in the assessments as part of their normal studies, giving the final results to their teachers at the end of the second semester. The instrument did not appear in the Junior textbook, and was administered separately to Junior students in 1999. Results (section 8.3.2, below) therefore reflect the fact that the support and approval of the participants was an important factor, and that the decision to investigate and use self-assessment was part of the attitude-change process for a number of teachers.

This deficiency analysis, based on Oscarson's (1980) self-assessment instrument (cf. TWA pages 106/107, appendix C-58), appeared twice in each book, in identical form (TMM[6] pages 22/23, 189-190; NYT pages 20-21, 157-158), each question having five possible responses along a Likert Scale, ranging from "즉시 질문에 대답할 수 있다" ("I can do this absolutely" - 5 points) to "전혀 할 수 없다" ("I can't do this at all" - 0 points). The first 9 questions in research instrument 1 cover topics from the first half (Chapters 1 – 6) of TMM (except for Q 9), some of which are reviewed in NYT (table A-53). Questions 10 to 20 then cover topics that can be found in the remaining sections of the first two books (table A-54). Having filled in "suitable" responses, students calculated a score, which was used for comparison purposes when they did the activity again. Any "familiarity" factor (maturation – cf. section 1.3.2) was largely diffused by the fact that there was a minimum of three months (maximum 9 months) between each use of this instrument. Comprehension was facilitated by instructions and questions being in English and in Korean (cf. appendix C-34).

The function of this instrument, in addition to addressing all four research questions (section 1.2), was to provide students with feedback on their perceived progress, and to show that improvement in target oral abilities was in fact taking place. In this way it could be used as a measure of student confidence as well as of student performance on syllabus goals. Since it dealt with perceived abilities ("Yes, I can do this"), this self-assessment instrument (and the score) was not dependent on variations and learning plateaux indicative of developing interlanguage (cf. sections 6.2.4.1, 6.4.2.1) but described how the student saw his/her progress, irrespective of starting level. Results for this instrument are presented in the following section.

8.3.2  Research instrument 1: Results
Table A-63 (below) shows results as submitted in 1998 and 1999. Upper numbers in columns 3 to 8 refer to average student scores (minimum 0, maximum 100[7]), and numbers in parentheses in columns 3 to 8 refer to the number of students completing the instrument. Two assessments were carried out each year (since "My English Ability" appeared at the beginning and end of TMM and NYT), but it was also suggested by teachers (in teacher meetings in 1999) that more detailed learning profiles could be obtained by performing the instrument at the end of the first semester (week 16 - 1/99/2) and the beginning of the second semester (week 17 - 2/99/1). This was then made an option for teachers to suggest to students.

TABLE A-63: RESEARCH INSTRUMENT 1, RESULTS, 1998, 1999. 

Row 1 (table A-63, above) shows the total number of students who completed this self assessment as Freshmen in 1998 and as Sophomores in 1999, and their average self-assessment scores (39.8, 53.8, 48.3, 53.4, 43.7, 50.5; appendix D-1). Row 2 describes those (n = 44) who performed four self-assessments during that time (appendix D-2). Rows 3 and 4 show similar data for Freshmen in 1999 (n=839), and for those who completed all self-assessments in 1999 (i.e. the two assessments in the book, and two extra assessments, n = 248; appendices D-3/4). Rows 5 & 6 present the same data for students who were Sophomores in 1998 and Juniors in 1999 (appendices D-5/6), and row 7 shows Sophomore students who did every self-assessment in 1999 (including the two extra ones; n = 95; appendix D-7). Row 8 details results for 27 Junior students in 1998 (appendix D-8). Since English was for them a selective course, these students were a pilot group, as were the 106 Freshmen (row 1).

Table A-63 and appendices D-1-8 show a general "stepwise" trend in terms of self-assessment results, increasing during semesters, and dropping during vacations. The maximum possible score on this instrument was "100" (a response in the "A" column = 5 points; "B" = 3 points; "C" = 2; "D" = 1; "E" = 0, for each of 20 questions [cf. Appendix C-34, pages 504/5]), and the mean-point was "44" (due to the absence of a "4" mark). Thus the 106 "pilot" students who were Freshmen in 1998 (appendix D-1) showed an average increase in scores from 39.8 to 53.8 over the year. This marker of perceived proficiency fell to 48.3 when they returned as Sophomores in 1999, rising to 50.5 by the end of the year. This levelling out of scores in 1999 could be perhaps attributed to the extra number of students at that time (table A-63, row 1), most of whom were new to it. However, original pilot students who performed all four SA[8]s in 1998/9 (table A-63, row 2; appendix D-2) showed the same trend in their Sophomore year, the initial leap from 41.2 to 57.2 (1998) being balanced by 51.1 and 51.2 in 1999. Row 3 of table 63 shows Freshmen in 1999 (the first full year of the programme) climbing step-wise from 35.8 to 49.3 by the end of the year (appendix D-3) (35.4 to 51 for students who completed all four SAs in 1999 – appendix D-4). Students who were Sophomores in 1998 (these students were took the Freshman course, since this was their first year on the programme) showed stepwise improvement over two years (44.2, 46.7, 42.7, 52.4 – appendix D-5), and those who performed all 4 SAs in 1998/9 showed continual perceived improvement, irrespective of vacations (tables 63, row 6; appendix D-6). The downturn (e.g. from1/99/2[9] to 2/99/1[10]) in appendices D-1-5, 7/8 is notable, indicating a drop in perceived ability during university vacations. Row 7 of table A-63 shows that Sophomores in 1999 (appendix D-7) who performed all four SAs that year mirrored the overall two-year scores (appendix D-1), with 1/99/2 (54.1) being higher than 2/99/2 (51.6), and 2/99/1 (after the summer vacation) being the lowest of all (44.6). Finally, one pilot group of Juniors in 1998 progressed from 37.8 to 45 in the year (table A-63, row 8; appendix D-8).

8.3.3 Instrument 1: discussion
These figures (above) indicate a general enthusiasm for the programme in the first year (cf. teachers' comments – tables A-73, A-82), with general perceptions of stepwise improvements in oral skills, and a "cooling off" in the Sophomore year. This could be attributed in some part to the fact that Korean university students typically become disillusioned with the university system in their second year, and explore their new social situation, prior to renewing academic efforts in the Junior year. However, there was also some feeling among teachers that NYT was very similar to TMM, and that students who had successfully taken on the innovations in learning in TMM were ready for the project-based approach of TWA[11].

The general can hide the specific, and closer examination of the data for research instrument 1 showed some interesting examples of students "taking off", with individual initial and final scores in 1999 (1/99/1 to 2/99/2) showing marked increases: e.g. 14 to 20, 24 to 34, 30 to 52, 45 to 62, 65 to 92. The figures in table A-63 therefore represent a "global" view of multiple unpredictable "local" events, each of which is notable in its own right - from the totally unconfident student (scoring 14 to 20 over the year) who is gradually finding out that he/she can improve, to the 20-22 student who sees him/herself on a learning plateaux, and to the 99-96 student whose (perhaps unrealistic) perceptions are gradually being revised. All of these scores provide useful information for the teacher, whose role is to talk to the student about them, discuss learning plans, and promote attitude modification.

Continue reading Chapter 8: "Research Instrument 2"