CHAPTER 8: PROGRAMME EVALUATION (Continued)
Previous pages: 1, 2, 3

8.5 Research instrument 3: Teacher questionnaire (internal).
8.5.1 Description
In similar manner to Horwitz's (1988) BALLI, (cf. Peacock 1998) and Hills' (1976) investigation of student/teacher needs, research instrument 2 (section 8.4) was also administered to teachers (at the end of semesters 2/1998, 1/1999 and 2/1999) as research instrument 3. Items were identical (cf. tables A-55-59), except that "I" and "me" (sections A, D, E) were replaced by "My students". "The teacher" (section C) was replaced by "I", and items 56 & 57 were combined into item 56:  "My students need more English classes". It was therefore possible to examine and compare student/teacher perceptions (section 8.5.3), including how they perceived each others' perceptions (Hills 1976).

8.5.2 Instrument 3: results
Teachers were given their version of the student questionnaire in December 1998, June 1999 and December 1999. Of the 14 teachers working at the Language Centre in December 1998, 7 (50%) responded, perhaps indicating a preference not to comment or a lack of interest on the part of some teachers. Results for these 7 teachers are presented in appendices D-34-38 as totals (without differentiation for separate student-years), and as percentages in table A-66. In June 1999 there were 20 teachers working full time at the Language Centre, of whom <[2] 19 reported opinions on Freshmen students, <13 on Sophomores, and 3 on Juniors. Of the 20 teachers working on the programme in December 1999,  < 11 reported opinions on Freshmen students, <9 on Sophomores, and 4 on Juniors. These figures reflect the fact that a number of teachers specialised in teaching one or two student-years (e.g. only Freshmen classes, or only Sophomore and Junior classes), and a relatively small number specialised in Junior classes. 9 new teaching staff were appointed to the Language Centre in March 1999 (due to expansion of the programme to include the Junior year) and 5 new staff in September 1999 (replacing 5 who had moved on to other positions), so the responses for this instrument are not always from the same teachers. Results for June 1999 and December 1999 are presented in appendices D-39-78 for each "student-year" in addition to totals for all three years, and in tables A67/8 as percentages.

Numbers in the shaded rows of tables A-66-8 refer to question numbers (e.g. "5" = Question 5: "My students ask classmates for help when they need it"). Numbers under these shaded rows refer to the percentage of the total number of responses (e.g. "Always: 42.9%, S'times: 57.1%, Never: 0%). Discussion of results for research instrument 3 can be found in section 8.5.3.

8.5.3 Research instruments 2 and 3: discussion of results
Tables A-89-93 and appendices D-122-131 present comparative results of the internal questionnaire for students and teachers. The groups of three columns of figures in tables 89-93 refer to the three possible answers for each item (cf. tables 64/5), in the order that they appear on the original instrument. Thus the figures 26, 64, 10, in the "Students Dec 98" column of table A-89, item 1, indicate that in answer to the statement "I speak English in class", 26% of students replied "always", 64% replied "sometimes", and 10% replied "never".

8.5.3.1 Research instruments 2 & 3, section A.
Responses to items in Section A (table A-89) (cf. appendices D-9, D-14-17) show a general positive movement in students' perceptions of their participation in class, though the "always" scores sometimes decrease. 10% of students state in December 1998 that they "never" use English in class (item 1), and this number reduces to 1% in June 1999, though the shift is to "sometimes" (rather than "always"), which rises from 64% to 74%. There is greater perceived sharing of ideas (item 2), participation in activities (item 3), punctuality (item 7), and homework completion (item 8), while scores for asking/giving help (items 4-6), visiting the teacher (item 9), and sending him/her emails (item 10), show little change. Students in different years (appendices D14 – 16) show little variation of opinion, except for increased focus on "help" (items 4 & 6) and on communicating with the teacher (items 9 & 10) in the Sophomore and Junior years. This slight positive trend is mirrored in the teachers' results (table 89) though there is noticeable difference in perceptions. Teachers see their students as using more English[3] over the 12 months from December 1998 to December 1999 (item 1), sharing more ideas (item 2), participating more (item 3), asking for help more (items 4, 5, & 6), communicating with the teacher more (items 9 & 10), being less punctual (item 7) and doing less homework (item 8). 

8.5.3.2 Research instruments 2 & 3, section B.
Section B (cf. table A-90, appendices D-10, D18-21) shows students asking for "more" of every item, though in the case of "homework" a greater number ask for "less". The trend over the six months is for communication-based activities (items 11, 12, 13 & 20) to become more important, though grammar (item 17) also receives more attention (student responses asking for more grammar increase from 44% to 55%), in striking contrast to the teachers (table A-90, appendices D-35, 43-46, 63-66), of whom the percentage asking for more grammar decreases from 43% to 29% over the period of study (though see teacher comments re grammar in tables A-72/3, 78, 80/1, 86, 88). Teachers initially agreed with the students on the need for more discussion (item 18), communication activities (item 12), and listening (item 13), though it is noticeable that these become less urgent over the 12 months. Teachers also disagreed with the students in terms of other "needs": 

1.      Item 14, language games: 44% (December 1998) and 46% (December 1999) of students wanted more of these, as against 27% (December 1998), 82% (June 1999) and 83% of teachers, who thought that there were enough language games in the syllabus.

2.      Item 16, correction: 39% and 53% of students asked for more correction, though 100%, 80% and 79% of teachers thought the amount of correction was sufficient

3.      Item 17, grammar: see section 8.5.3.2, above.

4.      Item 19, homework: 66% and 68% of students thought there was enough homework, where 71%, 32% and 8% of teachers saw the need for more. Teachers' opinions show a remarkable progression in these figures, in that by December 1999, 83% of teachers felt that there was enough homework. This could have reflected changing attitudes about the role and form of out-of-class assignments. Teachers typically perceived a lack of "homework" in the textbooks, this opinion changing as they became familiar with the way in which assignments tended to grow out of the classroom activities (especially in the Junior year).

5.      Item 20, advice: 35% and 54% of students see a need for more advice in the classroom, whereas teachers increasingly (43%, 90%, 97%) see sufficient advice as being available.

Results for the different years (appendices D-18-20, 43-45, 63–65) show little variation, except that Sophomores see a need for more language games, teacher-help, and correction (Items 14-16), and teachers identify discussion (Item. 11) and listening (Item. 13) as particularly important in the Junior year.

Student references to a need for more grammar were understandable, given their previous learning history (section 2.3.3.1), but teacher comments asking for more grammar represented a different problem, since various information tools (e.g. mission statement, teachers' resource books) made it clear that the textbooks were not intended to focus on specific grammatical objectives (though TMM does this implicitly) and that teachers were expected to supplement the books with their own learning content as appropriate. The response to this feedback was therefore to make programme information more accessible to teachers (e.g. by putting it on the homepage - http://lc.andong.ac.kr/eng "Teacher resources"), and to concentrate more on teacher-training, encouraging a greater focus on (and trust in) CMI ("Are my students becoming more confident, motivated and independent?"), and on clarification of specific items (grammar, pronunciation, stress, intonation, etc.) when requested by individual students ("the right thing for the right students at the right time").

The shared focus on the need for more "listening" (item 13, table A-90) led to the design and implementation of a "listening programme" by one of the teachers. This programme (begun in the first semester of 1999) consisted of a selection of listening activities being made available to students in the audio-visual (A/V) library, along with question sheets, textbooks and scripts. Freshman students were each given a "Listening Card" which they were asked to stamp during the semester, indicating that they had been to the audio-visual library, and had performed a minimum of three (self-selected) listening assignments. In the spirit of self-assessment and independence, students were asked to decide for themselves whether to complete the activities, to do extra work in the A/V library, or to pass over these assignments. If they chose the latter course, this provided teachers with an opportunity to counsel them about attitudes and learning goals, though a number found difficulty with this approach at first, and were reluctant to "trust" the students, often administering "tests" on what had been "learned". The intention of the listening programme was, however, to "take the horse to water" by inviting students to access the A/V facilities, after which a number of teachers noticed increased listening skills (cf. teacher-comments in table A-76) and found students ready to take on extra listening assignments.

An independent evaluation of this listening programme (cf. MacLaughlin 2000; appendix C-80) assessed the extent to which it encouraged independent learning behaviour, and reported "quite positive" results, with over eighty percent of students responding positively to four of the five questions. MacLaughlin (a teacher at the Language Centre) concluded that it was "unlikely that there had already been a significant effect on learning habits as a result of the project", but saw possible influential effects on learning habits and motivation as a result of the combination of the general language-learning environment in the Language Centre, the conversation classes and the listening programme:

... pulling the students towards serious consideration of independent learning activities. The awareness that there are things they can do on their own to improve their English is forming. (MacLaughlin 2000)      

At the end of the 1^st semester 1999 (subsequent to this independent assessment), teachers saw a decreased need for more listening (86-60%; table A-90), but students perceived an even greater need (60-70%), indicative either of greater awareness of the importance of listening, or of the need for the listening programme to be revised.

8.5.3.3 Research instruments 2 & 3, section C.
Section C results (table A-91, appendices D-11, 22-25, 36, 47-50, 67-70) are more consistent for both parties, but show interesting divergences of opinion. Students see their teachers as decreasingly (62-50%) "working hard" to help them, whereas the teachers mark this (item 21, table A-90) very highly (100-83%). Error correction is seen as taking place (71-65%) by the students (item 27), but not by the teachers (14-38%), and teacher/student interaction outside of class (item. 29) is more apparent to teachers, who also feel that they are in general showing the students how to be good learners (item. 30).

There is difference in opinion concerning the ability of students to comprehend the teacher (Ss 13-9%; Ts 43-57-32%). 23-25% of students chose "No", in contrast to the 0% marked by teachers, and neither group seemed to think that the teachers "know" the learning needs of the students sufficiently (item. 22). As with Sections A & B, students showed increasing awareness of their needs, and teachers placed less stress on managing the classroom (cf. decreased scores for working hard, promoting study, emphasising punctuality, rewarding [item. 28], comprehension, and making English "easy" [item 32]). Mutual respect is seen as highly important, but does not necessarily result in satisfaction (item 33), since the change in classroom roles can be unsettling, students having to take on responsibility for learning, and teachers coming to terms with their more intensive and personal role (cf. "Question 5", table A-77).

8.5.3.4 Research instruments 2 & 3, section D.  
In terms of perceptions of the programme (table 92; appendices D-12, 26-29, 37, 51-54), students are again fairly consistent, and teachers again changeable. 26% and 24% of students feel that the English course is helping them in the long-term (item 35), and 19% and 20% feel that they are becoming good learners (item 41 - 66% and 60% of students replied "maybe"). Teachers are more convinced of the value of the programme, particularly in response to item 34 ("The English lesson in the LC help students speak English; Ss 52-52%; Ts 86-100%). By June 1999 all teachers agree that the English course is not helping students in their other studies (item 43 – though this was not the original intention of the programme), a consideration that was noted by the programme designer, for future reference. Freshmen and Sophomores (appendices D-26/7) show lack of awareness of learning strategies, despite the optimistic scores of the teachers, who can see the seeds of independence being planted (cf. teachers' comments in table A-73). At least one teacher drew attention to the need for a greater emphasis on independence at this time (cf. table A-73), and this emphasis was incorporated into teacher-training, with observations and discussions becoming more frequent, stressing the need to "empower" students (cf. table A-75). 

8.5.3.5 Research instruments 2 & 3, section E.
Section E (table A-93; appendices D-13, 30-33, 38, 55-58, 75-78) shows a similar lack of dramatic change on the part of the students, in contrast to more marked variance by the teachers. All agree that the textbook is appropriate (increasingly so for the teachers - items 44, 45, 46), but students in general continue to be unconvinced about their progress (item 47). Self-perceived confidence (item 48) improves by 35% in Dec 1998 and 31% in June 1999, though teachers report greater increases (86-73%). "Enjoyment" (item 49) receives similar scores, with a notable upturn in teachers' marks (43-73%). Items 50 - 57 were designed to discover student/teacher opinions on the class-allocation system and on the frequency of lessons, but returned inconclusive results. Students were split over the need for a "High-level" course, but agreed on the need for a special "basic level" course (item 51), though this was not matched by the teachers. Streaming (item 52) was also popular with students, but not so with teachers, though the idea of multi-level classes (the existing situation - item 53) was uniformly disliked. This was important information for the programme designer, but could not be acted on at once, since Language Centre schedules were dependent upon Departmental schedules. The question of selectivity (items 54-57) shows similar numbers for and against, but there is general agreement that there should not be fewer English classes.

8.5.3.6 Research instruments 2 & 3, conclusion.
Apart from the general consensus that the programme was helping students to speak in English (Item 34, section D, table A-92), major differences in opinion are apparent in various items from these two research instruments, confirming Hills' conclusion (1976:28) that "students and teachers not only see their own needs in ways which differ from each other, but they also see each other's needs in a somewhat different light." Thus teachers in general see more evidence of communication and participation (section A), greater development of learning strategies (section C), more confidence and enjoyment (section D), and less need for direction (section B). Students see themselves as more punctual (section A), needing more communication activities, games, correction and grammar (section B), showing poor comprehension of the teacher (section C), lacking learning strategies (section D) and preferring streaming (section E). Given the focus on positive attitude change in the programme, these results can seem disappointing, since it is evident that students in general have not perceived notable progress. However, the change in teacher attitude is marked (cf. table A-775), and it could be said that positive attitude change has begun with them. Development of CMI is apparently noticeable in the students (cf. section 8.5.3.1; teacher comments in table A-76), but the implicit nature of the learner training has not drawn students' attention to this change. Teacher-training subsequent to the June 1999 results therefore concentrated on encouraging teachers to use the interactive classroom environment to relate to students individually, and talk to them about their learning, while still being available as a language resource for the whole class.

Continue reading this Chapter: "Research Instrument 4"

Continue ch 8