CHAPTER 8: PROGRAMME EVALUATION (Continued)
Previous pages: 1, 2, 3, 4

8.6 Research instrument 4: student questionnaire (external)
8.6.1 Description
The university (ANU) administered a course-assessment instrument to all students in all subjects in semesters 1/1998 and 2/1998, so that comparison with other programmes in the university became a possibility for the present study, though it had not been included in the author's original plans[1]. The author was not involved in designing this instrument, but its focus on student perceptions of the syllabus, the teacher and the course, was relevant and useful in providing another (externally administered) perspective.

This 15-point instrument was in Korean, a translation of which can be seen in table A-60, and students responded on a Likert scale from 0 ("I disagree completely with this statement") to 5 ("I agree completely with this statement"). The reference to the syllabus in item 1 (table A-60) was potentially informative, being absent from the "internal" instruments 2 and 3, but was qualified somewhat ambiguously in terms of how the syllabus helped the students to "understand the whole course", and one wonders what definition of "syllabus" was being used here, and how it was interpreted by the students. Items 2, 3, 4, 5, 6, 7, 8 & 11 concerned student perceptions of the teacher, items 5 ("The teacher did his best for the class"), 7 ("The teacher was punctual and kept to the schedule"), and 8 ("The teacher respected and was open to the opinions and requests of the students") being covered in section "C" of research instrument 2 (Section 8.4.1.3; table A-57, items 21, 23, 26).  Items 2 ("The teacher clearly explained the goals of the course"), 3 ("The teacher helped the students by explaining everything systematically") and 11 ("The teacher gave the students sufficient information in advance about the evaluation criteria and methods") were particularly relevant to this study, since they dealt with an area identified by Hills (1976) as representing an "overwhelming need" for both students and teachers (cf. section 7.3.7), i.e." better communication and understanding between students and teachers" (Hills 1976:32-33). Items 4 ("The teacher tried to motivate the students") and 6 ("The teacher prepared for the class thoroughly and expertly") can also be found in Hills' combined student/teacher needs (section 7.3.7). Item 9 in the university instrument ("The textbook and teaching materials were appropriate for the students") covered similar ground to items 44, 45 and 46 of research instrument 2 (table A-59), and item 10 ("The evaluation method was appropriate for the course") introduced an important issue not yet covered in the other instruments (cf. Hills' item iv, section 7.3.7). Items 12 ("The students feel that the course helped them to become more interested in the subject matter than before"), 14 ("The students were satisfied with the course") and 15 ("The students feel that the course was of value and that it was worth taking") were similar to items 35, 41, 42, 47, 48, 49, in sections "D" & "E" of research instrument 2 (tables A- 58/9). Item 13 ("The students feel that they learned new things during the course") was a little ambiguous, since in 1998 the Junior year of the programme was being piloted with two groups of students, and the "process" approach to syllabus design had not been universally introduced. Learner-training was not overly explicit in the 1^st & 2^nd year syllabi, and students, who were learning how to access existing language and use it in communication, did not identify the development of CMI as "learning something new" (cf. section 8.5.3.6). Item 15 ("The classes are valuable") was also open to various interpretations.

Results for this instrument can be viewed in section 8.6.2 (below), tables A-70/1, and appendices D-79-92 and D 93-108 . Discussion of these results can be found in section 8.6.3 (below).

8.6.2 Instrument 4: results
Research item 4 was administered during lessons in the final week of semesters 1 & 2 1998, by Language Centre administration staff, on behalf of the University Central Administration Office (which collated the data and published it in a research booklet [Andong University 1999]), with results presented as mean scores from 0 to 5 for each item. Abbreviations in tables A-70/1 are explained in table A-69 and refer to the seven departments of the university, along with the Language Centre and "other" schools. Bar-chart representation of the results can be seen in appendices D-79-92 and D 93-108. Similar instruments were administered in 1999, but there was some problem with the computerised data collection process, and a booklet of results was not published. Tables A-70/1 and appendices D-79-92 and D 93-108 therefore refer only to 1998 results. Discussion of the results for instrument 4 follows in the next section.

8.6.3 Instrument 4: discussion
Immediately noticeable in tables A-70/1 and appendices D-79-92 and D 93-108 is the fact that the Language Centre received the highest average score in the University (3.93) in June 1998 (table A-70), and that it increased that score to 3.99 in December 1998 (table A-71). Looking at the items individually, there is general improvement over the 6 months, but English Conversation does not always score the highest marks compared with other departments. Thus, the syllabus (item. 1, table A-60) was not seen to be as "helpful" as for other subjects (appendices D-79/80), and students did not feel that they were learning new things (appendices D-103/104) compared with some other subjects. Apart from these items, the English Conversation Programme received consistently very high marks from students, reflecting the general opinion amongst teachers that "What we are doing is radically different from what's happening in every other class on campus" (table A-82). Thus students felt that the goals of the course were explained clearly (item 2, averages scores 3.82-3.95, appendices D-81/82), the teacher explained things systematically (item 3, averages scores 3.87-4.02 appendices D-83/84), the teacher tried to motivate the students (item 4, averages scores 4.05-4.04 appendices D-85/86), and the teacher did his/her best for the class (item 5, averages scores 4.21-4.24 appendices D-87/88). Language Centre marks were noticeably higher than those of other departments in terms of teacher preparation (item 6, averages scores 4.18-4.21, appendices D-89/90), teacher punctuality (item 7, averages scores 4.28-4.38 appendices D-91/92), mutual respect (item 8, averages scores 4.02-4.04 appendices D-93/94), the textbooks (item 9, averages scores 3.85-3.9 appendices D-95/96) and the evaluation method (item 10, averages scores 3.87-3.94 appendices D-97/98). Assessment (mid-term and final oral tests) had not been mentioned in research instruments 2 and 3, and it is notable that students were comfortable with them (item 10), and with the advance notice they received about the tests and the evaluation criteria (item. 11, 4.1-4.2, appendices D-99/100) - noticeably more so than in other departments, vindicating the policy of posting such information on the home page (http://lc.andong/ac/kr/eng/criteria.html) from day 1 of the course.

While scoring higher than all but one department in terms of "interest" (item. 12), these marks (3.68-3.72, appendices D-101/102) were lower than most other LC scores, and lower than the "new things" scores (3.71-3.82, appendices D-103/104) which have already been mentioned. "Satisfaction"  (item. 14) was also relatively low (3.81-3.84, appendices D-105/106), though higher than all other departments in June 1998, but the students felt that "the course was of value" (item 15, 3.94-3.98, appendices D-107/108) compared with other university courses.

This external instrument provides reassuring evidence on the perceived value of the language programme, its methods and teachers, since it was administered to every student and was independent of the Language Centre. Results showed that students placed conversation classes above all other classes in the university in terms of the assessed topics, with only the Music/Arts/Physical Education Department matching scores in the second semester. While this instrument gave no specific information in terms of CMI, it made it evident that students valued the English programme highly.

Continue reading Chapter 8: "Research Instrument 5"