CHAPTER 8: PROGRAMME EVALUATION (Continued)
Previous pages: 1, 2, 3, 4, 5

8.7 Research instrument 5: Student interviews.
8.7.1 Description
Kazuko draws attention to "renewed interests in both concurrent and retrospective verbal report techniques" (1996:143) resulting from studies on learner strategies, learner beliefs, attitudes, and perceptions (Wenden and Rubin 1987; Cohen 1990; O'Malley and Chamot 1990; Oxford 1990b; Wenden 1991a), so it was decided to complement research instruments 1 to 4 with structured interviews, investigating student perceptions of the programme. These interviews took place between the writer and three or four students (at a time) from each of 21 randomly selected classes (a Freshman, Sophomore and Junior class from each of the 7 university departments), during both semesters in 1999. This instrument (table A-61), which was designed to address research question 1 ("changes in beliefs", section 1.2) by monitoring students' responses over the year, was based on conversations with students and teachers during the pilot year (1997), and largely replicated issues that students had already met in the "Information section" of TMM, Chapter 2 of NYT and pages 21 & 105 of TWA (appendix C49; cf. section 7.3.2.2).  Research question 2 ("CMI") was highlighted in items 4 ("What are your learning problems?") and 5 ("How can you solve your learning problems?"), and item 6 ("How do you feel about your speaking skills?") related to research question 3 (perceived development of oral skills).

8.7.2 Research instrument 5: results
The first of the three groups of student-interview sessions (April 1999) took the form of evaluator-directed "learning conversations" (Harri-Augstein & Thomas 1991) about the programme, in the hope that matters significant to the students (and therefore of value to the study) would be raised. However, students either found it difficult to express themselves, or felt uneasy conversing with the interviewer in Korean, or were not comfortable expressing themselves in a recorded interview, A different approach was therefore taken in the 2^nd & 3^rd sessions (June and November 1999), when a list of 11 questions (table A-61, page 409) was devised, printed in English and Korean, and given to students before they came to the interviews. In the interview session, the interviewer read each question aloud, and students were free to respond in either language, either to the interviewer, or to a Korean student intern who was present, and who generally helped the students feel at ease.

Results for this instrument can be seen in appendices D-109-121. These graphic representations were obtained by listening to the recorded cassette tapes of the interview sessions, and classifying students' responses. By semester 2 (in addition to those unavailable due to sickness) some of the selected students had changed classes or had left the university to do their military service. As with other instruments, it was therefore difficult to collect comparative data over both semesters.

8.7.2.1 Research instrument 5, question 1: "What are your long term goals?"
Long-term goals covered a wide range of options (appendix D-109), but when the various preferred careers were combined (appendix D-110), it became apparent that "get a job" was the main priority[2], increasing from 22% to 51% of the responses over the two semesters. "fluency in English" came next (10.2/12.5/7.8%), rather surprisingly taking priority over "get married" (10.2/10.2/7.8%), though if "studying hard", "study TOEIC", "study my major", "study English" and "study computers" are combined in a general "study" category, they represent the second priority (13.6/19/7.9%). Responses such as "find a boy/girl-friend", "graduate from university", and "study my major", indicate some confusion regarding the meaning of "long-term", but it is noticeable that these decreased in importance over the research period, showing greater awareness of learning goals .

8.7.2.2 Research instrument 5, question  2: "What are your short-term goals?"
"Study" (including English) was again important in responses to this question (22.3/41.8/33.4% in its "combined" form) (appendix D-111), and "get a job" also figured (20/13/3.7%), though decreasingly so, perhaps indicative of decreasing confusion over definitions. A number of university-related topics emerged ("graduate", "get good grades", "prepare my thesis" "win a scholarship", and "pass university tests"), reflecting the reality of student life.

8.7.2.3 Research instrument 5, question 3: "Why do you need English?"
This question (appendix D-112), was designed to investigate student motivation and awareness of learning needs, particularly since "entry into university" tended to be the main motivation for studying English in high school (Lee 1991:77), and the intention was to monitor the change in students who had achieved that goal. "International language" (14.6/35.6/34.5%) was the most popular response to this question, with "necessary" (14.6/22.2/23.6%), "travel abroad" (31.2/15.6/7.3%), "conversation" (18.7/13.3/12.7%) and "get a job" (8.3/11.1/16.4%) following.

8.7.2.4 Research instrument 5, question 4: "What are your learning problems?"
This question (appendices D-113/114) was an important item, aiming to chart student awareness of learning strategies. Items such as "grammar" (3.9/10.4/3.8%), "vocabulary" (3.9/12.5/3.8%), and "lack of ability" (7.8/6.3/7.5%) appeared less prominently than expected (in view of students' learning history), whereas there was some awareness of the importance of confidence (11.7/14.6/7.5%). "Lack of effort" (5.9/12.5/17%) grew as a contributory factor over the two semesters, with "English is difficult" (13.7/14.6/13.2%) being the most consistently perceived learning problem. Although there is evidence of some passivity in these results (i.e. attribution of learning problems to external factors), there is also agreement (increasingly so in students from years 1 to 3) that learning problems, though not easily definable, are not solved by studying structure, and that individual effort and confidence are important.

8.7.2.5 Research instrument 5, question 5: "How can you solve your learning problems?"
This question (appendix D-115) continued to probe awareness of learning strategies, and student willingness to become more active in dealing with learning problems. Once more, "traditional" solutions, such as "studying TOEIC" (2.8/0/3.7%), "studying vocabulary" (2.8/0/0%), and "go to an institute" (0/2.1/0%) were not given panacea status. Instead, students showed some awareness of learning-style issues, mentioning items such as "study pop songs" (8.3/0/7.4%),  "language study tape" (decreasingly so - 16.7/8.5/3.7%), "thinking in English" (8.3/6.4/0%) and "speaking, listening, reading" (13.9/0/13%)[3]. As with question 4, however, the more frequent responses (e.g. "trying and trying" – 13.9/12.8/16.7%, "regular study" – 2.8/10.6/5.6%, "continue studying" – 2.8/21.3/16.7%) still confirmed general non-directional "study-related" ideas, typical of conventional "quantity-over-quality" learning styles, which Korean students adopt in order to pass the fact-based examinations which drive their secondary and tertiary studies.

8.7.2.6 Research instrument 5, question 6: "How do you feel about your speaking skills?"
Self-appraisal of oral skills (a factor closely associated with the "confidence" aspect of the CMI goals) was examined in this question (appendix D-116). The 100% "low" score of the first interview sessions confirms the general lack of confidence and the socially-acceptable modesty in terms of linguistic ability exhibited by Koreans in general[4].  The weakening of this perception in subsequent sessions (85.4/90%) can therefore be seen as important evidence of some sort of attitude change (however small) over the two semesters.

8.7.2.7 Research instrument 5, question 7: "How do you feel about your English class?"
The intention in this question (appendix D-117) was to monitor intrinsic motivation, and, as with the other questions in this instrument, responses are those which came from the students, rather than selections from which they were asked to choose. Various positive answers ("wonderful", "good", "fun", "interesting") account for 71.1/62.5/66.6% of the scores, more neutral opinions ("not bad", "so-so") provide 19.4/12.4/17.7%, and comments such as "difficult", "not interesting", "boring", "not helpful" and "too short" total 9.7/25.1/15.8%. Given the Sophomore "boredom" factor already mentioned (section 8.3.3), and the inevitable decrease of interest (for some students) in a 3-year compulsory language programme, these figures can be seen as positive.

8.7.2.8 Research instrument 5, questions 8 & 9: "How do you feel about pair-work/group-work in the class?"
Student opinions regarding pair-work and group work (appendices D-118/119) included "wonderful", "good", "not bad", "so-so" and "interesting", though a number of students found these study methods de-motivating if paired with peers of lower level. Since such a situation was almost inevitable in multi-level classes, and since Koreans place a high value on interacting with and helping friends, it is interesting that co-operation and peer-instruction (as advocated in the student-centred approach) had not been welcomed in these cases. The Korean culture encourages its members to join and to be loyal to groups such as the family, the company, the university department, the church and the political party. "Wang–da" thus refers to a person outside of the group, and is frequently heard in Korean classrooms. University students spend a weekend at the beginning of each academic year on "Membership Training", when "senior" students instruct their "juniors" (Freshmen) regarding appropriate conduct in school, and when strong bonds are made between students in the same departments.  In view of these considerations, it is possible that the "lower-level" students in questions were from another department (this was the case in some classes) and therefore not part of the "group".

8.7.2.9 Research instrument 5, question 10: "How do you feel about the classroom?"
This question (appendix D-120) was devised to investigate the affective issue of classroom environment (cf. Fraser 1985b; 1986). Responses, though generally positive were however general, since there was little opportunity in the interview to discuss what this question implied.

8.7.2.10 Research instrument 5, question 11: "How do you feel about homework?"
The final question (appendix D-121) investigated the topic of "homework", which students reported as increasingly "good" (22.2/24.4/52.8%) and decreasingly "difficult" (44.4/14.6/3.8%).

8.7.3 Instrument 5: discussion
Various concepts in this instrument (eg. "long-term-goals" - item. 1, "learning problems" – item. 4) (cf. table A-61) were initially unfamiliar to Freshman students who agreed to take part in the interviews. It was also a rare experience in ANU to be interviewed by a "professor" for research purposes, and even more so by a "foreigner". Having said this, research instrument 5 shows evidence of: 

1.      intrinsic motivation for learning English. "Get a job" was first in the list of long-terms goals (question 1, section 8.7.2.1), and English was an increasingly popular means of doing this (question 3, section 8.7.2.3). Studying English was also seen as important because of its international nature (question 3).

2.      awareness of the important of confidence and motivation. It was notable that some students chose "lack of confidence" and "lack of effort" as their learning problems (question 4, section 8.7.2.4), since these reflected goals of the programme.

3.      language-learning awareness. Also interesting are the responses to question 5 (section 8.7.2.5), in which students talk of "speaking, listening, reading", "trying and trying", and "continue studying". Although these ideas show general non-directed study-skills ideas, the fact that they are more popular than "studying TOEIC",  "studying vocabulary" and "go to an institute", is notable in the Korean situation, and indicative of gradual rejection of generally accepted "truths" about language learning.

4.      lessons and teaching methods are generally well-received. It is notable that responses concerning the English class itself were consistently positive (question 7, section 8.7.2.7), despite students' low perceptions of their abilities (question 6).

5.      lack of direction in terms of learning strategies. This has already been mentioned (question 5, section 8.7.2.5) and could be due to the implicitness of the approach to learner training in years 1 & 2 of the programme. It was evident that teachers had noticed their students becoming more confident, motivated, independent, and communicatively competent (cf. research instrument 3, section 8.5.3.1), but also evident that students were not noticing this.