CHAPTER 8: PROGRAMME EVALUATION

8.1 Introduction 

... how remarkable it is that the scientific study of the world led to the content of consciousness as an ultimate reality. (Morowitz 1982:39)

Principles of programme evaluation, and the rationale behind choosing a formative, naturalistic evaluative approach, have been discussed in sections 1.3 and, so this section concentrates on issues specific to this chapter, one of which is the perspective of language as a complex dynamic system (Parlett & Hamilton 1975:141; Larson-Freeman 1997:143; section 6.6), in which "details are all that matters" (Gould 1993), and in which "the act of playing the game has a way of changing the rules" (Gleick 1987:24). Thus meaningful use of a language can change not only the grammar system in the user (Diller 1995:116), but also the language itself ("by virtue of using the target language, it is transformed" [Larson-Freeman 1997:151]), since "a language such as English is a collaborative effort of its speakers, and changes in the system of English are ‘emergent'" (Diller 1995:117). There is no final goal in this process, since the very act of studying the language changes it (cf. interlanguage, "Konglish"[1] [ Chang, Suk-Jin {1996}; Kent {1999}], "Finglish"[2] [Karttunen {1975}]), and the act of researching these changes contributes to that which is being researched: "the target is always moving" (Larson-Freeman 1997:151):

... the behavior of the whole emerges out of the interaction of its parts. Studying the parts in isolation one by one will tell us about each part, but not about how they interact. (Larsen-Freeman 1999:157)

Examination of the dynamic psycho/social construct of the language learning classroom and its linguistic outcomes must therefore examine "what is in fact meritorious, or really valuable" (Scriven 1991:26), in terms of these interactions (see above).

The present study, while not attempting to derive Lucas' (1999) "methodology of considerable scope"[3], does attempt to show how complexity theory and evaluative reasoning (Scriven 1991:216) might be applied to the diversity of the research situation (the language learning classroom), the uniqueness of its participants and the impracticality of attempting to predict local results (cf. Section 6.6). In doing this, the study places prime value on the qualitative analysis of the opinions and perceptions of its participants (in addition to some quantitative representation of changes in perceptions). This "evidence" is necessarily subjective and inferential, but represents reality for the people who are the programme:

The human psyche is part of the observed data of science. We can retain it and still be good empirical biologists and psychologists. (Morowitz 1982:42)

Therefore, before proceeding onto presentation and discussion of results, it is appropriate at this stage to clarify terms such as evaluation, results and success, in terms of the view of reality (above) that is being used in the study.

Evaluation. In the "emergent, variable design" (Guba 1978:14) of formative evaluation, the emphasis is on "observing, describing, interpreting and understanding how events take place in the real world" (Lynch 1996:14), and on "stimulating learning and understanding" (Rea-Dickins & Germaine 1998:11). In this approach there is no meaningful separation of facts from values (Guba and Lincoln 1989:45), and the educational programme is seen as a process that is continuously changing, shaped by "the nature of the project itself" (Williams & Burden 1994:22), and evaluated through interviews, participant observation, journals, etc. (Lynch 1996). This type of evaluation is "participative ... principled, systematic and an integral part of curriculum planning and implementation" (Rea-Dickins 1994:72). The various stages of a formative evaluation are cyclic and overlapping, and the programme is "always in a state of being established, implemented, evaluated and reformed, each stage interacting with and influencing the others", the evaluator producing an "interpretation of a highly complex system" (Parlett 1981), rather than making predictive generalisations (Cronbach 1975), and using the information to modify the programme as it is being implemented (Long 1984:417).

Results. From this standpoint, results include the processes of innovation (introduction, implementation, diffusion) (Kennedy 1988:329), and of programme growth, in addition to the data which feed into that process, and this chapter, while investigating findings from research instruments (sections 8.3 – 8.8), will also comment on formative processes as they appear.

Success. This study researches the success of a task-based conversation programme through triangulated[4] qualitative-quantitative instruments (Lynch 1996:10), as indicators of relative achievement of programme goals, with reference to specified research questions. These questions were subject to the formative nature of the evaluation, contributing to their own change as they were being used (section 1.2). Originally outcome-based (e.g. "1. Did learner/teacher beliefs change during the research period?"), they were expanded to include process-oriented sub-questions (e.g. "How did they change?") as the emphasis moved to the process of learning (Legutke & Thomas 1991), and to connectivity over composition  (Waldrop 1992:66; Lucas 1999) (cf. Allwright's comment that "it may be that interaction is what somehow produces linguistic development", 1984b:9). "Programme success" therefore includes "process" and "customer satisfaction", on the assumption that a more confident, motivated and independent (i.e. "successful") learner is more aware of the language-learning process and is satisfied that he/she has the tools to proceed with it.

8.2 Evaluation instruments
Six instruments were used to triangulate data over the years 1998-1999. These consisted of four quantitative/qualitative[5] questionnaires and two interviews/verbal reports (Cohen 1996). In addition to their research role, instruments 1, 2 and 3 were also integral formative components of the programme, being consciousness-raisers (helping participants progress from awareness of beliefs to reflection on and modification of those beliefs, and to more efficient and effective learning) and providing data for the ongoing evaluation and transformation of the programme. The research instruments are listed below:

1. Research instrument 1: "My abilities" (self assessment), section 8.3.
2. Research instrument 2: Internal questionnaire (students), section 8.4.
3. Research instrument 3: Internal questionnaire (teachers), section 8.5 .
4. Research instrument 4: External questionnaire, section 8.6 .
5. Research instrument 5: Student interviews, section 8.7 .
6. Research instrument 6: Teacher interviews, section 8.8.

Continue reading Chapter 8: Research instrument 1: Self-assessment.