Impact

=Evaluating What Really Matters in Computer-Based Education= toc Systematic evaluation of computer-based education (CBE) in all its various forms (including integrated learning systems, interactive multimedia, interactive learning environments, and microworlds) often lags behind development efforts (Flagg, 1990). There are several reasons for this lack of evaluation. First, consumers of technological innovations for education seem to assume that because these innovations are advertised as effective, they are effective. The fallacy of this assumption should be clear to anyone familiar with the generally poor success of CBE in most educational contexts (cf., Cuban, 1990; Siegel, 1994, Shlechter, 1991). Nonetheless, the dominant strategy of the business interests that underwrite the development of CBE has been and continues to be investing much more money in marketing CBE than in evaluating it.

Second, evaluation of CBE has often been reduced to a numbers game wherein the value of CBE is represented by 1) the amount of money spent on hardware and software, 2) the ratio of students to computers, or 3) the amount of time students have access to CBE within a school day, week, month, or year (Becker, 1992). The utility of such indicators in evaluating the ultimate effectiveness and worth of CBE is extremely limited, but their pervasiveness is obvious in the reports produced by national, state, and local education agencies around the world (National Center for Educational Statistics, 1993). This type of quantitative data is relatively easy to collect, analyze, and report. Further, judgments concerning progress within a specific educational entity (school, district, state, or nation) as well as comparisons among different entities can be rendered with a "certainty" that is untainted by the complexity of more ambiguous indicators such as measures of implementation, motivation, and learning.

A third reason for the lack of the evaluation of CBE is the inadequate utility of the evaluations that have been previously conducted. Evaluation reports are usually presented in the format of social science research reports, a format that "is almost useless for most clients and audiences" (Scriven, 1993, p. 77). Further, evaluations of CBE are rarely carried out in a manner timely enough to have sufficient impact on the decisions that must be made in the midst of significant development or implementation efforts. The inadequate utility problem will not be resolved unless educators create evaluation systems that are as integral to educational practice as student assessment systems are today (Reeves, 1992a). In addition, the results of evaluations must be communicated in formats that are accessible to as wide an audience as possible.

A fourth factor in the paucity of useful evaluations of CBE may be that evaluators often rely upon traditional empirical evaluation methods that compare an instructional innovation with another approach. Frequently the results of these studies have been disappointing (Clark, 1992). A major weakness in traditional empirical approaches to evaluation is that the treatments being compared (e.g., interactive multimedia versus classroom instruction) are often assumed to be cohesive, holistic entities with meaningful differences. Berman and McLaughlin (1978) and other implementation researchers (Cooley and Lohnes, 1976) have illustrated the fallacy of assuming that meaningful differences exist between two programs just because they have different names. It is imperative to open up the "black boxes" of instructional alternatives and reveal the relevant pedagogical dimensions they express if evaluations are to be meaningful and have utility. Pedagogical dimensions are the keys to unlocking the black boxes of various forms of CBE.

Pedagogical dimensions can be used to compare one form of CBE with another or to compare different implementations of the same form of CBE. Scriven (1993) maintains that there is an "almost universal necessity to do comparative evaluations" (p. 58), despite the tendency of some evaluation theorists to deny the utility of such comparisons (Cronbach, 1980). The "universal necessity" to conduct comparative evaluations is evidenced by the strong desire of most clients and audiences for such comparisons. Therefore, it is imperative that criteria for evaluating various forms of CBE be developed that will result in more valid and useful evaluations. That is the intent of this paper.

Purpose
The purpose of this paper is to describe fourteen pedagogical dimensions of CBE that have the potential to provide improved criteria for understanding, describing, and evaluating CBE. In physics, dimensions are used to describe a physical quantity or phenomenon in terms of certain fundamental properties such as mass, length, time, or some combination. For example, velocity has the dimensions of length divided by time as in "the car has a maximum speed of 120 miles per hour." Similarly, the phenomena that are forms of CBE can be described in terms of pedagogical dimensions. Pedagogy is defined as the art, science, or profession of teaching. Pedagogical dimensions are concerned with those aspects of the design and implementation of CBE that directly affect learning.

Pedagogical dimensions
Pedagogical dimensions refer to the capabilities of CBE to initiate powerful instructional interactions, monitor learner progress, empower effective teachers, accommodate individual differences, or promote cooperative learning. My first attempt to describe these dimensions was made at the 1992 Information Technology for Training and Education Conference, in Queensland, Australia (Reeves, 1992b). Since then, the dimensions have been revised based upon feedback from colleagues in Australia and the USA. This current set of dimensions is by no means final and further modifications are inevitable.

**1 - Epistemology**
Epistemology is concerned with theories about the nature of knowledge. A dimension of CBE important to users of these systems is the theory of knowledge or reality held by the designers. Figure 1 illustrates a dimension of CBE ranging from an objectivist theory of knowledge to a constructivist one. Tobin and Dawson (1992) describe these two theories in relation to interactive learning environments.

Objectivist epistemology (Thorndike, 1913) encompasses the following facets:


 * knowledge exists separate from knowing,
 * reality exists regardless of the existence of sentient beings,
 * humans acquire knowledge in an objective manner through the senses,
 * learning consists of acquiring truth, and
 * learning can be measured precisely with tests.
 * Constructivist epistemology (von Glasersfeld, 1989) encompasses different facets:
 * knowledge does not exist outside the bodies and minds of human beings,
 * although reality exists independently, what we know of it is individually constructed,
 * humans construct knowledge subjectively based on prior experience and metacognitive processing or reflection,
 * learning consists of acquiring viable assertions or strategies that meet one's objectives, and
 * at best, learning can be estimated through observations and dialogue.

If the designers and users of CBE lean toward an objectivist epistemology, they will be primarily concerned with assuring that the content of the CBE they create and implement is comprehensive and accurate with respect to ultimate "truth" as they know it. They will seek to establish the definitive structure of knowledge for a given domain based upon the advice of the most widely accepted experts in a field. For example, in science education, they will seek to transmit to students the "immutable laws" of any given field.

Advocates of constructivist epistemology, on the other hand, are much more concerned with assuring that the content in CBE reflects the complete spectrum of views of a given domain, ranging from the traditional academic perspectives to the views of the most radical "fringe." Constructivist epistemology calls for a multiplicity of perspectives so that learners have a full range of options from which to construct their own knowledge. In science education, constructivists might provide students with opportunities to rediscover the currently accepted theories of a given science as well as rival theories that may eventually replace the current positions. They might provide coaching or scaffolding to assist students in their discovery, but they would not overly direct the learning process. Constructivist pedagogy is increasingly popular in educational literature today, but few examples exist of its adoption in schools (Nix and Spiro, 1990).

Within education today, there is a tension between those who promote objectivist epistemologies and those who espouse constructivism. Within the context of CBE, the objectivist perspective is perhaps best represented by those who promote integrated learning systems (ILS), (Levinson, 1994), whereas the constructivist perspective may be best represented by those who promote electronic "mindtools" (Jonassen, in press). Major corporations have developed ILS for various sections of the primary, middle, and secondary school curricula. ILS are large scale, networked systems that integrate instruction, assessment, and management functions. In the USA, examples include the Integrated Learning System from Jostens Learning, and SuccessMaker developed by the Computer Curriculum Corporation, a subsidiary of Paramount Communications. These ILS have been produced to take over large portions of the established school curriculum and relegate teachers to the roles of "facilitators." Efficiency in attaining prespecified educational objectives is a frequently touted value of ILS.

In contrast, other educators are leading a movement away from a predominantly "instructivist" pedagogical culture to one that is "constructivist" in nature (Jonassen, in press; Papert, 1993). Instead of regarding knowledge as something that exists outside students which they must passively ingest, knowledge is recognized as being socially and individually constructed on the basis of experience. A recognition is growing that there is no "absolute" knowledge and that there is more than one viable perspective on knowledge in many areas, including mathematics and science. Electronic "mindtools" such as hypertext and multimedia provide opportunities for teachers and students to collaborate in the construction of unique knowledge representations. "HyperCard" from Apple Computer as well as spreadsheets and database programs are examples of "mindtools."

**2 - Pedagogical Philosophy**
Rieber (1992) and others (Duffy and Jonassen, 1992; Papert, 1993) make a clear distinction between instructivist and constructivist approaches to teaching and learning. Another way of thinking about these orientations is in terms of pedagogical philosophies. Figure 2 illustrates a dimension of CBE ranging from a strict instructivist philosophy to a radical constructivist one.

Instructivists stress the importance of goals and objectives that exist apart from the learner. These goals and objectives are drawn from a domain of knowledge, e.g., algebra, or extracted from observations of the behaviors of experts within a given domain, e.g., surgeons. Once goals and objectives are delineated, they are sequenced into learning hierarchies, generally representing a progression from lower to higher order learning. Then, direct instruction is designed to address each of the objectives in the hierarchy, often employing instructional strategies derived from behavioral psychology (Rieber, 1992). Relatively little emphasis is put on the learner per se who is usually viewed as a passive recipient of instruction. CBE based on instructivist pedagogy generally treats learners as empty vessels to be filled with learning. Direct instruction demands that content be sharply defined and that instructional strategies focus as directly on prespecified content as possible.

Alternatively, constructivists emphasize the primacy of the learner's intentions, experience, and metacognitive strategies. Rieber (1992) describes the constructivist view of learning as involving "individual constructions of knowledge" (p. 94). In this view, learners attain a state of cognitive equilibrium through reconstruction of concepts, schema, mental models, and other cognitive structures in the face of new information and experience that may conflict with earlier constructions. A major goal in constructivist pedagogy is to ensure that the learning environment is as rich as possible. Emphasis is placed on identifying the unique interests, styles, motivations, and capabilities of individual learners so that learning environments can be tailored to them. Instead of an empty vessel, the learner is regarded as an individual replete with pre-existing knowledge, aptitudes, motivations, and other characteristics that are difficult to assess, much less accommodate. Constructivists often argue for replacing direct instruction with self-directed exploration and discovery learning.

Different forms of CBE are based upon different pedagogical philosophies. Traditional computer-based tutorials, drill-and-practice programs, and contemporary ILS mesh well with instructivist pedagogies. Alternatively, interactive learning environments (Hannafin, 1992), microworlds (Rieber, 1992), and "mindtools" (Jonassen, in press) are forms of CBE that enable the implementation of constructivist pedagogy. It must be noted that the degree to which educators, parents, and community leaders emphasize one pedagogical philosophy over another appears to be strongly influenced by religious and political beliefs

**3 - Underlying Psychology**
At the risk of ignoring a number of other important theoretical perspectives (e.g., developmental psychology), a dimension related to the basic psychology underlying CBE is proposed. Figure 3 illustrates this dimension with behavioral psychology at one end of the continuum and cognitive psychology at the other

Debunking behavioral psychology has become quite fashionable, despite a few staunch defenders (Gilbert and Gilbert, 1991). Therefore, it seems ironic that behavioral psychology continues to be the underlying psychology for many forms of CBE. According to classical behavioral psychology (Skinner, 1968), the important factors in learning are not internal states that may or may not exist, but behavior that can be directly observed. Instruction consists primarily of the shaping of desirable behaviors through the scientific arrangement of stimuli, responses, feedback, reinforcement, and other contingencies. First, a stimulus is provided, often in the form of a short presentation of content. Second, a response is demanded, often in the form of a question. Third, feedback is given as to the accuracy of the response. Fourth, positive reinforcement is given for accurate responses. Fifth, inaccurate responses result in either a repetition of the original stimulus or a somewhat modified (often simpler) version of it, and the cycle begins again.

Cognitive psychology, on the other hand, has captured the attention of many educators today, and virtually all self-respecting instructional design theorists now claim to be cognitivists (Gagn and Glaser, 1987). Without ignoring behavior, cognitive psychology places much more emphasis on internal mental states than behavioral psychology. Kyllonen and Shute (1989) have proposed a taxonomy that represents the spectrum of internal states with which cognitive psychologists are concerned. Their taxonomy begins with simple propositions (e.g., stating that Japan sells more electronic products than any other nation), proceeding through schema, rules, general rules, skills, general skills, automatic skills, and finally, mental models (e.g., analyzing the potential of a trade war between Japan and the United States based on an analysis of balance of trade trends). The latter type of knowledge seems particularly important because mental models are the basis for generalizable problem-solving abilities (Halford, 1993).

Cognitive psychologists recognize that a wide variety of learning strategies may have to be employed in any given instructional setting depending upon the type of knowledge to be constructed. Learning strategies include memorization, direct instruction, drill-and-practice, deduction, and induction (Schank and Jona, 1990). Different forms of CBE vary in their capacity to implement these different learning strategies. Whereas an ILS may provide adequate opportunities for direct instruction and drill-and-practice, some sort of mindtool or microworld may be required to support deductive and inductive learning strategies

**4 - Goal Orientation**
The goals and objectives of CBE can range from sharply focused ones (e.g., following strict protocols for handling medical emergency situations) to more or less unfocused ones (e.g., learning to appreciate modern art). Figure 4 illustrates a dimension of CBE related to the degree of focus represented in the goals of an interactive program.

Cole (1992) clarifies the relevance of different types of goals to the design of CBE. She maintains that some knowledge "has undergone extensive social negotiation of meaning and which might most efficiently and effectively be presented more directly to the learner" (p. 29). In such cases, direct instruction, perhaps in the form of a computer-based tutorial, may suffice for learning. Other knowledge is so tenuous, creative, or of a higher level (e.g., mental models) that direct instruction is inappropriate. In the latter cases, CBE programs that promote inductive learning such as microworlds (Rieber, 1992), virtual reality simulations (Henderson, 1991), and learning environments (Hannafin, 1992) are much more appropriate.

Although, there are many advocates of discovery-based environments for the learning of social studies, science, and even mathematics in schools, most of these people would probably prefer their brain surgeons to be trained via direct instruction. However, there are examples of alternative approaches to learning being applied even in medical schools. Bransford, Sherwood, Hasselbring, Kinzer, and Williams (1990) describe the sequencing problem in the context of medical education. Most medical schools follow a sequence whereby students memorize great quantities of factual information during their first two years of training and then spend the next two years in various clinical settings where they may or may not have opportunities to use the memorized knowledge. A few enlightened medical schools have begun to place students in clinical settings from day one while providing them with the pedagogical support to learn basic knowledge and skills as needed. Perelman (1992) describes medical schools in Canada and The Netherlands that successfully employ this innovative approach.

Although it might be tempting to delegate the teaching of sharply-focused goals and objectives to commercial ILS, tutorials, and drill-and-practice programs, insufficient research has been conducted on which to base this decision. In addition, the infusion of mindtools, learning environments, and microworlds within traditional school curricula has been so limited that their effects on various types of learning goals and objectives are unclear

**5 - Experiential Validity**
The earliest type of systematic learning activity probably involved some sort of apprenticeship whereby a novice worked side by side with a master. Apprenticeships have high, i.e., concrete, experiential value. More abstract learning activities, e.g., classroom lectures, were developed much later in history. A major criticism of much of our current dominant pedagogical schemes is that they are too abstract, removed as they are from "real world" experience (Brown, Collins, and Duguid, 1989). Figure 5 illustrates an experiential value continuum ranging from abstract to concrete.

An important concern for educators and trainers alike is the degree to which classroom learning transfers to external situations in which the application of knowledge, skills, and attitudes is appropriate. The cognitive theories of Newell and Simon (1972), Anderson (1983), Brown (1985), and others support the fundamental principle that the way in which knowledge, skills, and attitudes are initially learned plays an important role in the degree to which these abilities can be used in other contexts. To put it simply, if knowledge, skills, and attitudes are learned in a context of use, they will be used in that and similar contexts. This principle is especially important in vocational education.

In traditional instruction, information is presented in encapsulated formats, often via abstract lectures and texts, and it is largely left up to the student to generate any possible connections between conditions (such as a problem) and actions (such as the use of knowledge as a tool to solve the problem). There is ample evidence that students who are quite adept at "regurgitating" memorized information rarely retrieve that same information when confronted with novel conditions that warrant its application (Bransford et al., 1990; Perelman, 1992).

CBE can be designed to present a focal event or problem situation that will serve as an "anchor" or focus for collaborative efforts among instructors and students to retrieve and construct knowledge (Brown et al., 1989). Cognitive psychologists at the Cognition and Technology Group at Vanderbilt University (CTGV) call this type of instruction "anchored instruction" (Bransford et al, 1990; CTGV, 1992) because the process of constructing new knowledge is situated or anchored in meaningful and relevant contexts. They maintain that events and problems presented in CBE should be designed to be intrinsically interesting, problem-oriented, and challenging. They have evidence that in response to these types of events and problems, students construct useful as opposed to inert knowledge (Bransford et al., 1990; CTGV, 1992).

**6 - Teacher Role**
CBE can be designed to support different pedagogical roles for teachers. Some CBE are designed to place teachers in the role of a "facilitator." Other programs are designed to support the more traditional didactic role of an instructor as "the teacher." Figure 6 represents a continuum of teacher roles ranging from didactic to facilitative.

The didactic roles of teachers are well-established. A quarter century ago, Carroll (1968) told us that "By far the largest amount of teaching activity in educational settings involves telling things to students..." (p. 4). More recent analyses of teaching indicate that little has changed since then (cf., Goodlad, 1984; Kidder, 1989; Perelman, 1992). Where teacher exposition is an appropriate instructional strategy, CBE can be designed to support, reinforce, and extend teacher presentations.

It has become commonplace today in education circles to talk about changing the teacher's role from a traditional didactic one to that of a facilitator. The Cognition and Technology Group at Vanderbilt (CTGV, 1992) describe a shift in the teacher's role "from authoritarian provider of knowledge to a resource who at times is consulted by students and at other times can become the student whom others teach" (p. 73). In addition to the constructivist learning environments such as the Jasper Woodbury Problem-Solving Series (CTGV, 1992), producers of large scale integrated learning systems (ILS) claim to assign teachers roles as facilitators. However, there may be important differences between the facilitator tasks of a teacher using Jasper and those carried out by a teacher implementing a commercial ILS. For instance, there is a danger that teachers using ILS may be so occupied with making sure the ILS are functioning properly and troubleshooting any problems, that they might not be able to conduct the one-to-one and small group teaching that the systems were supposed to allow teachers to conduct.

**7 - Flexibility**
A hidden agenda of some forms of CBE seems to be making them "teacher-proof," perhaps because of a belief that earlier instructional innovations have failed as a result of teacher interference (Winn, 1989). Alternatively, other forms of CBE exist in which teachers have considerable leeway to modify program activities. Figure 7 represents a continuum of program flexibility ranging from "teacher-proof," i.e., unchangeable, to "easily modifiable."

Teacher-proof approaches have fervent contemporary advocates. Some forecast the replacement of teachers with increasingly humanlike CBE. For example, Winn (1989) wrote "Educational technology can only become a viable discipline and profession if it concentrates on developing alternatives to the teacher-based model of public education rather than trying to alter it, improve it, or even just serve it" (p. 36). A new video created by ATandT as its vision of the future portrays students sitting in front of individual terminals interacting with computer-generated teachers while an adult stands nearby, seemingly in the role of monitoring their behavior (ATandT, 1993).

On the other hand, proponents of program flexibility must deal with the history of inadequate implementation that has hindered decades of educational innovations (Berman and McLaughlin, 1978). Modifying an innovative program has often resulted in insufficient fidelity in implementing a program's effective dimensions. Nonetheless, prohibiting local adaptation will lessen opportunities for creative modifications that may actually enhance effectiveness. The issue of program flexibility is a complex one that must be addressed by efforts to assess implementation very carefully during any evaluation. Forms of CBE must be designed to walk the fine line between being so "teacher-proof" that they do not allow local adaptation (and may even encourage sabotage) and being so open or unstructured that they do not provide sufficient guidance and support for valid implementation

**8 - Value of Errors**
The old maxim that "experience is the best teacher" reflects a belief that we learn much in life through trial and error (CTGV, 1992). Although this approach is inefficient and even dangerous in some contexts, experiential learning is highly valued simply because it provides opportunities for us to "learn from our mistakes." On the other hand, some educational theorists, especially proponents of programmed instruction, have maintained that ideal learning involves no errors. These developers attempt to arrange the contingencies of instruction in such a way that learners can only make correct responses. Figure 8 presents a continuum of perspectives concerning the value of errors ranging from errorless learning to learning from "trial and error" experience.

An example of a CBE program that prohibits errors is the Principles of the Alphabet Learning System (PALS) designed for the IBM Corporation by Dr. John Henry Martin (1986). PALS uses interactive videodisc technology to teach basic literacy skills to adolescents and adults. At specific intervals, learners are required to type in letters to form the words that on-screen characters say. However, only those keys that match an acceptable spelling of the words are enabled. Pressing the wrong keys puts nothing on the screen except more and more refined directions as to the desired response. This "errorless" approach is also an element of IBM's Writing To Read program (Freyd and Lytle, 1990).

Such an errorless approach contrasts sharply with forms of CBE that employ high fidelity simulation as an instructional strategy. In "The Case of Dax Cowart," an interactive videodisc simulation created at the Center for the Design of Educational Computing at Carnegie Mellon University (Covey and Cavalier, 1989), college students are placed in the roles of members of a hospital ethics panel that must decide whether a horribly burned patient can be allowed to die as he has requested or must undergo months of excruciatingly painful treatments. Regardless of a student's decision, he or she is confronted with the negative outcomes of that decision. In this simulation, each choice is treated as an "error" from which valuable lessons can be learned.

9 - Origin of Motivation
Motivation is a primary factor in instructional models (Carroll, 1963). Rieber (1992) describes five design principles for CBE derived from constructivism. The first is to "provide a meaningful learning context that supports intrinsically motivating and self-regulated learning" (p. 98). Intrinsic motivation has been held forth as the "Holy Grail" to which all CBE programs should aspire (Malone, 1984). Figure 9 illustrates a motivation dimension that ranges from extrinsic (i.e., outside the learning environment) to intrinsic (i.e., integral to the learning environment).

Intrinsically motivating instruction is very elusive regardless of the delivery system, but virtually every new approach to come along promises to be more motivating than any that have come before. Interactive multimedia is the latest type of interactive learning system that is supposed to motivate learners automatically, simply because of the integration of music, voice, still pictures, text, animation, motion video, and a friendly interface on a computer screen. In practice, as Keller (1987) has specified, motivation aspects must be consciously designed into multimedia just as rigorously as any other pedagogical dimensions. An assumption underlying many commercial multimedia packages seems to be that students will be intrinsically motivated to explore these systems in search of new knowledge. Very little research exists that examines this assumption, but Harmon (1992) found that students using these programs were more likely to seek out confirmation of things they already knew than to seek new knowledge. It seems that the current state-of-the-art of CBE is such that extrinsic motivation will remain a critical factor in many educational contexts

10 - Accommodation of Individual Differences
Although it might be assumed that the main reason for employing CBE would be accommodating individual differences among learners, this is not always the case. Some CBE programs have very little, if any, provision for individual differences whereas others are designed to accommodate a wide range of individual differences including personalistic, affective, and physiological factors (Ackerman, Sternberg, and Glaser, 1989). Figure 10 illustrates a continuum of accommodations of individual differences that ranges from non-existent to multi-faceted

The impact of individual differences is a major factor in the effectiveness of CBE. Learning is a function of the learner, the content to be learned, and the features of the instruction (Sternberg, 1985). Many theoretical models of learning treat individual differences among learners as the major predictor of differential learning outcomes (cf., Carroll, 1963). In most educational contexts, we cannot be guaranteed that learners will be homogeneous in terms of aptitudes, prerequisite knowledge, motivation, experience, learning styles, eye-hand coordination, etc. Therefore, we must provide scaffolding, cognitive bootstrapping, and other types of metacognitive support to promote learning (Resnick, 1989). Examples of CBE that provide comprehensive metacognitive support are difficult to identify (Cates, 1992)

**11 - Learner Control**
Learner control has been one of the most heavily researched dimensions of CBE in recent years (Steinberg, 1989). Figure 11 illustrates a dimension of CBE that can range from complete program control to unrestricted learner control.

Learner control refers to the options in CBE that allow learners to make decisions about what sections to study and/or what paths to follow through interactive material. The popular wisdom is that learner control makes CBE more effective by individualizing the instruction and making it more motivating, but all too often experimental studies have led to no significant results in terms of the predicted main effects (Williams, 1993). Reeves (1993) describes critical theoretical and methodological flaws in learner control studies. Ross and Morrison (1989) concluded that "research findings regarding the effects of learner control as an adaptive strategy have been inconsistent, but more frequently negative than positive" (p. 28). Better research is needed before questions about the learner control issue can be answered (Reeves, 1993).

**12 - User Activity**
Hannafin (1992) identified another important dimension of CBE, especially those forms of CBE that he and others characterize as "learning environments." He maintains that some learning environments are primarily intended to enable learners to "access various representations of content" (p. 59). He labels these "mathemagenic" environments. Other learning environments, called "generative" by Hannafin, engage learners in the process of creating, elaborating or representing knowledge. Figure 12 illustrates this continuum of user activity.

Generative learning environments are aligned most closely with constructivist pedagogy whereas mathemagenic environments are often based upon instructivist pedagogy, but this is not necessarily always obvious. Contemporary CBE programs such as the ABC News Interactive series (ABC News Interactive, 1991) and the IBM Ultimedia programs (IBM Corporation, 1991) include generative capabilities nested within otherwise mathemagenic presentations of content. On the other hand, mindtools such as HyperCard have great potential for enabling generative learning (Jonassen, in press).

**13 - Cooperative Learning**
Support for the value of cooperative learning is growing throughout education circles (Slavin, 1992). CBE can be designed to thwart or promote cooperative learning. In fact, some CBE programs require cooperative learning (IBM Corporation, 1986) whereas others make no provision for its support. Figure 13 illustrates a cooperative learning dimension ranging from a complete lack of support for cooperative learning to the inclusion of cooperative learning as an integral part of CBE.

Cooperative learning refers to instructional methods in which learners work together in pairs or small groups to accomplish shared goals (Slavin, 1992). Johnson and Johnson (1987) and Slavin (1990) present evidence that when CBE (and other instructional delivery systems) are structured to allow cooperative learning, learners benefit both instructionally and socially. Some commercial ILS have been designed to be used by two or more learners working cooperatively. In addition, multimedia construction programs (such as Authorware Professional and Macromind Director) are so complex that they usually require team-based usage in school contexts.

**14 - Cultural Sensitivity**
Henderson (1994) provided a valuable critique of an earlier version of these pedagogical dimensions (Reeves, 1992b). Henderson maintains that the assumptions underlying a specific point on any of these dimensions has a cultural element that should not be ignored. For example, whereas a constructivist pedagogy advocates, indeed demands, persistent questioning on the part of learners, questions, especially "why?" questions, are inappropriate in cultures such as the Torres Strait Islanders of Australia. Although CBE may not be able to adapt to every cultural norm, they should be designed to be as culturally sensitive as possible (Powell, 1993). Figure 14 illustrates a cultural sensitivity dimension ranging from non-existent to integral

Powell (1993) revealed that few instructional design courses include cultural diversity as an important factor in designing effective instructional programs. Therefore, it should not be surprising that few CBE programs have been developed in which cultural sensitivity is integral to their design. To be sure, a few instances of CBE include what Henderson (1994) labels "tokenistic gestures" by allowing an occasional minority role for an actor or perhaps by including culturally diverse, albeit safe, references in terms of music, location, or other cultural aspects. It is difficult to describe what comprehensively culturally sensitive CBE would be like, but at the very least such programs would accommodate diverse ethnic and cultural backgrounds among learners.

Application of the dimensions in CBE evaluation
To illustrate the potential utility of the pedagogical dimensions of CBE described above, the last part of this paper presents an analysis of two examples of CBE employing these dimensions. Experienced developers and users of CBE possess the necessary background and objectivity to provide a reliable and valid assessments of the pedagogical dimensions of these systems. Although the final ratings reported below are mine, they have been influenced by colleagues in Australia and the USA with whom I have discussed these programs. There is an inevitable degree of subjectivity in this analysis and additional applications of these dimensions involving experienced personnel in other education contexts are invited.

The two examples of CBE used in this application are the Writing To Read program designed by Dr. John Henry Martin and widely disseminated by the IBM Corporation (IBM Corporation, 1985) and the Jasper Woodbury Problem Solving Series developed by the Cognition and Technology Group at Vanderbilt University (CTGV, 1992). The Writing To Read (WTR) program is intended to improve the reading and writing performance of students in kindergarten and first grade. During WTR periods lasting an hour per day, children rotate among five workstations, two of which involved CBE. The primary computerized workstation in WTR provides students opportunities to learn and practise phonics skills. Computer guided activities include keying in sounds, words, and eventually sentences. The program emphasizes the learning of 42 phonemes, letter-sound combinations that can be used to "spell" any words in the English language. Often, the computer requires verbal as well as keyed in responses, and occasionally students are prompted to clap or stomp their feet in time with computer presentations.

Few examples of CBE have been more extensively evaluated than WTR and few programs are more controversial. Slavin (1990b) concluded that the results of WTR are disappointing. On the other hand, Chira (1990) described the enthusiastic reception of WTR as a statewide program in Mississippi. Estimates are that more than ten percent of the kindergarten and first grade students in the USA used WTR in the 1992-93 academic year, making it one of the largest implementations of CBE in any setting.

The Jasper Woodbury Problem Solving Series (CTGV, 1992) was created in an academic environment within the context of a long term research and development program. Its use until recently has been confined largely to a few dozen schools in the southeast section of the USA, but it is now commercially available. The Jasper Series represents an attempt to implement constructivist learning principles. These programs (which are provided in both interactive videodisc and linear video versions) provide students with opportunities to develop advanced mathematical problem-solving skills within the context of a series of high-interest video adventures. Students discover the need to develop mathematical skills within the context of flying planes and operating motor boats to solve simulated dilemmas. Numerous studies have been and are being conducted using the Jasper series of programs (Bransford et al., 1990).

The Jasper series is an example of what Hannafin (1992) calls a "generative" learning environment, i.e., a program that requires students to construct or generate their own knowledge as opposed to one that requires them to select knowledge from prepackaged options. Knowledge constructed in generative environments is more likely to generalize than the inert knowledge acquired in traditional passive learning environments (CTGV, 1992).

Figure 15 presents a profile of the Writing To Read and Jasper programs using fourteen pedagogical dimensions. My ratings of these programs are based on limited observations in schools, demonstrations of the programs at professional conferences, and reading several extensive reports about them, but not first-hand experience in implementing the programs myself. My analysis reveals that WTR is based on objectivist, instructivist, and behavioral foundations. It is a highly structured program. One of its most notable features is its provision for errorless learning, e.g., students are not allowed to key in incorrect responses to questions. Alternatively, the Jasper programs are grounded in constructivist and cognitivist foundations. Teachers are integral facilitators in implementing Jasper, and they are encouraged to modify it according to their local needs. Collaborative learning is strongly supported in this program. It appears to be an advanced example of a generative learning environment.

Conclusion
The preceding analysis is an admittedly preliminary investigation into the value of these pedagogical dimensions. Hence, the following recommendations are made for improving their utility. First, the dimensions should be subjected to rigorous expert review by leaders in the design and application of CBE. Second, once there is evidence for the qualitative validity of the dimensions, quantitative scales should be integrated into each dimension, e.g., a ten point rating system. Quantitative values have not been added to the dimensions up to now for fear that reviewers might get too distracted by the numerical values to concentrate on the qualitative aspects of the dimensions themselves. However, there is certainly merit and utility in eventually grounding the ratings in quantitative values. Third, the validated dimensions should be applied to many different forms of CBE in a wide variety of educational contexts to provide evidence for their utility. Fourth, research should be initiated into the relationships among ratings of the pedagogical dimensions of CBE and actual data regarding the instructional effectiveness and impact of these same programs.

The fourteen pedagogical dimensions described above are by no means the final answer to improving evaluations of CBE in education. A comprehensive approach to evaluating CBE requires multiple levels of design, data collection and interpretation. We must explore many alternatives. Each month sees the introduction of new commercial CBE packages advertised as effective instructional systems. Yet systematic evaluation of the implementation and efficacy of these systems is sadly lacking. In addition, many evaluators continue to employ outmoded experimental designs. Papert (1993) sums up the inadequacy of these traditional evaluation designs: "The method of controlled experimentation that evaluates an idea by implementing it, taking care to keep everything else the same, and measuring the result, may be an appropriate way to evaluate the effects of a small modification. However, it can tell us nothing about ideas that might lead to deep change" (p. 27).

In education today, we need "deep change," and therefore improving evaluation of CBE has never been more important. Technological advancements are increasing at an ever faster pace especially with respect to telecommunications and multimedia. At the same time, few teachers feel confident and competent with respect to the goals and functions of CBE in their classrooms (Becker, 1992; Siegel, 1994). Despite some efforts to introduce pre-service teachers to computer education in their teacher preparation programs, Becker (1992) found that over half of the future teachers he surveyed never used a computer in any of their college courses. At least part of the problem may stem from a restricted vision of CBE as simply an alternative delivery system for traditional pedagogy rather than as a tool for implementing alternative pedagogical dimensions. Evaluation approaches based upon clearer delineation of the pedagogical dimensions within different types of CBE will surely be a step forward.