Validity and the Relevance of Research to Practice

2005

Validity and the Relevance of Research to Practice

Ernest R. House

How is it possible for research to be relevant to practice if the researcher cannot guarantee that event A will be followed by result B, the bedrock of the standard view? This issue has never been successfully resolved in the “standard” view of research, either in theory or practice. The usual attempt at resolution is to divide research findings into the internally valid and the externally valid. In the standard view the researcher discovers that a program has certain results in particular settings (internal validity) and then attempts to find similar settings in which the program will produce the same results (external validity). Like causes produce like effects. However, the research literature is full of cases in which programs did not have similar effects, and practitioners do not use much educational research.

As Glass has noted, "The internal validity of an experimental finding cannot be judged solely in terms of the properties of the experimental design.... Internal validity of a causal claim is a complex judgment that ultimately rests on a larger number of empirical facts that necessarily extend beyond the context of a single experiment. The external validity of an experimental finding rests similarly on judgment and a network of empirical facts.... the facts are more numerous and are less well established" (Glass, 1982). One cannot depend solely on the formal characteristics of the research design to draw inferences. One must have substantive knowledge. This is true both in drawing inferences from a study and in applying the study elsewhere. To address this problem, Cronbach (1982) has argued that external validity is more important than internal validity, contending that there are so many unexplored and unknown interactions between the program and the setting that participants in the implementing site must themselves extrapolate the research findings into their own settings by adjusting the findings to the circumstances they encounter, and only the practitioners themselves can do this. Researchers can help by conducting studies in ways that help participants extrapolate the research findings, but researchers cannot guarantee universal results for a particular program.

In a realist view although patterns of events do not repeat themselves exactly, there are transfactual causal structures that influence events and that operate in different settings, even though their interactions with other causal mechanisms may not produce the same events from site to site. The realist would expect programs not to have the same effects in different sites and circumstances. However, transfactual entities can be causally efficacious across sites, though effects might be amplified or cancelled by other factors. Hence, a goal of research is to discover entities that tend to produce effects. For example, teachers are causal agents, and it is a commonplace that particular teachers make a tremendous difference not only in the classroom but also in the implementation of educational programs. Parents even act on this knowledge by putting their children into the classes of the best teachers, if they can. However, one can never be certain that a particular program or a particular teacher will produce good results. The standard view has carried with it the implicit image of program participants as compliant agents who follow administrative directives and whose own views and particularities make little difference, a view refuted strongly by many implementation studies that have been conducted (Fullan, 1982).

In the standard view programs are conceived as discrete, reproducible activities that are sufficient to produce particular results under similar circumstances. However, if we can conceive of programs themselves more like events than like causal entities. The programs themselves are produced by such entities. We can evaluate programs on a particular site with considerable efficacy, even while not knowing exactly which important causal structures are at work in what interactions, but we cannot expect that same program to have the same effects on other sites with great confidence. We can expect a major causal entity, such as a good teacher, to be effective in other settings with similar students, but even this transfer is not certain. There are ways of dealing with this complex social reality. If all these interactions occur in somewhat unpredictable ways so that causal laws can never be more than statements of tendencies, we might try to average across these situations in some way to discern the tendencies, such as in meta-analysis studies (Glass, McGaw, and Smith, 1981). Meta-analysis does not depend on a single critical experiment but rather summarizes across many different studies in an attempt to discern general tendencies. A program that contains strong causal structures might be expected to produce effects on average. Meta-analysis has indeed yielded results when other methods of investigation have failed, such as in discerning the effects of reducing moderately high blood pressure on heart disease.

There is a further extension of this reasoning. If teachers and other practitioners themselves are strong causal agents, able to affect the production of events dramatically, then their intentions and their knowledge are also important factors in good educational programs. A teacher's knowledge consists not only of subject matter but also of knowledge of concrete interactions of particular students in the classroom. Good teachers possess knowledge of what is likely to happen with particular students when certain activities occur, and in fact teachers may know that each single student may respond in a different way to certain classroom activities. That is, the teacher possesses specific causal knowledge built on inferences constructed from different sources over a period of time. This knowledge is focused on particular students and the concrete conditions of particular classrooms.

Being able to act effectively in such settings entails knowledge of concrete causal entities--what is likely to happen with these students, even with this particular student. So teachers can strongly influence particular students by specific activities based on knowledge of that student. Of course, the teachers may be wrong in the inferences drawn and the activities initiated. Improvement of the teachers’ causal inferences themselves, based on the particulars of the teachers’ students and classroom, would seem to be an important strategy for improving practices like education. Unfortunately, in the search for general laws, not much attention has been paid to improving particular teacher and practitioner concrete inferences directly. Given such a strong conception of agency from a realist point of view, the standard distinction of internal validity and external validity is inadequate. Rather we might think of the validity with which researchers draw conclusions from their studies, the validity with which practitioners draw conclusions from these studies to their own situation, and the validity with which teachers and other practitioners draw conclusions for themselves based upon their own experiences (House et al, 1989). A critical test for realism in evaluation and research is that it be realistic in practice as well.

            Validity

Few concepts are as fundamental to educational research and evaluation as that of validity. Notions of validity are based on particular views of causation. The regularity theory of causation undergirds traditional conceptions of validity, but such a theory is inadequate for describing and explaining many the causal inferences required in professional practice and everyday life. Teaching, for example, is a practice better understood in terms of a theory of intentional causation. This has significant implications for what counts as valid teacher knowledge.

An Example of Practical Knowledge: Learning to Teach

Let us imagine a situation in which a person is learning to teach for the first time, say, a woman instructor who is faced with teaching her first class. How does she proceed? A likely scenario is that as a student she has had teachers she thought were particularly effective or ineffective. She tries to remember what they did that worked with classes in which she was a student, as well as what didn't work very well. Based on her own experiences in the classroom as a student, she has notions of cause-and-effect relationships, of what works and what doesn't work. Some of these ideas may well be mistaken, but she holds them nonetheless.

From this repertoire of ideas and techniques, she selects notions around which to organize her class. How many of these considerations will there be? Ten? Fifty? No doubt it depends on the person and situation. There will be many. What they have in common is that most will be based on the new teacher's actual experience as a student participating in former classes. A student, after all, is a participant rather than merely an observer. But even all this is only preparatory to learning to teach--trying out these ideas in the classroom. As the new teacher begins to teach, the general considerations of how to act (should she be highly organized? authoritative? flexible? well-dressed?) give way to more specific considerations of exactly what to do (should she lecture? lead group discussions? show movies?).

The new teacher learns cause-and-effect relationships through direct participation, through participating first as a student and then as the teacher. This direct experience is gained mostly by performing and acting rather than by passively observing, and this direct personal experience is so intense and powerful that it shapes what the teacher will do and try to do throughout her career. After a few years, her learning rate will decline because she will feel that she has mastered her environment. Her teaching repertoire will be largely formed.

This direct firsthand experience of learning to teach is better explained by the intentional theory of causation than by the Humean regularity theory. The teacher has something in mind, tries it out, and judges its success or failure. The determination of whether the lesson works is based largely on firsthand experience, on performing, and through those experiences the teacher develops a personal set of cause-and-effect relationships about teaching. These are singular causal claims and not dependent on universal laws or regularities that assert universal correlations of events. Rather, they are based on personal experience. They well may be mistaken, but mostly they are not. The teacher can develop a reasonable set of cause-and-effect relationships to guide her through the day, just as most of us manage to drive our cars to work, feed ourselves, and conduct our daily affairs. All this is not ordinarily a problem, except perhaps when the car won't start, because causation is not always the problem that the skepticism of the Humean regularity theory suggests. Causal inference guides our lives every day, which is not to say that either our lives or the teacher's performance cannot be improved. It is to say that most of what we do is rational and makes good sense.

The teacher learns to teach not through observing her own actions as a spectator but through performing certain actions. The direct experience of acting is the basis for the cause and-effect relationships she learns. She does not infer the essential cause-and-effect relationships from repetition or regularity or universal causal laws. One can ride as a passenger in a car and witness the passing scene, yet not be able to retrace the route that one has taken. If one is the driver of the car, however, there is intentionality to one's action, perhaps responsibility, which makes it highly likely that one has learned the route. Similarly, one can sit in hundreds of classes for 20 years and not learn how to teach, but one can learn to teach by performing the teacher role for only a few semesters. For better or worse, many cause-and-effect relationships seem directly discernible and form the basis of professional practice. Many cause-and-effect relationships cannot be discerned through personal performance.

If this is a reasonable account of how teachers learn to teach, then what can we say about how valid their knowledge is? Is the validity of the inferences of the teacher captured by our traditional notions of validity? Or must we look elsewhere for conceptions that more adequately represent the state of their knowledge, and consequently discover new ways of improving the validity of teacher inference? We address this issue by examining two powerful conceptions of validity, those by Cook and Campbell (1979) and Cronbach (1982).

Cook and Campbell's Formulation of Validity

The traditional conception of validity has been explicated by Cook and Campbell (1979) in their revision of Campbell and Stanley (1963) and Campbell (1957). Cook and Campbell pose four research questions with corresponding types of validity: statistical conclusion validity, internal validity, construct validity, and external validity. Cook and Campbell believe that a precondition for inferring a causal relationship between two variables is to establish that the two variables covary. For the most part, Cook and Campbell's discussion of statistical conclusion validity is limited to variability and sampling error in units.

The second question is whether one variable caused the other, whether the treatment really caused the outcomes. The truth of this relationship is called internal validity. At this stage of Cook and Campbell's formulation, neither the treatment variables nor the outcome variables have been given a name that might generalize.

To generalize, one must label the cause and the effect. In other words, the cause and the effect must be related to higher order constructs, and the researcher generalizes to higher order constructs from the research operations. Inferences are based on the fit between the operations and the conceptual definitions (Cook & Campbell, p. 38). In practice, construct validity usually involves specification of the outcome measures and what they represent. As Cook and Campbell note, proper labeling of the treatment is a critical and often overlooked problem of construct validity.

External validity, the fourth type of validity for Cook and Campbell, addresses the question of how the causal relationship generalizes to and across other persons and settings. When researchers want to generalize to a particular population, it is essential that their samples be representative in some way. When all four types of validity are taken together, they permit the final inference. In Cook and Campbell's judgment, internal validity is the most important concern of all. In applied settings, they believe that external validity and construct validity of the effect are also relatively important, but that internal validity is still the most important.

Cronbach's Formulation

By contrast, Cronbach (1982) divided the world of inquiry and action into two domains: the domain of admissible operations and the domain of application, using a notation system of his own device, similar to the traditional X's and O's of experimental design. The first domain is the area of the study or investigation, and it is further subdivided into four components: units (U), treatments (T), observations (O), and the setting (S), or, collectively, UTOS.

However, UTOS represents only the stated plan for the study. Actual participants must be selected, actual treatment procedures applied, and actual observations made. The actual study is represented by utoS. The small letters indicate that one has sampled participants, treatments, and observations. The setting is ordinarily uncontrollable and does not warrant a small s, Cronbach believed.

What one finds in the utoS is only an imperfect manifestation of the original plans. The domain of admissible operations, UTOS, defines what is admissible to the study, that is, what range of U's, T's, and O's can be included. The reproducibility of the inference about UTOS from the actual utoS is what Cronbach calls internal validity. This inference is internal to the UTOS domain.

In Cronbach's formulation, there is also a second domain, the domain of application. Suppose that the school superintendent in Seattle hears about the evaluation of a Direct Instruction program in East Saint Louis, reads the evaluation report, and decides to do something similar in Seattle. But Direct Instruction has certain features that she doesn't like. The U, the T, and the O are different, not to mention the setting. Yet the superintendent does not disregard the evaluation findings. She makes some mental adjustments and arrives at her own conclusions, which are not exactly the same as those of the original study. Cronbach represents the new domain of application by *UTOS. This is the domain of the Seattle superintendent, and the leap made by her is from the sample of the original study to her own situation (utoS^*UTOS). Cronbach calls this “external” validity because it is external to the original UTOS domain of the study.

As formulated by Cronbach, internal validity (utoS^UTOS) is a matter of judging whether u, t, and o belong to the stated domain, and this is done by the investigator. It is the investigator's task to draw the proper inferences, given the stipulation of the domain. External validity for Cronbach is different, however. It requires an inference from the study to a domain outside the investigation (utoS^*UTOS), and it is not a matter of sampling. The rules of statistical inference do not apply. Cronbach calls this external inference an extrapolation, a projection of the information outside the range of the study. This inference requires substantive modifications in reasoning contrasting the similarity of the two situations or domains.

Note that the external validity of the inference (utoS^*UTOS) is not directly dependent on the internal validity of the inference (utoS^UTOS), as the inference is in Cook and Campbell's formulation. A conclusion may have good internal validity but may not extrapolate to the domain of application. And, more surprisingly, a conclusion may extrapolate externally without being first validated internally, contrary to the familiar idea that the external validity of a conclusion first must be established as internally valid.

It is apparent that there are important differences between these two conceptions of validity. For Cook and Campbell, statistical conclusion validity and internal validity lead to a conclusion that A caused B in a particular instance. Then, through construct validity and external validity, the researcher can generalize beyond the particular research operations to higher level causal relationships in other populations and settings (utoS^UTOS and possibly UTOS^*UTOS). The reasoning procedure is one of relatively strict inference from the study to a larger domain.

By contrast, Cronbach split the world into two domains. In the first, the researcher generalizes from the particular study to a larger domain, like Cook and Campbell (utoS^ UTOS), and this is called internal validity. External validity consists of drawing conclusions from the particular study to a different domain altogether (utoS*^UTOS), which may not resemble the original study in important ways. This generalization may be done by someone other than the researcher, and this other person may make substantive adjustments to the conclusions of the study based on experience and knowledge of the domain of application.

These differences between the validity formulation of Cronbach and that of Cook and Campbell reflect deeper differences about how things happen in the world and, in fact, how the world is constituted.

The Regularity Theory of Causation

The orthodox theory of causation in the social sciences is called the regularity theory, as we have seen. The recurrent example used in illustrating this theory is that billiard ball A rolls across the table and strikes billiard ball B. At this point billiard ball A ceases to roll and billiard ball B starts to roll across the table. According to the orthodox view, when we witness this scene, we can never observe any causal connections between the two events. All we can see is the event of A's striking B followed by the event of B's moving. We observe only one event followed by another event and nothing more.

However, by observing repetitions of similar events we can infer eventually that these types of events are related causally to each other. The regularity and repetition of the events make causal inference possible. This is the paradigmatic example of the regularity theory, the deeply skeptical view of causal knowledge attributed to Hume, who believed that we never can observe causation directly: It can be inferred only by observing regular successions of events.

Although there are many versions of the regularity theory, three main principles are common to such theories (Searle, 1983). The first principle is that the causal nexus is not itself observable. We can observe regular sequences of events where one type of event is followed by another type of event, and we may infer from these sequences that the regularity is causal. We observe only temporal sequence, contiguity, and regularity.

Second, when a pair of events is identified as cause and effect, that pair must be a particular instance of some universal regularity. The universal regularity is usually referred to as a causal law. We may not know which particular law is entailed by the causal statement, but we know that such a law exists. We can discover a particular causal relationship without knowing the form of the universal law, but the law is there.

The third principle is that causal regularities are distinct from logical regularities. The aspects under which one event causes another must be logically independent aspects. For example, we would not say that something's being a triangle caused it to be three sided. Causal events must be logically independent of one another, according to the regularity theory.

In conscientiously explicating their conception of causation, Cook and Campbell embrace their own sophisticated version of the regularity theory.1 They believe that the causal nexus is not directly observable (1979, p. 10). They interpret particular pairs of cause-and-effect events as instances of universal regularities or laws. In their eight concluding statements about causality, they say, "In these, the term molar refers to causal laws stated in terms of large and often complex objects" (p. 32). Cook and Campbell see treatment outcome relationships revealed by experiments as instances of universal regularities or laws, More to the point, "We have a great deal of sympathy with the position that all aspects of research design test propositions of a general and universal nature" (p. 87). Causal laws are mentioned repeatedly in their summary discussion.

In identifying Cook and Campbell's (1979) view with the regularity theory of causation, I have oversimplified somewhat. In their analysis of causation, Cook and Campbell survey dozens of theories of causation, taking bits and pieces from several theorists, many of whose positions are not consistent with each other. Hence, their position is neither fully consistent nor fully representative of any one theory. believe our characterization of their position is fundamentally accurate although insufficiently detailed to capture Cook and Campbell's eclecticism. Campbell later wrote to me and embraced a view of causation similar to the realist view, which I have quoted earlier.

Ultimately, Cook and Campbell's conceptions of validity is based on the regularity theory. "Causality may well be such a logical hodgepodge of nonentailing but useful clues in the diagnosis of dependably manipulable correlations on the basis of fragmentary and momentary perceptual evidence" (1979, p. 30, emphasis added). Their theory of causation is reflected in the four major validity questions that they believe any researcher must face. First, is there a relationship between the two variables? In other words, does a regularity exist? "Covariation is a necessary condition for inferring cause, and practicing scientists begin by asking of their data: 'Are the presumed independent and dependent variables related?" (p. 37).

Second, given the covariation, is there a causal relationship between the two, that is, is the regularity causal? "Is there a causal relationship from variable A to variable B . . . ?" (p. 38). Third, if it is causal, what are the "higher-order constructs that the research operations represent? "Researchers would like to be able to give their presumed cause and effect operations names" (p. 38). Fourth, how generalizable across persons, settings, and time is the causal regularity? In other words, given the relationship, where else can we find it repeated? Universal regularities will be repeated, if we can only find and describe where.

In Cook and Campbell's view, the researcher is to discover the underlying causal regularities and their range of application so that the treatment can be reproduced. Presumably, once these causal relationship are known, the treatments can be reproduced or replicated at will, or at least with a reasonable degree of probability.

Cronbach's Version of the Regularity Theory

Cronbach sees causation as more complex and less certain, and hence less useful, than do Cook and Campbell. However, Cronbach's conception of causation is still based on the regularity theory. According to Cronbach, there are so many interactions of treatments with units and observations, and so little is known about how events occur, that speaking in causal terms is not very useful. Social events are too complex to yield simple formulations.

Following Mackie (1974), Cronbach formulates a causal law this way: "In S, all (ABC or DEF or JKL) are followed by P" (Cronbach, 1982, p. 139), where the letters refer to kinds of events or situations or possibly to the absence of some objects or actions. Now ABC is sufficient for the P to occur but not necessary because P may be preceded by DEF or JKL just as well. In other words, P may occur without ABC.

On the other hand, ABC is sufficient for P to occur if all elements—A, B, C— occur together, but not if only AB or AC or BC occurs alone. Yet the situation is even more complex than this. Mackie's original formulation of causal regularities is, "All F (A . . . B . . . or D . . . H . . . or . . . .) are " (Mackie, 1974, p. 66), where the ellipses indicate missing events or conditions that affect the outcome , but which are not represented in the stated law and about which we know little. Such "elliptical" or "gappy" propositions represent the true state of our knowledge of social causation better than the statements of simple regularity, according to Mackie and Cronbach. Cronbach cites the occurrence of numerous strong interaction effects in educational research as evidence of these gappy propositions.

Now here is the problem Cronbach's formulation of causation poses for the researcher/evaluator. If event A is the treatment one implements in an educational program, the complexity of the causal relationships becomes apparent. The treatment A is neither necessary nor sufficient for the effect to occur. The treatment is only part of a larger package of events that may be followed by P. Furthermore, we are ignorant of what many of these events are, as represented by the ellipses. Hence, specifying a treatment in an experimental design may actually be misleading because it may lead one to believe that treatment A is either necessary or sufficient for P the outcome, to occur when it is not. We may draw erroneous conclusions about A, the treatment. In other words, an experiment cannot provide a definitive test for the effectiveness of a program. Cronbach analyzes major field experiments to show that the researchers had to use knowledge gained outside the experiments to draw their conclusions.

All is not lost, however. The gaps can be partially filled in by experience outside the causal statement. In addition, over a long period of time one might attempt to localize the missing causes through an extensive program of research, thus filling in the gaps, so to speak. However, this long time scale is hardly adequate for applied research, such as evaluation. Thus Cronbach contends that the proper concern of generalization in evaluation is not establishing the internal causal relationships (utoS --- UTOS) but extrapolating to the external domain of application (utoS ---*UTOS) and that this latter extrapolation must be done in part by the person applying the results of the study. In a sense, the gaps in the causal relationships will have to be supplied by the interpreter. The task can be made easier for the interpreter by the researcher asking the right questions in the study.

Given this more complex notion of social causation, Cronbach insists that the critical inference from an evaluation study is from the actual data of the study to the domain of application. Only through the knowledge and experience of the interpreter can the gaps be filled. Thus, external validity becomes more important than the internal validity of the original study. In summary, Cronbach's and Cook and Campbell's differing conceptions of validity depend significantly on their differing degrees of confidence in being able to discern causes.

Agency and Intentional Causation

The regularity theory does not exhaust the possibilities of how to construe causation. One objection to the regularity theory is that it is contrary to common sense and psychological research. In spite of Hume's analysis, we do not see billiard ball A stopping and billiard ball B continuing. We actually see billiard ball A striking B, causing it to move. In recent years, Hume's analysis of causation has come under increasing attack.

Searle (1983) has advanced another theory of causation based on realist presumptions, which he calls intentional causation. Here is Searle's primary example:
"I now want to call attention to the fact that there are certain sorts of very ordinary causal explanations having to do with human mental states, experiences, and actions that do not sit very comfortably with the orthodox account of causation. For example, suppose I am thirsty and I take a drink of water. If someone asks me why I took a drink of water, I know the answer without any further observation: I was thirsty. Furthermore, in this sort of case it seems that I know the truth of the counterfactual without any further observations or any appeal to general laws. I know that if I hadn't been thirsty then and there I would not have taken that very drink of water. Now when I claim to know the truth of causal explanation and a causal counterfactual of this sort, is it because I know that there is a universal law correlating "events" of the first type, my drinking, under some description or other? And when I said that my being thirsty caused me to drink the water, was it part of what I meant that there is a universal law? Am I committed to the existence of a law in virtue of the very meaning of the words I utter? Part of my difficulty in giving affirmative answers to these questions is that I am much more confident of the truth of my original causal statement and the corresponding causal counterfactual than I am about the existence of any universal regularities that would cover the case." (pp. 117-118)

Searle's example departs significantly from the regularity theory of causation. First, he knows the answer to the causal question and the truth of the corresponding counterfactual without any observations other than the experience of the event. He might indeed be wrong, but the justification for his claim doesn't depend on further observations. The experience is all he needs.

Second, his causal claim does not commit him to the existence of any causal laws. There might indeed be such laws, but his singular causal claim does not commit him to their existence. His knowledge of the truth of the counterfactual claim, that if he hadn't taken a drink he would still be thirsty, is not derived from his knowledge of any such laws. Searle contends that because there is a causal relation in a particular instance does not logically entail that there be a universal correlation in similar instances.

Third, in Searle's example, the cause and effect are logically related to one another. That is, the notion of thirst, no matter how described, is inextricably connected to the notion of drinking, no matter how described. When we say, "My thirst caused me to drink," we are connecting logically related events. His desire to drink is logically related to his taking a drink, even though one caused the other. Searle's account of intentional causation contradicts all three principles of the regularity theory.

According to Searle's theory of intentional causation, in some cases we can actually experience causation directly. Suppose that in the classic billiard ball example, instead of being observers watching ball A, then ball B, that we actually take the cue stick in our hands with the firm intention of making ball B go into the corner pocket, by means of hitting ball A with the cue stick and making it strike ball B. What we actually experience is our intention of doing so and our execution of the task. If we are successful, we actually make it happen, and we directly experience the causation of making ball B go into the corner pocket. We do not experience our intention, watch ball A, then ball B, then infer from the events that we have a case of causation. We do not observe two separate events, and we do not need a covering law to explain their correlation. Rather, in Searle's analysis, the causal nexus is internal to the experience itself: Part of the experience of acting is an awareness that the experience itself is causing the action.

In Searle's account, by manipulating things, by making things happen, we gradually discover other "by-means-of" relations and extend our notion of causation to events outside our direct intentional action. For example, a child who breaks a vase with a rock learns not only that he or she can break a vase but also that hard objects can break glass. Eventually, one is able to observe causal relations even when one is not the actor but only an observer. Our primitive notion of causation is extended to other situations; we extend our causal knowledge by accumulating experience. Other theories of causation, such as Searle's, open up new possibilities for validating knowledge.

Implications for Validity

Earlier I analyzed two major formulations of validity and suggested that these two views are based on a particular theory of causation, although differently conceived. I introduced another theory of causation and suggested that validity considerations might be different yet again if we accepted a theory like intentional causation. More specifically, the validity concerns might include the validity of the inferences that practitioners themselves draw from their own experiences because these inferences are primary influences on practice, as in the case of the beginning teacher. Where does this leave us with our overall conception of validity?

We have three distinct situations: (a) the researcher draws inferences from an evaluation study and expects the practitioner to apply them, (b) the practitioner draws inferences from an evaluation study but modifies those inferences based on his or her particular domain of application; (C) the practitioner draws inferences based on his or her own experience and applies them in context. In each of these three situations the inferences might be wrong or, if you will, invalid. Ways of improving the validity of the inferences in the first situation are covered in the Cook and Campbell (1979) formulation and the traditional research literature. And Cronbach has argued that the second situation is more important and the validity considerations are significantly altered thereby. I think that the third situation is more important than the first two as far as the conduct and improvement of professional practices are concerned and that the validity concerns for practitioner inferences have been very much ignored.

The differences between causal inferences in formal research studies and practitioner inferences may be deep ones. Researchers usually express their findings as propositions, and it is the validity of propositions we test. By contrast, much of the knowledge of practitioners is tacit rather than propositional in form, elicited only when practitioners face a particular problem. In fact, practitioners often cannot state what they know in propositional form. Nonetheless, it is the validity of their causal knowledge that is critical for professional practices like teaching.

Accepting other theories of causation, like the intentional theory, does not mean that experimental design is useless. Rigorous designs may be useful for many purposes, including validating effects far removed from the practitioners' perspective, such as long-term effects of actions, or investigating controversial issues in which the practitioner’s knowledge might be biased. But accepting such a theory of causation means that correct causal inferences can be arrived at in other ways as well. There are means of validation other than experimental designs.

On the other hand, few of us would be willing to accept practitioners' causal inferences at face value. We need ways of checking and validating practitioners' inferences. The choice is not between valid scientific knowledge on the one hand and invalid practitioner superstition on the other, as the issue is often posed. The real problem is to arrive at valid causal inferences, and this can be done in a number of ways, most of which we have yet to invent.

Unfortunately, we don't know much about practitioners' inferences or how they arrive at them. We have few procedures or resources for helping teachers and other practitioners improve their critical inferences. Although the validity considerations in the experimental modes of reasoning have been well explicated, the inferences critical to practice lie hidden, subterranean. We must invent new ways of helping practitioners improve their causal inferences if we want to improve practice. (There is no apparent theoretical problem with discerning how practitioners reason causally. See, for example, Ennis (1973) on causal reasoning and Scriven (1973) on how historians draw causal inferences. Also Weir (1982) has applied the powers theory of causation to naturalistic evaluation.)

Thus, an expanded and pluralistic conception of validity would seem useful. It is neither correct nor useful to think of only one narrow pathway for reaching valid causal inferences. There are many pathways, some more efficacious than others, depending on one's circumstances.

References

Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54, 297-312.

Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation. Boston: Houghton-Mifflin.

Cronbach, L. J. (1980). Validity on parole: How can we go straight? New Directions for Testing and Measurement, 5, 99-108.

Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass.

Ennis, R. H. (1973). On causality. Educational Researcher, 2(6), 4-11.

15. Glass, G.V. (1982). Experimental validity. In Mitzel, H. (Ed.), Encyclopedia of Educational Research (5th edition). N.Y.: Free Press.

House, E. R., Mathison, S. and McTaggert, R. Validity and Teacher inference. Educational Researcher, Vol. 18, No. 7, pp. 11-15, 26

Mackie, J. L. (1974). The cement of the universe: A study of causation. London: Oxford University Press.

Scriven, M. (1973). Causes, connections and conditions in history. In H. S. Broody, R. H. Ennis, & L. I. Krimerman (Eds.), Philosophy of educational research (pp. 439-458). New York: Wiley.

Searle, J. R. (1983). Intentionality. Cambridge, England: Cambridge University Press.

Weir, E. E. (1982). Causal understanding in naturalistic evaluation of educational programs. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign.