The Oral History of Evaluation: The Professional Development of Ernest House

2011

 

The Oral History of Evaluation:

The Professional Development of Ernest House

 

Robin Lin Miller, Jean King, Melvin Mark,

Valerie Caracelli, and The Oral History Project Team

 

Since 2003, the Oral History Project Team has conducted interviews with individuals who have

made particularly noteworthy contributions to the theory and practice of evaluation. In 2011, Mel

Mark, Robin Miller, and Miles McNall sat with Ernest (Ernie) House in Anaheim, CA, at the site

of the American Evaluation Association (AEA) annual conference. The interview was taped and

subsequently transcribed verbatim into a 20-page document. Working with House, we edited the

transcript for clarity and length. House reviewed and approved the article prior to its submission

to American Journal of Evaluation as well as the final version following revisions made in response

to editorial feedback.

 

Ernest R. House is Professor Emeritus at the University of Colorado, Boulder, where he was

Professor of Education from 1985 to 2001. He was on the faculty of the University of Illinois,

Urbana, where he was affiliated with the Center for Instructional Research and Curriculum Evaluation

from 1969 to 1985. House has been a visiting scholar at numerous universities nationally

and internationally, and at the Center for Advanced Study in the Behavioral Sciences.

 

House is probably best known for his writing and practice regarding the inclusion of values in

evaluation. In an article published in 1976 in the first edition of Evaluation Studies Review Annual,

House drew on a major work in philosophy, John Rawls’s A Theory of Justice. In that article and

in other publications, House advocated for evaluators to incorporate social justice concerns in

their work, and particularly to consider a program’s consequences from the vantage of the least

well-off. More recently, with colleague Ken Howe, House has written a book on deliberative democratic

evaluation. In this work, House and Howe argue against traditional stances on the so-called

fact-value dichotomy. Instead, they argue that through careful processes involving relevant stakeholder

groups, evaluators can achieve relatively unbiased value judgments. Consider one indication

of the importance of these two approaches to values in evaluation: House is the only person to

have been listed in two places on any single version of Christie and Alkin’s (2008) evaluation

‘‘theory tree’’ (both on the valuing branch). House is the author of many other books, articles,

chapters, and reports. In addition to his work as a practicing evaluator, he has been called upon

to conduct meta-analyses of several major evaluations. Among House’s awards is AEA’s Lazarsfeld

Award for evaluation theory.

 

In this contribution to the oral history project, House describes his journey from English major

to evaluator and evaluation theorist. He talks about his first evaluation project, as well as the major

evaluation figures he worked with early in his career. House notes an important lesson from his

early experience as an evaluator, that is, ‘‘just how political evaluation was.’’ He explains that

this realization helped lead to his interest in values. After talking about a wide range of topics,

including meta-evaluation, randomized trials, the qualitative–quantitative debate, and the risk

of conflicts of interest in evaluation, House comes back to his early experiences and their role

in his focus on social justice.

 

The interview includes references to numerous individuals with whom House has worked over

the years. In the case of people outside the evaluation community, such as program and agency

staff involved in an evaluation, a brief explanation of their roles is given. Many of the individuals

to whom House refers are luminaries in the history of evaluation. These include Egon Guba,

Michael Scriven, Bob Stake, and Dan Stufflebeam, with whom House interacted early in his career

at the University of Illinois, as well as Don Campbell and Lee Cronbach, who made important contributions to social science methods as well as evaluation theory and practice. Their influential

work is described variously in Alkin (2004, 2013), Shadish, Cook, and Leviton (1991), other

entries in this oral history project, evaluation textbooks, and in their own many publications.

House also refers to Gene Glass, a research methods and statistics expert, perhaps best known for

early contributions to the practice and methods of meta-analysis; Lloyd Humphries, a noted individual

difference psychologist and measurement specialist; Barry MacDonald, a British evaluator

who emphasized democratic participation in evaluation; Roy Bhaskar, a British philosopher who

espoused a particular approach to realist philosophy; House collaborators Ken Howe, Steve

Lapan, Decker Walker, and Les Maclean; and current evaluation scholars and practitioners Leslie

Cooksy, Valerie Caracelli, and Gary Henry.

 

Interview With Ernie House

 

Mel: You were an English major at Washington University in St. Louis. How did you get from that as

your undergraduate interest to evaluation?

 

Ernie: I started in Engineering, but found it not that interesting. And then I decided to be an English

scholar. When I started going through all these books in the Washington University library in St.

Louis, I found these books that hadn’t been read for 50 years, 70 years. I thought, ‘‘Do I want to

do this?’’ I decided to go into medicine, to become a doctor. I have some science background. After

carrying this cat soaked in formaldehyde around in a bag for a semester, I decided, ‘‘I don’t want to

do this either.’’ I went back to my hometown and thought about what I would do. Within a month

of being drafted, I decided to teach English. This was between the [Korean and Vietnam] wars. You

could get any kind of a job, just about, particularly teaching, and you’d get out of the draft. I started

teaching English in high school. I taught for 4 years. People from the University of Illinois asked me to

teach a new curriculum, semantics and linguistics. I could do that. They liked what I did and asked me

to help write the materials and be the state consultant for the program. After that, the College of

Education there offered me a fellowship with James Gallagher, who was a special education expert

in the area of the gifted. When I finished my degree, they asked me to evaluate the Illinois Gifted Program.

I never intended to be an evaluator.

 

Mel: I wanted to ask you about that. How did you get chosen at this relatively early career stage to

evaluate that program?

 

Gallagher was the leading authority in the United States on the gifted. I had a fellowship with

him, and they asked him. Actually, evaluating this program was a political move on the part of David

Jackson, who was the main political actor in the College of Education at Urbana. He wanted the

program evaluated, and so did the state legislature, because gifted programs are not very popular.

Maybe they could show some effectiveness. Evaluation was all new territory. Jackson got the legislature

to provide the money. He asked Gallagher, ‘‘Who would you recommend to do the evaluation?’’

I was one of his students.

 

Mel: Well, what was the program like? It was a statewide gifted program?

 

The program had 20 demonstration centers all over the state demonstrating ideal teaching methods,

model teaching methods, new curricula, new math, new science, new English materials, most of them

out of the University of Illinois.

 

Mel: What did the evaluation look like? This was your first evaluation, right?

 

Well, I had a half million dollars, which was really a lot of money in 1967 terms and 4 years to

evaluate the program. I did many different kinds of studies. The program had five parts. The experimental

program funded studies. The demonstration program set up and funded the 20 centers. The

reimbursement program provided funds that districts could apply for to adopt these programs. The idea

was that you would do experiments and have proven methods for teaching and dealing with gifted

kids, and then you would have these things demonstrated someplace like Evanston High School.

Teachers would see these practices and put them into practice in Elgin or Alton, Illinois, where I’m

from, or someplace like that. It was a research, development, diffusion model of educational change.

We selected a random sample of school districts, and I tried a survey the first year because I had to

provide something for the state legislature right away. And I found out after I did that, that people lied

like hell on what they were doing when you sent them this form. So I selected a stratified random sample

of school districts in IL that had this program. I sent in trained teams of people to these districts.

The teams spent a couple of days there collecting materials and interviewing people intensively about

what they were actually doing, enough that we could make an assessment of what they were doing.

Sometimes our team would show up and the principal would say, ‘‘Oh, hell, we’re not doing anything,

really. We just wanted the money from the state to do this.’’ That’s the kind of stuff you’d find. We had

a set of instruments, 40 or 50 different kinds of measures and interviews. We developed a special

instrument to measure classroom atmosphere, a classroom climate–type instrument. One of the guys

working for me, Joe Steele, had developed this for his dissertation. We had a lot of different measures.

We didn’t use test scores as the outcome measure. We were looking at many different grade levels.

 

Mel: Well, what did you draw on in trying to put together this evaluation design at that point?

 

I read the whole literature in evaluation, which I put into a small cardboard box. I read the whole

thing in 30 days. I mean, that’s all there was, really. I talked to my statistics instructor, who happened

to be Gene Glass. ‘‘Look, I said, I don’t know very much about evaluation.’’ He said, ‘‘Nobody does.’’

He said Stake knows a lot about it and he said put Scriven and Stake on an advisory panel. Scriven had

moved to California and didn’t want to do it. I had Guba, Stufflebeam, Stake, and a couple of other

people. They provided me with advice. I used Bob’s countenance paper as the model (Stake, 1967).

That’s how I laid out the program. I set up a data collection team headed by Steve Lapan. By that time,

Jim Gallagher was the U.S. Deputy Commissioner of Education. I took the plan to him. He looked at it

and said, ‘‘This is a great plan. Take you about 15 years to do this and probably as much money as the

Vietnam War. You’re going to have to cut it down.’’ So I cut it way back.

 

Mel: What, if any, were the kinds of lessons from that experience that stuck with you?

 

Well, for one thing, how political evaluation is. And people taking money. At the end of the

evaluation, we had our results, and I had battles with the demonstration centers because by tracking

teachers going to the centers and going back into their district, we found that teachers would go to

the centers and see these model materials. They would say, ‘‘These materials are fantastic.’’ Then

they’d go back home and say, ‘‘My principal will never let me use this.’’ You know? ‘‘My principal

wouldn’t agree to this.’’ And, so you’d have tremendous enthusiasm at the demonstration centers

and everybody assumed that this would translate into practice back when they went back home.

But it didn’t. That was a big fight with the demonstration directors, who by that time had become

a powerful force within the program, the 20 of them. And they fought. They even tried to do their

own study to counter ours. In the end, the demonstration centers did some good. What we found is

that you have to go to work in the person’s district, in the teachers’ districts themselves, to get things

done. In order to implement, you had to do it there. The other thing was that when we got the results

after 4 years, I said to a guy who was a state representative, ‘‘I want this to be used.’’ He said, ‘‘Well,

here’s what you do. There’s this guy. Here’s the guy’s name,’’ I think it was Gene Hoffman. ‘‘He’s

the head of the School Problems Commission. Convince him that you’ve got something and take

it through that way.’’ So I asked the head guy to go out to dinner in Springfield. We drank about

three or four martinis apiece. I told him about the evaluation. At the end, he said, ‘‘Oh, hell, I believe

you.’’ I gave a presentation of our findings to the legislative committees. I don’t know if they

knew what they were looking at, but the program got a 20% increase in their budget because we did

find that the program had a lot of effective parts. The program was restructured based on the work

that we had done. I thought the evaluation was pretty satisfactory. But there were many political

interactions.

 

Mel: You mentioned an impressive set of people who were involved in this and who you studied with.

Gene Glass, Egon Guba, and Scriven. I think you worked with Stufflebeam and Bob Stake. What was

it like, as this group of people we now look at as evaluation luminaries and you were this young buck

trying to figure out how to do this project and learn this field?

 

I was 30 years old. And, they were good. Now Gene had gone off to Colorado by that time, so

he didn’t actually serve on the committee. He gave some advice about the program. Bob and the

others were really good. I just said, ‘‘I don’t know what I’m doing. Here’s what I plan to do. What

do you think I should do?’’ It was really—actually it was one of the best advisory groups I’ve ever

served on or been served by.We’d meet once every 3 or 4 months. I had money to bring them in and

pay them, and they were great. Guba and Stufflebeam, Stake, all of them. Dan was still developing

his CIPP (context, input, process, and product) model. Egon was changing his view of things. He had

not yet gone qualitative. They were just really a great group to work with. Intellectually, a very

exciting time. Really, really great time.

 

Mel: And you knew it?

 

Oh, I knew it at the time. I knew it was very exciting. This new field and new ideas, a lot of people to

talk to I didn’t know. These were the leading guys in the field, I thought, at the time.

 

Mel: One of the things you’re probably best known for is addressing the issue of values in evaluation.

What do you think it is that drew you to that as an issue to be addressed?

 

Thinking about how political evaluation was and thinking, ‘‘Is there anything beyond the politics?

Is there something beyond this other than just who’s got the most power and influence?’’ I started

thinking about value judgments and, of course, Scriven later on had written something about this

(Scriven, 1980). I started reading material on this, and I eventually ended up conceiving this idea

in terms of social justice. Social justice is an important thing to do. John Rawls’ book, A Theory

of Justice came out in the early ’70s. I had the traditional view that most people have when they start

out in social sciences: values are subjective. I think Scriven more than anybody convinced me both

in reading and talking to him personally that value claims can be objective, if we understand what we

mean by that. That is, they can be relatively unbiased. That doesn’t mean some other things that people

often take the word objective to mean. You can have well-founded value claims. I got interested

in the idea of values.

 

After finishing the gifted programs, I needed something. I thought, ‘‘Is there anything beyond

the politics of this? Is this all politics?’’ I saw this book, A Theory of Justice reviewed in New

York Review of Books, the leading book of our time in terms of social justice. So I bought a copy,

took it to England with me on leave and read the book. From that came my article on justice in

evaluation.

 

Mel: Did it feel at all dissonant? I mean, Rawls’ book was thick in more ways than one and you were

trying to apply it to this very practice-based field.

Well, I think that was the difficult part. I read the book thoroughly—read it a couple times, and so I

had actually a good conception of it. But then how do you apply it? The application was the hard part.

There were different ways you could think about this, but I didn’t want to take it as the ‘‘original position.’’

The idea would be taking Rawls’ two principles of justice and saying ‘‘we should adhere to

these in evaluation,’’ one of which deals with inequality. Inequality, if it’s allowed, should be to the

benefit of those least advantaged in society. I saw that as being an entre´e into programs for the poor,

the impoverished. The other principle is a rights issue. You shouldn’t violate student rights.

 

Mel: In a commentary in a handbook, you wrote about the work of several evaluation theorists, and I

want to quote you. ‘‘For their part, the theorists know from experience that the half-formed ideas they

toy with today may form the basis for proposals in evaluation studies tomorrow, or that the ideas may

be adopted by governments and organizations as a basis for policy. Fuzzy ideas of yesterday have a

way of becoming orthodoxy tomorrow.” When I read that I wondered how much is that an autobiographical

statement because there was some skepticism originally, I think, in terms of the response of

many people to your ideas about social justice and Rawls.

 

Oh, yeah, tremendous disagreement, I mean, just shock. First shock, ‘‘What’s justice got to do with

evaluation?’’ They just didn’t conceive it in the same—those two things live in two different worlds.

You’ve got this science, which is value neutral or value-free, and then you’ve got social justice, which

is connected to values. That’s how a lot of people still conceive the role. Economics is supposed to be

value-free. How evaluation can be value-free is hard to understand. So people were shocked and had

tremendous reactions. People at the university, some of the leading professors of psychology and education,

decided that they didn’t know if I was right. They would have a seminar with me leading the

seminar. There was Lloyd Humphreys. We got together, about 10 or 15 people, and started reading the

Rawls’ book, which they found very tough sledding. And we went through it. They were so concerned

about it that they actually wanted to have this seminar with no students in there. They didn’t want to

make fools of themselves in front of the students. So, you know, we just had this 30-year-old guy, I

guess I was about 35 then, doing this seminar. That was interesting. Lloyd never did agree with me. He

came to terms with it, but he never did buy it. Don Campbell was one of the first guys who asked for

some copies. Campbell was always pretty open-minded. He never bought into it fully because he still

held to some kind of value-free position; this in spite of the fact that the same arguments that he used

against foundationalism [of knowledge] apply to foundationalism in values. Campbell espoused anti-foundationalism with respect to knowledge claims, which involves arguing that all knowledge is fallible and contingent.

That is, if he used the same reasoning on values, he would have to go with a different conception of values. But he was

always open-minded. He was great to talk to.

 

And even the government people started picking it up in Washington and mentioning social justice.

I remember one guy at the Department of Education in charge of evaluations. He said, ‘‘Damn, I wish

I’d thought of that. If I had read that book first, I could have written that paper before you did.’’ And

my response was, ‘‘Well, take it. I don’t have any ownership of it. You can take it and do with it

whatever you want to.’’

 

Mel: As you know, a year ago at the AEA conference, some of your work was highlighted, including the

truth, beauty, and justice ideas.

 

It’s nice to see it still might be relevant, a book you did 30 years ago. Yeah, that’s a nice feeling

to have. I don’t want to get too carried away because I always think that our influence is very limited.

In my view, our influence is a lot more limited than what we realize. You know, you’re a big name

guy, and you go give talks and a lot of people show up, or maybe they don’t. But you give these talks,

man, just going on forever. Well, a few years after you die, hey, that’s pretty much gone. I don’t think a

lot of it hangs on. Campbell’s perhaps the exception to that. Take even somebody like [Lee] Cronbach.

You don’t hear of his name so much anymore, and what an enormous figure he was in the field.

 

Mel: And he wrote stuff worth rereading today.

 

Oh, yeah, Cronbach had great stuff. No, it’s nothing about the quality. It’s something—there’s

another dynamic in operation, where the work gets replaced with other ideas, which probably aren’t

as good as the work replaced. What has happened with Campbell is that you’ve got a set of proprietors

who have taken these ideas and reformulated them. And have re-done them, considerably, I might add.

By the third edition, those ideas have been redone. That’s a difference between Campbell and Cronbach.

Cronbach’s ideas were terrific. Other than maybe in testing, he didn’t have a group of people

established who have carried on the evaluation ideas. Maybe the ideas will return. Usually, when they

do, they are renamed.

 

Mel: You more recently, working with Ken Howe, have been advocating notions of democratic deliberative

evaluation.

 

The idea would be, let’s try to make evaluation more democratic. If you’re going to bring people’s

values into evaluation, how could you bring a lot of values together? Bring the values together with the

stakeholders and make sense out of them somehow. Our idea would be to base values, the evaluation,

and the value judgments on people involved in the program, the beneficiaries of the program, and all

the major stakeholders in the program. Bring those together, and you’ve got inclusion, discussion, and

deliberation, three broad principles. You try to include the people who are important and not exclude

people unless you have good reasons for excluding them. You try to have serious discussions and listen

to them and their point of view. That’s an error evaluators often make. They listen to the sponsors,

but not to people in the program and in the field. Then bring it all together in a process of deliberation.

We haven’t defined deliberation very well, I think. Deliberation is a tough concept. You get people

together in a room, you sit around, you talk about this theory, that theory. You argue back and forth.

And you have an evaluation based on the values of the community.

 

Mel: What’s your sense of the state of the art in practice in carrying this out?

 

I think it’s hard to do. I did the project in Denver, which worked all right when I was a federal

monitor for the bilingual program for about 5 or 6 years. Ken and I talked about this. I said, ‘‘You

know people. You have to give people a set of procedures.’’ He was arguing, being a philosopher,

‘‘No, let’s just go with the principles because there are many ways to do it, and people have to fill

in.’’ I think it’s difficult to do, to tell people to be inclusive without telling them how to do it. I can

say, ‘‘Be deliberative.’’ But how do you do it? You need some ideas about how to do it. I’d say that

where the ideas get caught right now is in the implementation. I’ve always been better at conceiving

abstract stuff than implementing. I can do the evaluation and can write about how I did it, but it

would be difficult to write a protocol for, ‘‘You do this. You do that.’’ I think that’s where it sits;

a conceptual scheme right now. Some people can do it. It doesn’t take that much to do, and it may

not be far from what a lot of people do already.

 

Mel: You think there’s much risk of things exploding when you put these diverse value bases of people

on the table?

 

Yeah, it’s a possibility.

 

Mel: How do you avoid that?

 

You have to be careful. I think you try not to get some groups together. In the Denver project, we

always had a lot of militant people who wanted Spanish taught in all the schools. I didn’t try to bring

them in directly. I never had a town hall meeting where everybody got together. In the past, those

ended in shouting matches, if not worse. Rather, I talked to the militants. I went to their headquarters,

talked to them, recorded their views, and listened. I didn’t try to have them confront the school district

people. I did get the lawyers together and present them with data twice a year. Part of the evaluation is

to accept that you have to structure the interaction just as you have to structure the data collection.

 

Mel: In the late ’70s, you and a group of people published a critique, a kind of meta-evaluation of

Follow Through, a program designed for kids after completing Head Start. You criticized the evaluation

in terms of a couple of things, one being that the measures that they selected fit better with some

program models than others; they were comparing across a few different variants or types of Follow

Through programs. And I think part of the point was that any claims about relative success were kind

of illusory given that differential fit. Now, I think you also used the concept of justice in making that

criticism. Can you talk about that project, including how it came to be that you and others were doing

this critique?

 

The Ford Foundation asked me to consider doing it. When the Follow Through evaluation was

coming out, a lot of people—this had been going on for a while—a lot of the people with the programs

were unhappy with the way the evaluation was going and how the evaluation had developed. The evaluation

started with a very broad array of measures. The original idea was to have [a collection of]

measures [that together would be appropriate] for all 12 types of programs. In other words, you’d have

measures across all types of programs. You’ve got rather different kinds of programs. But the evaluation

was cut back to only a few evaluation measures. Some program people complained to Ford.

A woman at Ford asked me if I would be interested in taking a look at the evaluation. I said, ‘‘Well,

let me check it out.’’ I went to a Follow Through meeting and talked to a bunch of people. I thought,

‘‘Yeah, maybe they’ve got a point here.’’ What had happened was that the Department of Education

had started with this vast array of measures and at the first data collection the evaluators had collected

two tons of data. They couldn’t possibly manage it. Nobody could manage it. So they reduced it down

to a few outcome measures. You’ve got all these measures for everybody, now they’ve reduced it

down to a few, such as the metropolitan reading test. That reduction of outcome measures tended to

favor certain programs. When we looked at the teaching materials of the programs, we found that a

particular program would perform very well in math. When you’d look at the internal subtests you’d

see they were about the same as everybody else. Except, oh, this one subtest jumps up like this, and

that’s what brings their average up. That’s why they scored high in math. Then I track this back to why

they do so well on that particular subtest. I look at their teaching materials and I look at the subtests,

and I find what they’ve done is teach materials very close to the subtest. For example, instead of

[formatting] multiplication problems up and down, the test items multiply [horizontally]. Some of the

programs knew that’s what was on the tests. This is an open test and they structured their materials that

way, a very close match to that test. It wasn’t exactly the same items, but very close. The whole format

was very close. That was suspicious because that’s where all their high scores are coming. That’s how

they get this high score in math. By doing that kind of analysis, we developed the idea that the outcomes

were a bit arbitrary because of the closeness of the materials to what was on the test. Perhaps

it was accidental. Nonetheless, it gave a great advantage to certain programs. Decker Walker was on

the panel, along with Gene Glass and Les McLean. Decker had done some research looking at the

closeness of fit in other tests. That’s why I had him on there. His research indicated that how closely

the curriculum fits the test makes a big difference. That’s been replicated over and over again since

then. The other big factor was that you had programs with six different sites around the country. The

outcome variation among those sites was terrific, huge variation. So the variation across sites within a

program was about the same as between-program variation, if you follow me.

 

How can you say that Program A is better than another program if there is so much internal variation

within a program? There was no consistency, internally, within those programs. For those reasons, our

critique was entitled, ‘‘No Simple Answer.’’ Glass also did some effect size comparisons. We ended up

with a critique challenging the Follow Through evaluation, which was the big evaluation of the day. It

cost a lot of money and was supposed to be definitive. The Department of Education was figuring they

were going to tell everybody in the country, ‘‘Use this, use this, and use this.’’ We thought that was

going too far with the results they had.

 

Mel: I think that your critique helped contribute to skepticism about the potential of these large-scale

federal evaluations as definitive guides to action.

 

Yeah. Yeah. I think it did. I think it wasn’t only our critique, but the fact they put so much money

into this evaluation over so many years, and they end up with an evaluation that is, one can argue,

rather arbitrary in the results and that has inconsistent results in many ways. Maybe these big definitive

evaluations just don’t pay off. There have been several other big evaluations. That was just one of

them. And that may have been the final one to tip over that had people thinking. ‘‘Well, let’s not

do these big, huge, massive things. We’ve got to wait for years and years to get the results from them,

and then find the results are rather arbitrary.’’ In Washington, they expected definitive results out of

this big chunk of money. And so the people in charge of it just thought something went wrong. Something

was wrong with the evaluation since it didn’t produce definitive results. Another way to look at it

is that the reality they’re trying to map is too complex. Reality is more complex than what they realize.

And you’re not going to get simple results and totally consistent results. Because if you put a program

in one place and you put the same program in another place, you end up with different results. You’ve

got different contexts in different places, and you end up with rather different results sometimes.

Mel: Any sense of irony that this critique had such widespread implications?

Well, I don’t know. I didn’t expect it would be quite that—the Washington people were very

unhappy with it and so were some of the sponsors who had won, what they called, the ‘‘horserace.’’

They thought they had won the horserace. So we say, ‘‘Well, the horse was doped’’ or whatever. We

didn’t say that, of course. But, it’s not that those programs weren’t good programs; it’s just that

the claims were too strong based on the evidence they had. Yeah, that was one of the times I

got—one of many times—I got blackballed in Washington for running counter to federal policy.

 

Mel: What’s your sense of how evaluations have done since then with these kind of issues of context

specificity and generalization?

 

 

Well, I think the evaluation community is far more sensitive to those issues now. I think that

we’ve tended to do other things, have other kinds of studies. I would say there are various things

that people have done. One is program theory by mapping out what the program is like. That limits

the possibilities if you do that, I think. Another is meta-analysis. If you put a program in place here,

here, here, here, and here, even though you get different results and under different circumstances,

I actually think that adds to the robustness of the overall findings you end up with. It doesn’t have

to be one massive study. Maybe it’s better to have a bunch of smaller scale studies in different

places and different contexts and then try to summarize with something like a meta-analysis. Even

if you’re going to do randomized studies. That makes more sense to me, and I think we’re more

sophisticated now. Sometimes the randomized stuff worries me in terms of claims made for it, such

as this huge focus during the Bush administration. I think people are over relying on randomized

studies. Indeed, my understanding of why Don Campbell invented internal and external validity was

to get away from Fisherian randomization. Because the students thought it solved all the problems,

and it doesn’t.

 

Mel: You’ve done actually a number of things that are critiques or meta-evaluations: The Follow

Through, Jesse Jackson’s PUSH-Excel, and the New York Promotional Gates program. Is there something

that draws you to doing that kind of work, or something that draws people who are looking for

that kind of work to you?

 

I think I like the political stuff. I’ve always had a penchant for the political: political theory and

political philosophy. Maybe if I did a degree again, I might do it in political science. It has a draw

to it, and every once in a while I think from my own personal background I have an urge to engage

in conflict. I also have this sense of trying to seek out social justice. So I’m willing to take on some

critiques. Like the New York people called me for the Promotional Gates program, which was the big

New York City program. The Mayor’s office called and said, ‘‘We’ve got this program set up and

we’re going to do an evaluation, but we need somebody to monitor the evaluation. We have to have

somebody who’s pretty tough. We’ve talked around, and people say you’re the toughest guy we could

find.’’ At this point, I was at the University of Illinois. That was flattery, right, from my point of view.

And so, of course, I signed on. I couldn’t help myself, could I? I fundamentally got myself in a lot of

trouble. We had a lot of newspaper coverage over the whole thing. I think it was the only time I ever

agreed to make my reports to the Mayor’s office and the Chancellor’s office confidential. They wanted

it confidential. I said, ‘‘Okay.’’ Well, we did it. But their bureaucracy leaked the reports. You put stuff

in one end and it runs out the bureaucracy. They had a leaking bureaucracy. Some of what we were

writing, saying, ‘‘You know, all you’ve got here are no effects in your summer programs, where you

think you’ve got these big test score gains.’’ They had failed to adjust for regression to the mean. ‘‘And

so you really don’t have these results that you’ve been touting in the newspaper.’’ The Mayor was

running an election campaign, and he was on television saying, ‘‘I raised the test scores in New York

City.’’ Well, the results he was talking about didn’t exist in reality when you analyzed the data the

proper way. They didn’t do it deliberately; they just didn’t know any better. Bob Linn served on that

panel with me. We had to go tell them they didn’t have any effects from their summer training program.

At the same time, they finally decided, ‘‘Okay, just help us do it the right way.’’ I thought they

turned out to be very reasonable about it. We had lots of ups and downs, and they tried intimidation,

you know, the usual stuff.

 

Mel: Despite doing several of these kinds of critiques, monitoring, and meta-evaluations, I don’t

think that you’re widely associated with meta-evaluation as a topic. Any thoughts you want to share

about the practice of meta-evaluation?

 

Well, I’ve taken on the meta-evaluations. I have written some, and Leslie Cooksy and Valerie

Caracelli interviewed me and used some of those critiques in their work about meta-evaluation.

I’ve done some writing, but I never attempted to do what they’re doing, which is to pull some of

that ideas together about how we approach the issue of meta-evaluation. I’ve tended to philosophize,

and although that’s interesting, I find the meta-evaluations interesting too. The other thing about it

that appeals to me is because it acts as a conscience for the field. I think we need to have people in an

important role, and possibly a critical role, evaluating people for their careers and their work.

There’s a lot riding on it. You really need checks within the field itself to say, ‘‘Look, we have

to police ourselves on this stuff. We can’t just let people do anything and everything.’’ The metaevaluations,

particularly if they’re big enough to be an example, have real appeal as one way of

saying, as with Follow Through, ‘‘Don’t do the studies this way.’’ The Follow Through critique

influenced a lot of people because it had a high profile. If it had been a small thing, it wouldn’t have

had quite so much influence. You need some meta-evaluation, some checks on what people are

doing. It serves as a conscience for the field or a way of exercising the conscience of the field.

We should check ourselves. We should evaluate ourselves. Scriven’s absolutely right on it.

Mel: I want to ask—thinking back to a past AEA conference when Chip Reichardt and Sharon Rallis

were Presidential Strand chairs. You were one of the people that they invited to talk on the qualitative/

quantitative debates. As I recall you ended up saying, ‘‘Let’s stop wasting all this human

capital on this debate. Let’s do the hard work of evaluation.’’ What’s your sense of where that discussion

has gone?

 

I think we have made a lot of progress. I think that conference helped. We still have a split, of

course, between the quantitative and qualitative people, but that’s all right. The randomization issue

got bent out of shape. I think it bent people out of shape somehow. So that was a little bit of a throw

back. Not that I’m against randomization but pushing it very hard was a bit of a throw back for a while.

Generally, I think we’re in really good shape in terms of quantitative/qualitative. I can understand why

quantitative people see themselves as playing an irreplaceable, important role. And they see themselves

as quantitative people, you know? Gary Henry showed me the latest analysis he had done

recently. He said, ‘‘I finally think I’ve done a real piece of work that I can show my granddaughter

and say, ‘This is really a serious piece of work that I’ve done.’’’ I think quantitative people identify

themselves with the rigor. Part of their identity is with that. It’s more than an issue about method. What

happened to the quantitative group was they felt their status was being diminished in some ways. It’s a

legitimate concern on their part. They wanted to reassert themselves as being important in the area.

Sometimes people with all the talk about the qualitative stuff tend to diminish the importance of the

quantitative. And I can see that would affect people who put a lot of their identity into the quantitative.

On the other hand, the qualitative people are sensitive about their methods as being too soft. If you say,

‘‘This evaluation isn’t worth anything,’’ they’re going to react. That’s why we get into these shouting

matches. I’ve been in a few back in the early days with Tom Cook, before Tom decided he liked the

qualitative as well as quantitative. Those were some of the early battles we had back in those days.

 

Mel: Is there a place for randomized trials from the perspective of social justice?

 

Oh, yeah, sure, absolutely, yeah. The trouble with randomization from my perspective is—well,

it’s hard to do sometimes, but it’s more about the metaphysics of what we’re dealing with than it is

about the methods themselves. Sometimes experimental design tends to treat the program like an X.

Like an X is unchanging. But we now realize when you put the X into a setting it’s not really X, it’s

really X sub-one, and you’ve got all these variations, and those variations in something like an educational

program can be very, very substantial. Put the same program with a different teacher, and

you’ve got pretty damn substantially different programming. That kind of program variation makes

the randomization more difficult, or at least what you can derive from it. It’s not that you can’t do it.

So I don’t agree with some of our realist colleagues. You and I are realists, of course, in our own

frameworks. But as you know, some British realists say, ‘‘Well, you shouldn’t do experiments.’’

Well, that’s not my view. Of course, you can do experiments. You just have to be careful of the

conclusions you draw from them. I mean, they take Bhaskar, who says that you cannot do social

experiments that are definitive in social work the same way you can do definitive experiments in

physical work (Bhaskar, 1978). The nature of reality is somewhat different. Social reality is much

more plastic. But that doesn’t mean you can’t do experiments. You just have to be more careful

about how you do them and interpret them. So there’s nothing wrong with randomization.

 

Mel: How did you get to the realist philosophers, by the way? I’m thinking of that Educational

Researcher paper (House, 1991).

 

I was down at the University of Sussex giving a talk back in the late ’70s probably or early ’80s.

There was a conference of social philosophers. I looked at the books they had there, and I saw these

Bhaskar books. I thought, ‘‘realist theory of science.’’ You know that I read philosophy all the time and

I used to do it for fun. I mean I read a lot of philosophy, and I read widely in lots of fields. I thought,

‘‘Well, I’ll take a read at this.’’ And I read that and I thought, ‘‘You know, that would address the quantitative/

qualitative thing.’’ If you have a particular conception of a reality, and here’s the conception

and you could approach it, you could do that with quantitative methods, and you could do it with

qualitative methods. It’s still the same reality. I saw that as a way of getting through the quantitative/

qualitative dispute, and that’s how I wrote that original realist article I did. Well, I didn’t do so

much realist evaluation as I did a realist research perspective.

 

Mel: More recently, you’ve been paying a good deal of attention to conflicts of interest, drawing on

medical evaluations but applying it to evaluation more generally. Can you talk a little bit about how

serious a threat you think this poses to our work?

 

I think it’s a very big threat. I think it’s a threat to the whole society. And I got into doing this

because I was looking at how politics influences experiments, even randomized experiments. I saw

all these studies, these medical studies, changing their findings, such as ‘‘This medication has these

effects, and nobody found this out.’’ I started looking into that issue, which some of the medical

people had looked into as well. I’m reading these medical articles and discovering there’s a much

deeper issue. This phenomenon is systemic. It doesn’t just happen from time to time, rather there

turns out to be a systemic bias built into the system itself. I became alarmed the more I looked at

it. It is quite widespread in medicine, unfortunately, and I found it’s widespread in other fields, too.

As you know, I’m an active investor and read a lot in economics and financial investing. One of

the big things that happened in the 2008 financial crisis was that there was conflict of interest in the

financial system from top to bottom. The economist Jeffrey Sachs calls it the collapse of civic virtue

by our financial and political elite. It’s a pretty big issue. And it’s important to us because if you

have evaluation being bought and sold for its results, that is, ‘‘We’re going to have you do the evaluation,

and here are the results that we want.’’ Well, what use is evaluation? Conflict of interest has

tremendous pernicious effects when you start doing evaluations of medical and education programs.

If you let the sponsors call the shots on the findings, it’s dangerous. It’s the biggest danger to the

field for the future, in my view.

 

Mel: I want to ask a couple of questions about writing. If I remember correctly, in some of your early

books, the chapters had appeared in earlier versions as journal articles or book chapters elsewhere.

Did you have a map as you were working on the individual projects that led to the book, or did you sort

of connect the dots after the fact and create a larger work?

 

I think the 1980 validity book was more like that (House, 1980a). I did the article on social justice,

and I did an article on truth, the argument logic one. These were responses to particular issues, like

the persuasion paper, the monograph I did at University of California, Los Angeles, one summer

(House, 1977, 1980b). That was a response to people who were arguing that all you had to have was

a method. I was saying, ‘‘A method’s not enough. You really have to make an argument. The method

fits into the argument, but you’re making an argument. You have to make an argument. Science

makes arguments. There’s a method within the argument, that’s part of the argument, but it’s the

total argument that counts, not just the method that counts. Although the method is important, of

course.’’ Some articles were to address social justice. Social justice addressed the question, ‘‘Is it

only politics out there? Have we got anything beyond politics that we can value?’’ So, there were

specific topics of interest. But then I took some of them for the validity book and put them all

together. Some places I needed to fill in the pattern, you know, the dots between the political and

social justice and methods with the political theory. I found ideas to fill in the things I wanted.

 

Mel: How did you start working on writing fiction?

 

I was an English literature major, going way back, and every English lit major wants to write a

novel. Probably psychologists do, too, they just don’t want to admit it. But I always wanted to write

something that’s creative—creative writing. I gave a talk at the Center for Advanced Study in the

Behavioral Sciences at Stanford where I was a fellow for a year. I wanted to explain to these other

fellows, who were very bright people, economists and historians, and so on, what evaluation was.

A few knew about it, but most didn’t. I picked out three concepts: causes, values, and politics. Now

I could handle causes with my realist framework, with how I thought our conception of causes had

changed. For values, I had how our conception of values had changed from value-free to value-embedded.

However, when I got to politics, the third concept, I didn’t have a ready analysis. And

you have to understand the politics. So I just told a story. I told a story of a project that seemed

to work pretty well. And I couldn’t think of an analytic framework that would handle that concept.

The speech went over very well, I thought, from my own point of view. The staff thought it was the

best talk of the year. Totally unbiased opinion, of course. I got to thinking, ‘‘Well, maybe you could

do that in a larger framework.’’ I was talking to this old colleague and good friend of mine, Barry

MacDonald, in England. He was retiring and complaining that the young people didn’t listen to him

Any more. He knew all about how to manage projects and deal with the politics, and all that. But he

couldn’t pass it on. So I thought one thing I could do is write a novel, and students can read that. I’ll

make it more fun. So the novel was written for students (House, 2007). I tried to include some of the

knowledge that we have and incorporate it in story form for them, which seems to be a natural form

for that kind of knowledge anyway. A political novel is a hell of a lot more interesting to read than

any political science text you’ve picked up recently. You would have that in there. And a lot of it is

detail. It has to be. There’s a reason why the narrative form fits politics so well, because it’s the little

ins-and-outs and backs and forth and nuances and all that detail that’s so important. I decided to

write the novel and put it online, but few people found it. Finally, I just put it in hardback.

I think part of the thing about writing something, if you can take a complex idea, a philosophic idea,

and make it transparent then you’ve done a very good job of writing. That’s one of the criteria I use.

Could I take this idea that may be confusing and very complicated and make it clear? You know, that’s

the challenge in trying to do it. If you do that, you’ll probably end up with something worthy.

 

Mel: In that quote I mentioned earlier there was a little earlier part to it and it was that ‘‘The role of

the theorist in the world of evaluation practice is ambivalent in some ways. The theorists are high

profile, lauded and sought after to deliver speeches and lend legitimacy to projects, proposals, and

meetings. At the same time practitioners lament that the ideas are far too theoretical, too impractical.

Practitioners have to do the project work tomorrow, not jawbone fruitlessly forever.’’ Again, a

little autobiographical?

 

Oh, yeah, sure, definitely. Been to many places, you know, and given a lot of advice that probably

wasn’t very useful to people, particularly in other countries. No, no, of course. I gave a talk in New

Zealand a couple of years ago in Maori country. You know the Maori have a huge influence there.

They were looking at culturally sensitive issues. And so I told the story of the Denver evaluation that

I did. I told the story rather than talk about democratic evaluation as a concept. I think it went over

much better—they could take the story and adjust it for whatever their own circumstances were.

Because I couldn’t conceivably know what their circumstances are. They could make whatever

adjustments they wanted to, the idea is there. That’s one way of dealing with the relevance issue,

but it’s always an issue. You get paid, and you become well known for writing books, and articles,

and more articles. You’ve got to put the ideas in a certain form to go into certain journals. But as

every practitioner will tell you, that doesn’t help the practitioners very much. I’ve always lamented

that. I mean, I always have. I always see that gap there. It’s also true—because I started as a teacher

for a few years, and I saw the big difference between what people are teaching teachers and what

the teachers actually have to deal with. There’s a huge difference. Actually, that problem exists

in all the professions to some degree.

 

Mel: Any ambivalence about sort of reaping the benefits of this highly lauded career—you know, the

flights to give talks at places, the requests, ‘‘Please be on our advisory board,’’ the honors—and the

focus on social justice? Does that ever twinge a little bit?

 

Yeah, yeah, yeah, sure, I notice the difference. You’re trying to do what you can. You go someplace

and people spend a lot of money to support these boards and all that, and sometimes you wonder,

‘‘Does this really make any difference for the people out there who are doing or suffering that

the programs are supposed to be for?’’ I mean, ‘‘Does that really help? Does this help them that

we’re doing all this?’’ I think anybody with some conscience would think about that. If you have

a sense of it, there are people out there suffering who need these programs. You have some sense

of that in the end. So yeah, there’s a twinge I can’t do more. Now I don’t know what else I would

do. I mean that’s the other thing. What would I do that . . . what would I do to help? So there’s still a

question of could I have done something else? I haven’t thought much about that. I think with evaluation

I’ve done about as well as I could for social betterment, given my position where it was and where my

thinking was at that time, certainly. But I would have killed a lot of people if I’d gone into medicine,

been a doctor, I’m sure. So, you know, at least I don’t think I’ve damaged anybody in that way.

 

Mel: Does Alton come back to you at times? Is that part of your concern for social justice?

 

Yeah, sure. I grew up in difficult circumstances. A lot of people are still back there. Those people

didn’t have a very good life and somehow I was lucky, fortunate, to get out of the circumstances we

were in. Some people did, but a lot of people didn’t. Yeah, that’s part of it, sure. I never forgot those

people, the kind of circumstances they were in. I know what it’s like to go to bed hungry. I’ve been

there a few times in my life. And, you know, my relatives didn’t get out of there since they stayed in

the same social class, same kind of circumstances. There’s first-hand experience that fuels the social

justice. It’s not simply an abstract concern. You know people who went through all this and who could

have used help from somebody at some time. And so evaluation, for me, is a way of trying to keep the

programs honest, but then you’ve got to keep the evaluations honest, too, to try to help these people.

 

Robin: A lot of these ideas were germinating during critical shifts in our country socially, politically,

culturally; how did that fit into the mix of the evolution of your ideas, reading about theories of justice,

thinking about where you came from? Can you situate some of your thinking in the context of those

times politically and culturally?

 

Well, context definitely has a strong influence on the work that I did and why I wanted to do that

work at that time. The social justice ideas I did during the time of the great expansion of social

programs in the ’70s, following the Great Society programs, which really set evaluation on its feet,

got it going. This was the expansion of programs and program evaluation. The idea Campbell developed

was this social experimentation idea (Campbell, 1998). But I was trying to develop the idea

that this should include social justice too as we looked at these programs. When you get to the

1980s, Reagan comes into power, and the whole tenor of the country changes. It has taken a different

direction since that time. So what are we involved with now? Conflict of interest is where we’ve

ended up. Serious, big, damaging conflict of interest from the top of the government to the bottom.

I worry about that now looking at it from the role of the evaluators, whether it’s evaluators doing

drug studies or doing education evaluations. The society has somehow become infected with conflict

of interest and other forms of corruption. Now, I can’t quite blame Reagan for all that, although

the trend started with Reagan. I don’t think anybody’s got a very good story on why the social backdrop

has changed. I find myself dealing with program issues that the context delivers. And social

justice is always there. I’ve discovered that from my own background and from the theory of John

Rawls. As of now, 30, 40 years later, you end up with conflict of interest and this systemic self-interest

eating up the society. The whole society has changed since that time.

 

Miles: Can evaluation play a role in saving us from ourselves?

 

I would say ‘‘yes,’’ but first you have to save evaluation. You have to save evaluation from conflict

of interest. That is, you can’t have evaluation for hire for results. You can’t have drug companies saying,

‘‘I’m giving your company the funds to do this study for me, but you know damn good and well if

you don’t get the results to come out the way we want, you’re not going to get another study from us.’’

We’ve got to save evaluation from that. We’ve got to save the field before we can do anything beyond

the field. That’ll be tough, tough.

 

Acknowledgments

We gratefully acknowledge the donation of transcription services by Gargani and Company and the editorial

assistance of Katherine Cloutier.

 

Authors’ Note

The Oral History Project Team consists of Robin Lin Miller (Michigan State University), Jean King (University

of Minnesota), Melvin Mark (The Pennsylvania State University), and Valerie Caracelli (U.S. Government

Accountability Office).

 

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication

of this article.

 

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

 

References

Alkin, M. C. (Ed.). (2004). Evaluation roots: Tracing theorists’ views and influences. Thousand Oaks, CA: Sage.

 

Alkin, M. C. (Ed.). (2013). Evaluation roots: A wider perspective on theorists’ views and influences. Thousand

Oaks, CA: Sage.

 

Bhaskar, R. (1978). A realist theory of science. London: Leeds.

 

Campbell, D. T. (1998). The experimenting society. In W. N. Dunn (Ed.), The experimenting society: Essays in

honor of Donald T. Campbell. Piscataway, NJ: Transaction Publisher.

 

Christie, C. A., & Alkin, M. C. (2008). Evaluation theory tree re-examined. Studies in Educational Evaluation,

34, 131–135.

 

House, E. R. (1976). Justice in evaluation. In G. V Glass (Ed.), Evaluation studies review annual (Vol. 1),

75–100. Beverly Hills, CA: Sage.

 

House, E. R. (1977). The logic of the evaluative argument. Los Angeles: University of California Center for the

Study of Evaluation

 

House, E. R. (1980a). Evaluating with validity. Beverly Hills, CA: Sage.

 

House, E. R. (1980b). Evaluation as persuasion: A reply to Kelly’s critique. Educational Evaluation and Policy

Analysis, 2(5), 39–40.

 

House, E. R. (1991). Realism in research. Educational Researcher, 20, 2–9.

 

House, E. R. (2007). Regression to the mean: A novel of evaluation politics. Charlotte, NC: Information Age

Publishing.

 

Scriven, M. (1980). The logic of evaluation. Inverness, CA: Edge Press.

 

Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice.

Newbury Park, CA: Sage.

 

Stake, R. E. (1967). The countenance of educational evaluation. Teachers College Record, 68, 523–540