2015
Ernest R. House
A little over twenty years ago, an editor from SAGE contacted several people to ask if the publisher should start a new evaluation journal located in London. Some said, no, there were too many evaluation journals already. I said, yes, we could use another journal, depending on what it did. The existing journals were mostly North American. They were good journals, but they had a particular style and sense of content that excluded many ideas.
I had spent time in Britain, Sweden, Norway, and Spain. It seemed to me that some interesting evaluation ideas in those places were unlikely to be published in the existing journals. After twenty years of Evaluation, I believe my assessment was correct. Many quality articles would not have been published. Based on this criterion, the journal has been highly successful. A second beneficial outcome has been that the journal has led to more interdisciplinary and international exchanges. Many authors would have published solely in the journals of their home disciplines, countries, and languages. Mixing ideas can lead to new combinations, some of which may be useful. A third positive outcome has been the furtherance of on-going debates. These debates can (sometimes) clarify issues. I attribute this last feature to a contentious and ubiquitous editor who is not afraid of controversy.
To illustrate these claims, I’ll discuss two streams of ideas that would not have received as much attention without the journal. One stream is from Britain and the other from Sweden. Since the editor has asked contributors to comment briefly about the journal’s history, I present these ideas as personal reflections.
Britain
In 1975, when I arrived on my first trip to England, a staff member from the Centre for Applied Research in Education (CARE), University of East Anglia, met me at Heathrow. On the train to Norwich, we went to the dining car, where the waiter greeted us with, “And what will you ‘gentlemen’ have?” I had never heard “gentlemen” spoken as an insult before. It was masterfully cast so that we would be insulted if we were gentlemen and also if we weren’t. On arrival, I told Barry MacDonald, my host, who laughed and began tutoring me in British social class behavior.
My first supposition is that MacDonald’s democratic evaluation was built on his assessment of the British social class and governing structures. This is hardly surprising. Approaches to evaluation elsewhere differ according to such structures. What was unusual was MacDonald’s drive and determination to construct an alternative. In his view, the British governing structure was elitist, closed, and deeply embedded in social class privilege. He had no love for it. He had suffered public and private humiliations from it, and, as a working class, Highland Scot, he already had a formidable head start on hostility. From his perspective, information was power and used mostly against those lower in the social hierarchy. In evaluation the critical issues were who gets to know what about whom and who owns the data. Evaluators should strive to reduce the power differential among participants. He classified evaluations as bureaucratic, autocratic, and democratic, and he developed procedures aimed at reducing the power differentials among participants and protecting them from abuse (Norris, 2015, Simons and Greene, 2014).
These were provocative ideas. Where did they come from? My second supposition is that MacDonald’s alternative view incorporated ideas from Scottish intellectual history. MacDonald was a Scottish Moralist with both a capital and small “M.” The Scottish Moralists included a long line of distinguished Enlightenment thinkers in the 1700s, including Francis Hutcheson, David Hume, Adam Smith, and Thomas Reid (Broadie 2012, 2010). For example, Reid, from Aberdeen, started the Common Sense school of thought. Knowledge could be based on observation and common sense and could be acquired by anyone. Ordinary language and reason were all people needed. Basic principles were “self-evident.” (Thomas Jefferson borrowed this phrase for the American Declaration of Independence.)
These Scottish thinkers held diverse beliefs and engaged in ongoing debates with each other, but they shared a few core ideas: first, belief in the autonomy of reason, sometimes called the autonomy of the individual, and, second, confidence in the importance of communal discussion, which required tolerance for the expression of novel ideas. Individuals were capable of thinking rationally, given a chance, and society was capable of rational direction. They were pushing against argument from authority.
Enhancing the autonomy of individual reasoning was a major theme throughout the CARE enterprise. I presume that Lawrence Stenhouse, the director, was the carrier of these ideas, as manifested in the Humanities Curriculum Project (HCP). The teacher acts as a discussion leader who does not make value judgments. That’s left to the students who are free to discuss and form their own opinions. The teacher leads students in discussions through which they make up their own minds. The similarities between HCP and democratic evaluation are strong.
In addition to these ideas, MacDonald was outstanding in practical wisdom, that is, in knowing how to do evaluations. Practical wisdom is what you learn from experience and use in particular situations. Much of his influence was communicated face-to-face as practical knowledge. Practical knowledge is often about the politics of the project. I would say that practical wisdom accounts for much of what evaluators do in practice (House, 2015).
What did MacDonald know that was practical? Perhaps his greatest talent was in understanding people. He studied people the way some study ideas. He conducted case studies of people. In short, he knew people’s stories. If you know people’s stories, you can be highly persuasive by connecting your story to their story. If your story becomes part of their story, you have secured a strong advocate for your views.
Using this talent, MacDonald obtained evaluation contracts in spite of on-going disputes with governing authorities. He knew that British establishment figures like Richard Hooper, head of the national computer program, shared common beliefs about democracy. Democratic ideals were part of how they saw themselves, even though they might act differently in the closed decision making system they were in. Hence, one might expect ambivalence in their actions as they struggled with their ideals on the one hand and access to power on the other.
In summary, MacDonald had a personal, class politics that led him to envision evaluation the way he did. His goal was to rectify deficiencies he saw in a closed, elitist governing structure deeply embedded in social class. He constructed an alternative with ideas from his cultural heritage. His knowledge rested on understanding the social structure and how people acted in it. His influence spread through practice, mentorship, and face-to-face contact. His colleagues, including Helen Simons, Saville Kushner, Nigel Norris, and May Pettigrew, have advanced these ideas in print and practice.
Sweden
Another stream of ideas comes from Scandinavia. Serving on advisory committees in Sweden led me to believe that the Scandinavians are more egalitarian than others. They attend more closely to what others say and practice more open communication (Karlsson, 2001). There seems to be greater equality in social and political relations. This is manifested in the relatively large number of women in high positions, openness to comment and critique, and widespread sharing of resources. (This is not to say that Scandinavians do not have inequalities. Of course, they do.)
The origin of these ideas is a bit obscure. Some point to the practice of the Ombudsman, to a long history of public hearings about legislation that involves stakeholders, and to large publicly funded welfare systems (Forss and Rebien, 2014, Bohni Nielson and Winther, 2014). Fifty percent of the national income of Scandinavian countries is spent on government services, compared to 40% in the larger European countries and 30% in the U. S. and Japan. These sizable differences in expenditures reflect significantly different societal perspectives.
In evaluation this disposition is manifested in greater inclusion of others in evaluations and more dialogue among involved parties. Stame (2013) has noted an emphasis on critical dialogue as an important component of evaluations and an inclination towards theory development. Early influential evaluation figures include Sigbrit Franke and Ulf Lundgren, both of whom have held high posts in government. Colleagues advancing these ideas include Evert Vedung, Ove Karlsson Vestman, Christina Segerholm, Peder Haug, Anders Hanberger, and Hanne Krogstrup.
A recent disagreement among Scandinavians is about whether there is a Nordic or Scandinavian tradition of evaluation. I would distinguish between having a single tradition as opposed to having several streams of ideas. There could be many streams of ideas and practices in a country or region without there being one unified tradition. Indeed, some ideas develop in opposition to others. The same is true for Europe and North America. There are many streams of ideas, not single unified traditions.
Elsewhere
When introducing social justice ideas to evaluation in the 1970s, I relied on American intellectual sources by recommending evaluators apply John Rawls principles of social justice to evaluations. Later, in the 1990s, when Ken Howe and I constructed a deliberative democratic approach, we were influenced by critiques of American governing structures by the deliberative democratic thinkers, such as political scientists Amy Gutmann, Dennis Thomas, and James Fishkin. And we were also influenced by the British and Scandinavian ideas.
Robert Picciotto (2015) has argued that democratic approaches to evaluation depend on societies being democratically inclined. He contends that democratic countries are becoming less democratic and that only 45% of United Nations member countries are liberal democracies. Hence, different approaches are needed to advance democratic evaluation. He offers a modification he calls progressive evaluation.
This approach would be “autocratic democratic.” That is, evaluators would take full responsibility and authority for the evaluation, and the content of the evaluation would be forthrightly democratic. Evaluators would not accept limiting contingencies on their studies. Second, evaluation funding would be independent of the sponsors and program. Funding sources often limit what evaluators can do. Third, evaluators would make alliances with other organizations that are democratically inclined. Fourth, the methods would be impartial and based on content from democratic approaches.
What can we make of these contentions? I agree that democratic approaches depend on democratic backing and that many societies are becoming less democratic. Picciotto’s argument is based in part on my analysis of the privatization of the evaluation function in the pharmaceutical and finance industries. In some cases, sponsors have captured the evaluations. Conflict of interest in evaluation has become so pervasive that it has biased the findings of major drug studies and contributed to the great financial crisis.
I agree that we should strive to meet these challenges, but I’m unwilling to give up on liberal democracy altogether. There are many places where democratic approaches can be applied and where stakeholders can be brought together. As a profession we can also act to curtail pervasive conflict of interest in evaluation, reforms I’ve outlined elsewhere. More generally, I believe that evaluations of all types can be more democratic, even randomized studies. Evaluators might start from what people are comfortable with and democratize from there.
As for transferring democratic approaches to non-democratic societies, I’m a novice. My experience is that ideas must find grounding within the local culture and that experts from outside are not good judges of the possibilities. I’ve been cautious about the prospects of transplanting democratic ideas to non-democratic settings. American-led efforts to install democratic regimes elsewhere have not been successful, to say the least. Nonetheless, I’m open to new ideas.
In spite of such reservations, Picciotto’s effort to advance democratic evaluation is commendable, especially at a time when democracies are seriously challenged by rising inequalities. Of course, there are significant issues. To what degree have the supportive cultures changed in democratic societies? Are the societies we live in democratic enough to host democratic approaches? Are there conditions in which democratic ideas might thrive? If so, which ideas and what conditions? Should we try to employ democratic approaches in non-democratic settings? Is an autocratic approach a good way to transfer democratic ideas? As illustrated by these various examples, the politics of evaluation are intricately bound to the evaluation of politics, which means that evaluators have to rethink what they are doing periodically. I expect the journal to be an important venue for forthcoming discussions of these issues, as it has been for the past twenty years.
References
Bohni Nielsen S and Winther DM (2014) A Nordic evaluation tradition? A look at the peer-reviewed evaluation literature. Evaluation 20(3): 311-31.
Broadie A (2012) The Scottish enlightenment. Edinburgh: Berlinn.
Broadie A Ed. (2010) The Scottish enlightenment: An anthology. Edinburgh: Canongate.
Forss K and Rebien C (2014) Is there a Nordic/Scandinavian tradition? Evaluation 20(4): 467-70.
House ER (2015) Evaluating: Values, biases, and practical wisdom. Charlotte, N. C.: Information Age Publishing.
Karlsson O (2001) Critical dialogue: Its value and meaning. Evaluation 7(2): 211-27.
Norris N (2015) Democratic evaluation: the work and ideas of Barry MacDonald. Evaluation 21(2): 135-42.
Picciotto R (2015) Democratic evaluation for the 21st century. Evaluation 21(2): 150-66.
Simons H and Greene J (2014). Against the odds, but worth it: Democratic evaluation. European Evaluation Society, Dublin Oct 3.
Stame NA (2013) European evaluation theory tree In: MC Alkin (Ed) Evaluation roots, 2nd ed. Thousand Oaks, CA: SAGE: 355-70.
No comments:
Post a Comment