Ernest R. House
Anyone who has conducted an evaluation knows there’s the way you planned to do the study and the way you actually did it. Usually, something happens that throws you off track and forces you to wonder what to do next. How well you can handle such disruptive events often involves practical knowledge or practical wisdom. Good evaluation doesn’t depend entirely on plans and methods. It also depends on practical knowledge learned through on-the-job experiences. The concept of practical wisdom goes back to Aristotle. Aristotle conceived the idea by watching blacksmiths, carpenters, shoemakers, boat pilots, and other craftsmen around Athens apply their skills. The workmen did not have rigid rules they followed. The materials they worked with and the tasks they performed were too varied and irregular. New applications often posed new problems. For example, when stonemasons had to carve rounded columns, they took a stiff, flat ruler and changed it to a flexible one that could measure curved surfaces. Aristotle thought that practical wisdom consisted of such appropriate actions taken in particular work contexts.
In their book Practical Wisdom, Schwartz and Sharpe (2010) have updated Aristotle’s concept. They tell the story of Luke, a custodian at a large hospital. A patient in one of Luke’s rooms was a young comatose man who had gotten into a fight in a bar and been seriously injured. He had not gained consciousness for six months and showed no signs of doing so. During this long period, his father spent a huge amount of time at his son’s bedside. One day, while the father was out smoking, Luke cleaned the room. When the father came back, he became angry and accused Luke of failing to clean the room. Luke started to argue, but then decided that wasn’t the best course of action. Luke cleaned the room again with the father watching him.
Schwartz and Sharpe call this is an excellent example of practical wisdom, of doing what makes the situation better, instead of doing what’s in the job description. Luke could have argued, insisting that he had cleaned the room. But Luke thought about what the father had been through emotionally. What good would arguing with him do? Instead, Luke cleaned the room again, making the man feel that he had done something worthwhile for his son, even if the son was far, far out of reach. Luke did the humanitarian thing, something no one else would ever know or appreciate. He did the practically wise thing.
Practical wisdom consists of doing the right thing in the special circumstances of performing the job. Professional practices like evaluation require similar choices. What do you do in difficult situations? The knowledge and wisdom to address such questions is practical. It depends on your ability to perceive the situation, figure out the appropriate feelings about it, think about what’s appropriate, and act. Practical wisdom involves learned character traits like self-control and self-knowledge. It involves making choices among desirable outcomes that conflict with one other and choosing between better and best possibilities. It requires nuanced judgments in context.
In the evaluation community, Tom Schwandt (2005) has written most insightfully about professional practice and practical knowledge. He’s emphasized the centrality of practice to evaluation. Professional practice is a complicated set of transactions in which practitioner judgments are tested, and there’s no way beyond these social interactions. “Our everyday practice as teachers, managers, social service workers, and health care providers tells us that no escape from these dilemmas can be found. We are…always on the ‘rough ground’ where values, personalities, evidence, information, feelings, sensitivities, emotions, affect, ambiguities, contradictions, inconsistencies, and so forth are simultaneously in play as we try to do the right thing and do it well” (Schwandt, 2005, p. 99).
Schwartz and Sharpe say the practically wise person--
--Is guided by the overall aims of the activity;
-- Knows how to improvise, to balance conflicting aims, and to interpret rules in light of the particulars of the context;
--Is socially perceptive and knows how to interpret the social context;
--Knows how to understand the perspectives of other people, to see the situation as another person might see it, and to know how the other person feels;
--Knows how to make emotion an ally of reason and to rely on emotions to signal what’s called for;
-- Is highly experienced: practical wisdom is based on lots of experience.
Luke framed the situation as how to care for and sustain the relationship of father and son. Not as one of simply cleaning the room. He framed the situation through a narrative he fashioned about what had happened and what was happening emotionally between father and son. He knew their story and attended to it respectfully. He also knew that the men with whom the son had been fighting were African Americans, the same as Luke. He considered the possible courses of action in light of the circumstances. Luke’s interpretation of the story framed the issue in a way that Luke knew what to do. His ability to construct such a narrative was based on the type of person he was and on his many prior experiences in the hospital.
The ability to construct appropriate stories and frame situations with them is critical to practical moral skill. So too is the ability to use analogies and metaphors to draw on past experiences. Luke remembered the consequences of similar actions he had taken in the past. Perhaps he had argued with someone in a similar situation and had not liked the consequences. Learning from experience often means learning from bad experiences. Figuring out what’s appropriate also requires moral perception and empathy. Luke had to imagine what the father must be feeling and thinking. And he was motivated to act on his perception.
Luke seems to be a compassionate man able to empathize with others. I doubt that I would have risen to such an occasion nearly as well. Based on my character, I suspect I would have argued with the father, not because I couldn’t see his plight, but because I have a tendency to push back when someone accuses me of something I didn’t do, especially if they are aggressive about it. I would have regretted the confrontation later when I thought it over, perhaps even been chagrined. I’m less compassionate than Luke. However, I can see how Luke’s actions were superior to how I might have acted. Perhaps if I had Luke’s practical experiences, I could have done the same compassionate thing. Interestingly, in this same hospital many other custodians also perceived themselves as part of the caretaking process. Practical wisdom is partly the ability to see how options will play out. Empathy is critical, discerning what others are thinking and feeling. Emotion is also critical as a signaling device. Luke could see how angry the father was, and he knew the father’s anger wasn’t only about cleaning the room. It was about the entire unfortunate, unfixable tragedy. Luke, in turn, had to manage his own emotions skillfully. Surely, he must have felt a surge of anger when the father lashed out at him unjustifiably. Luke had to assess and discipline his own feelings and reactions. He didn’t get defensive, at least not after his immediate reaction.
Practical wisdom is not about establishing rules and following them. It’s about if, when, and how to apply the rules. It’s context dependent and operates in areas of gray, not black and white. By themselves, rules tend to marginalize the importance of character traits like compassion, empathy, courage, patience, and kindness. You learn rules from books and teachers. You learn practical wisdom from experience, coaches, and mentors—if you’re lucky! We should aspire to this level of understanding and action in the evaluation profession. After all, evaluation has the capacity to inflict considerable damage if the evaluator is uncaring.
What are other components of practical wisdom? Cognitive components include fuzzy concepts, frames, stories, pattern recognition, and intuitive thinking. Fuzzy concepts are those not clearly defined. The concept of fruit is an example. The natural category of fruit has graded membership. Some fruits are core examples of the concept, while others have fewer degrees of membership. Humans have great capacity to handle fuzzy concepts like this, and this fuzziness allows them to make nuanced judgments in different contexts based on the fuzzy concepts.
For example, if a judge and jury must make a careful determination about whether an act is an “armed robbery,” their considered judgment depends on the many particulars of the action and the context. According to researchers, we think with such fuzzy ideas. When to apply and balance concepts like these and how to apply fuzzy concepts like “include stakeholder views in the evaluation” requires interpretation in context. There are no hard and fast rules for doing so. Practical wisdom improves with experience.
Another key element of practical wisdom is framing. Frames tell us what’s important, what to compare, and help us discern what’s relevant in a particular context. All humans have the capacity to frame, which allows them to interpret events. Even medical professionals prefer a medical treatment that offers a ninety percent cure rate rather than one that offers a ten percent mortality rate. It’s the same substance but different framing. Of course, the wrong frames can also bias and distort. Negative uses include political spinning of events. There is no neutral frame-free way to interpret or evaluate anything. The problem is to choose the right frame. I learned long ago to be especially careful of what words I use in reports to describe people’s actions. If the word can be interpreted negatively, it often will be.
Luke could have framed his situation as strict job performance. He could have told the father that he had cleaned the room and pointed to the signs that he did. Or he could have framed the situation as protecting his job. He could have gone straight to his supervisor. If the father had told Luke’s supervisor that Luke had failed to clean the room, Luke might have been reprimanded. Instead, Luke framed the situation with a narrative about where the father and son stood at this moment in their lives and acted humanely. Stories and narratives are powerful ways that we frame and understand lives. They depend on values, character, and experience.
Such thinking also includes pattern recognition. When people become expert at performing certain tasks, they recognize patterns and combinations of patterns they’ve seen before. They’ve learned these patterns through experience, including bad experiences. Learning depends on the community, watching others, and making decisions. Of course, some decisions require careful deliberation, especially when moral perception is conflicted or unclear. In Luke’s case, his thinking involved a considered balancing of different actions. It may have invoked a familiar pattern he recognized that enabled him to respond quickly and appropriately.
How might practical wisdom play out in evaluations? There are many opportunities for evaluators to employ practical wisdom when they conduct evaluations For example, in the late 1990s, I was chosen to monitor the implementation of a Denver bilingual program and decided to do so utilizing a monitoring evaluation (House, 2015). The Denver bilingual program was highly controversial and had been in the news for many years. Plaintiffs had gone to federal court and negotiated a language instruction program with the school district for students who could not speak English. Most were recent Latino immigrants. After years of disagreement, the antagonisms between the school district and Latino community had grown intense.
My first meeting with the Denver superintendent of schools was difficult. The superintendent wanted me to act as the total arbiter of the new court agreement. In other words, I would collect data as I saw fit and dictate whether the program was being properly implemented. He wanted me to assume the legal authority of the court and declare “what was what,” in his words. However, after thinking about the conflicted politics, I decided to be more conciliatory and inclusive. I had been involved in many politicized evaluations before, and my experience was that the long-term results of the evaluation would be better if you listened to people in the disputes, tried to understand their points of view, and addressed their concerns. At the same time, I had learned that you can’t let others run over you or you would lose control of the evaluation. I decided to strike a balance between using my authority and sharing it with stakeholders. This required balanced judgment.
I also decided to be open about what I was doing and try to build a sense of trust and credibility among the various parties. The superintendent thought that the Latinos would never accept those conditions and would take advantage of them. I wasn’t sure how things would work out either. To his credit, the superintendent accepted my approach, though he disagreed with it. After six years and many contentious interactions later, the parties reached a point where they could conduct reasonable discussions with one another.
For this project I built a monitoring plan around the new data system the Denver school district was implementing. The district had contracted the development of the data system to an outside group. My plan was to collect quantitative data on a few key indicators in each school. Where I saw that schools were showing poor performance on the implementation indicators, my plan was to send in an evaluation team to those schools to ascertain what the specific problems were. Employing the data system meant we would not have to visit every one of the hundred participating schools.
Unfortunately, the outside contractors were having many technical problems merging files, etc. In short, the data system wasn’t usable. Relying on what the contractors told them, the district said the new system would be ready shortly. Based on my previous experiences with installing new technologies, I was silently skeptical. An early evaluation I conducted was of the PLATO teaching and learning system at the University of Illinois in the early 1970s, one of the first computer-assisted learning attempts. It relied on computer terminals in the Chicago community colleges hooked up to mainframe computers on the Urbana campus. The whole system was ingenious technically, one of the first uses of touch screens. However, not even the high-powered engineers at the university could keep the physical equipment working long enough for the instructors to use the system. The instructors would try the terminals a few times, and if they didn’t work, they would quit. They had too many other things to do than mess about with nonfunctioning equipment. I knew from several evaluations that new data systems usually encounter long delays.
What should I do? I anticipated that the data system might take a couple of years to become functional (which turned out to be accurate). Should I wait two years and do nothing? The non-functioning data system wasn’t my fault, and no one would blame me for the delay, but the major stakeholders needed information right away about whether this controversial program was being implemented properly. A long delay seemed unacceptable.
Another possibility was to send questionnaires to the schools. However, I was reluctant to do that. In evaluating the Illinois Gifted Education program (my first evaluation), I mailed questionnaires to a thousand schools. The information I received back was garbage. The questionnaires to the Chicago high schools came back in one package, filled out in the same handwriting, tied together with a ribbon. According to the answers, every high school was providing every desirable option for academically talented children. Since that time, I’ve been highly suspicious of self-report questionnaires when respondents think they have something to lose.
Another alternative was to visit all the schools in the program over the next few years, using a checklist based on the required features of the program. That meant a labor-intensive approach. The approach was inefficient, but I didn’t see a better choice. I had wanted quantitative indicators initially because they would be more credible to all parties rather than relying on checklist visits conducted in a politicized atmosphere. However, given the circumstances, it would have to be the other way around, with the checklist visits backed by indicators from the data system eventually. To bolster the credibility of my approach, I enlisted the help of the major stakeholders in constructing the checklists. As it turned out, the checklist worked pretty well. Simple checklists can be valid and useful if handled properly.
Another problem was the achievement testing. The bilingual program depended heavily on testing students who were deficient in English to gain entry to the bilingual program and testing them again later to move them into mainstream classes. A large number of staff members around the district administered the tests. My previous experience indicated that such testing procedures would produce highly variable results. Testing was not reliable with minimal training for so many different testers. However, I didn’t mention this at the beginning of the project since we had many other things to do.
I waited until we obtained some peculiar results in the testing data. I suggested that one possible source of the peculiarities might be unreliable administering of the tests. Rather than claim that it was so, I suggested the school district run its own study. The assistant superintendent was quite competent in research and conducted a small study, revealing that the testing was unreliable. The results were so far off that the district personnel could see the unreliability could make the program look ineffective. Consequently, the district reduced the number of testers and instituted retraining for those remaining.
There’s no doubt that my previous experience in evaluation over many years alerted me to these problems and how to deal with them. I also treated people better and worked with them better as a result of being involved in many evaluations. The idea of practical wisdom means having the right values and information and being able to frame issues in the best way. This isn’t knowledge readily available except through prior experience. (For other ideas about learning practical wisdom and knowing when practical wisdom is valid, see House, 2015).
References
House, E. R. (2015). Evaluating: Values, Biases, and Practical Wisdom. Charlotte, N.C.: Information Age Publishing.
Schwandt, T. A. (2005). The centrality of practice to evaluation. American Journal of Evaluation, 26(1), 95-105.
Schwartz, B., & Sharpe, K. (2010). Practical wisdom. New York: Riverside Books
~June 5, 2017
No comments:
Post a Comment