The Evaluator As Moral Fiduciary

January 8, 2021

 

The Evaluator as Moral Fiduciary

 

Ernest R. House

 

            I have long advocated the idea of justice in evaluation and, in particular, the deliberative democratic approach. As society changes, we might add new techniques and practices to meet new challenges and strengthen our approach. What might these challenges be, and what should we do to meet them?

Evaluations don’t seem as effective as they once were. Evaluator Tom Schwandt (2019) contends that evaluation has entered a post-normal phase similar to that of post normal science. In normal science, scientific studies are widely accepted by the government and public and often acted upon. In post-normal times, findings are frequently disputed, criticized, and ignored. Environmental science is an example, as is the epidemiology of Covid-19.

            I believe these changes in evaluation and science reflect changes in the larger society. Over decades, American society has become increasingly fragmented, polarized, distrustful, unequal, and corrupt (House, 2020). In such a society, evaluations become less effective. How can we make them more effective? Evaluators might act as moral fiduciaries, cultivate cognitive empathy, focus on deep stories and deep values, control for biases, and be transparent. In this chapter, I outline significant changes in American society, delineate several practices that can strength evaluations, discuss conditions of their implementation, and briefly tie these practices to evaluation theory.

 

Societal Changes

 

Over the past seventy years, American society has changed dramatically. When I was young, say about 1960, there was a consensus among the majority that the government was good and that society was on the right track. This consensus emerged from World War II and Roosevelt’s New Deal (Kennedy, 1999). There was a palpable sense of unity and national purpose. During the 1960s, the consensus and trust in government began breaking down. There was the Vietnam War and the Civil Rights movement, followed by Nixon and Watergate. These events engendered discord and distrust.

By 1980 Reagan could run on the slogan, “The government is not the solution to the problem; the government is the problem.” He initiated an era of privatization, deregulation, and strong anti-government sentiment. In the new century, there was the contested Bush/Gore election, the trauma of Nine Eleven, the Afghan and Iraq wars, and the Great Financial Crisis. Distrust deepened. American society fragmented.

Three powerful trends emerged: a deep decline in trust, a sharp rise in inequality, and an increase in corruption. People did not trust the government, they did not trust their institutions, and they did not trust each other. Decline in trust was precipitous during Vietnam and Watergate when trust in government dropped from 77% in 1964 to 30% by 1980 (Rothstein, 2018). By 2014 trust in government was 20%. Trust in Congress was 6%. Not all countries are like this. Trust in government in the Nordic countries is 70%.

Another powerful trend has been steepening inequality. In 1981 Reagan lowered the tax rate on the wealthy, followed by more tax cuts by later administrations. America now has the most extreme inequality of income and wealth of any developed country (Saez and Zucman, 2019, Stiglitz, 2018, Piketty, 2014). The three richest Americans own more than the bottom 160 million. The richest .1 percent have as much wealth as the lower 90 percent (Hacker and Pierson, 2020, Krugman, 2017). CEOs of major corporations used to make 20 times the average worker wage. They now make 360 times as much (Dubchuk and Fried, 2004). Top hedge fund managers make 100 million dollars per month (Saez and Zucman, 2019). Worker wages have stagnated for decades. Social mobility has declined. Nobel economist Joseph Stiglitz (2018) said the economy is rigged in favor of the wealthy. Conservative media mogul Rupert Murdoch noted that America is developing permanent social classes.

A third trend has been worsening corruption. When Governor Blagojevich of Illinois tried to fill the Senate seat vacated by Obama, few expected him to auction the seat to the highest bidder for personal gain. Patronage had become corruption. American society was beset with cronyism, favoritism, and conflicts of interest. Special interests dominated Congress and legislatures to secure special favors (Lessig, 2018). Seventy-five percent of Americans believe corruption is widespread throughout the government (Rothstein 2018). Political scientists call such behavior institutional corruption

Distrust, inequality, and corruption interact in downward spirals. The more a government is distrusted, the less it can remedy corruption or inequality, and the more it becomes vulnerable to cronyism and conflicts of interest. Distrust allows corruption that generates more distrust and inequality. These trends are reflected in dystopian novels, apocalyptic films, and polarized politics (Hacker and Pierson, 2020).

 

Act as a Moral Fiduciary

 

            How should evaluators behave in such a society? Problems in the larger society are manifested in evaluation. Evaluators encounter fragmentation, distrust, inequality, and corruption, such as conflicts of interest, in their work. They are often mistrusted and work with stakeholders with quite diverse views. Some sources of inequality, such as tax policies, lie beyond evaluation, but evaluators can attend to inequalities, fragmentation, distrust, and corruption within the evaluation space. They can act as moral fiduciaries. They might assume some moral fiduciary responsibility for helping those lower in the unequal socio-economic structure. They can strive to protect the interests of those less advantaged. Here’s an example.

In 2009 eleven thousand forensic rape kits were discovered in a police warehouse in Detroit. The kits had not been processed. For the victims, undergoing a rape investigation is invasive. The process takes hours and involves swabbing samples from every orifice of the body. Rape is a crime often committed by repeat offenders, and rape kits provide an opportunity to identify serial rapists. Yet the kits remained unexamined for years.

In an examination of police files, Rebecca Campbell and her colleagues (Campbell, Shaw, and Fehler-Cabral, 2015) discovered that the police investigating the crimes repeatedly dismissed victim claims of rape on the grounds that the women were prostitutes, sexually permissive, or did not want their parents to know they had sexual partners. In the judgment of the police, the women’s claims were not worth pursuing as crimes. Such judgments reflected police racial framing about the presumed character of minority women. Police reports often referred to victims in highly pejorative terms.

Campbell and her colleagues worked through the cases with police until the rape kits were processed, an exhausting exercise, with police officials admitting that classifying women this way was mistaken. In a sense, the researchers appealed to empirical evidence and the police sense of fairness. This was a difficult study to conduct because it involved conflicts of deep stories and deep values. During the study, investigators kept in mind the welfare of those who had been abused. Their welfare was most at stake, not that of the police nor those who funded the study. In such conditions of inequality, evaluators might give priority to the interests of those less able to defend their interests.

What is a fiduciary? In the financial community, the idea of a fiduciary means that the financial agent must act in the best interests of the client. As an experienced investor, I can attest that investment transactions are riven with conflicts of interest. For example, if an investor asks the advice of a financial advisor, the advisor is free to recommend mutual funds that enrich the advisor, not the client. Advisors often receive fees or a portion of the investment from mutual funds they recommend. Clients suffer because this is not the best investment for them. This is standard practice in finance.

Fortunately, there are advisors who swear to a higher fiduciary ethical standard. That is, they will offer advice that is in the best interests of the client. If they fail, they can be sued. Reformers have tried to make the fiduciary ethic the legal standard for investment professionals, but professionals have fought vigorously against this, an indication of how prevalent not acting in the best interests of the client is. Nonetheless, some advisors do choose to act as financial fiduciaries.

Evaluators might act as moral fiduciaries, meaning they act in the best interests of those less able to defend their interests, rather than in the interests of the sponsors or other stakeholders. Such an ethic doesn’t preclude acting in other stakeholder interests as well, but it does mean those less advantaged will be given priority. In most programs, it’s not difficult to determine who’s less advantaged. In Detroit, Campbell and her colleagues chose to work through case records with police to process the kits. They did not seek to reform the police view overall, but to change their view about specific cases. Evaluators can take some responsibility for the severe inequality in their society by doing something about it in actual evaluations.

 

Cultivate Cognitive Empathy

 

In conducting the Detroit study, the evaluators exercised some complex skills. One was to understand what the police were thinking. That insight was key to figuring out why the rape kits had not been processed. In his analysis, Schwandt recommends that evaluators strive to understand other people’s perspectives. Evaluators cannot assume their perspective is the only perspective. This is critical in a fragmented society.

            Sociologist Mario Small (2019) calls such understanding “cognitive empathy.” He contends that social researchers are quantitatively literate, but not as qualitatively literate. Qualitative literacy entails understanding other perspectives in depth. Cognitive empathy is not feeling empathy. It’s not feeling the same as others. Nor is it sympathy, feeling sorrow or pity for them. Rather it’s the ability to understand people’s predicament as they understand it. Their view will seem rational within their perspective. We can understand why they believe the way they do. That doesn’t mean we agree with them. Certainly, Campbell and her colleagues did not agree with the police stereotyping of minority women.

Small cautions against overgeneralizing other views. He calls this avoiding “out-group homogeneity bias.” When groups are far removed from our perspective, their views appear less diverse than they are. Not all police are alike. Another caution is to be sensitive to data used as supporting evidence. For example, when journalists and others report standardized test scores, they often explain the results based on no empirical evidence whatsoever. They may contend scores are low because the education system is dysfunctional. But there’s no supporting evidence about causes in the test scores results. Where do such explanations come from? From beliefs and stereotypes held by those making the claims.

 

Focus on Deep Stories

 

Deep stories are the subjective prisms through which we view the world, including how we feel about it. They are stories that “feel right” emotionally. They are interpretations of events and situations that people act on. The idea of deep stories comes from sociologist Arlie Hochschild (2016), who studied Tea Party voters in Lake Charles, Louisiana. They voted for Trump overwhelmingly in 2016. In their view, life is a long march towards the American dream, which lies just over the hill. But the line they’re in is stalled. Others are cutting in line ahead of them, minorities and immigrants, people who used to be behind them. That’s not fair, in their view. Government agencies are helping these people. It’s the government’s fault. Trump also blames the government, minorities, and immigrants. He derides the elites. This is the “deep story” of the Tea Party. It’s also the deep story of Fox News, their source of news. Fairness is critical, construed a particular way.

Similarly, sociologist Katherine Cramer (2016) studied people in small towns and rural areas of Wisconsin. Their towns are struggling, though people work hard. They believe the government is taking money away from them and giving it to minorities and immigrants in the big cities. In their view, decision-makers don’t respect rural people. These resentful voters tipped the state to Trump in 2016. People act on their deep stories.

There are other deep stories, like that of the progressives in Berkeley. In their view, Americans built a magnificent public square, but marauders invaded the square, dismantled it, and stole pieces to build private mansions. Massive accumulations of wealth threaten democracy itself. That’s not fair, either. They supported Bernie Sanders (Hochschild, 2016). These are a few of America’s deep stories.

If I were evaluating an environmental education program in these communities, it would help considerably to understand their deep stories. Lake Charles has oil refineries and toxic industries. Even though people die of cancer at high rates, the residents see these industries as vital to their livelihoods and have ways of thinking about the problem, including religion. In rural Wisconsin my evaluation would be different, and different yet again in Berkeley.

 

Concentrate on Deep Values

 

Deep values are entwined in deep stories. The Louisiana story is built around deep values about race, gender, and social class, around assumptions of a hierarchy of race, gender, and social class. It’s assumed that some should be behind others in the natural order, a vision derived in part from Louisiana history, with its tradition of populist movements led by politicians like Huey Long (Hochschild, 2016). Within this perspective, social status is determined by how far a person is from the bottom. Being ahead of minorities and immigrants is a matter of entitlement.

Everyone has a hierarchy of values, with some values more central to the belief structure. These might be called deep values or core values. For most, they include family, fairness, and in-group loyalty. For some, they also include racism and sexism, deep-seated beliefs about who belongs where and deserves what. Not all deep values are good or benign. Everyone has a deep story and deep values, and it’s worthwhile for evaluators to reflect on theirs. I’ve discussed mine elsewhere, which began in childhood as a Roosevelt Democrat (House, 2015). Evaluators might locate their story within the population of deep stories. Evaluators grow up in a particular region, social class, and identify with those in their vocation. Having an idea of your position helps understand others and the constellation of perspectives in the evaluation space. An inability or reluctance to understand other perspectives is a common failure of American foreign policy. It can damage evaluations.

            I’m not suggesting that all views are relative or equally good. Some views are better than others because they are more moral and soundly based. An informed view includes a grasp of other views. A moral view takes into account the welfare of others. Even in fragmented societies, there may be room for agreement based on shared deep values. For example, in conducting an evaluation of environmental education in Lake Charles, I would focus on shared deep values. I would be unlikely to change their world view, but I might find agreement about the value of environmental education in specific areas, such as how pollution affects their children long term. They care deeply about their children’s future, and they have a sense of fairness. They want to be fair to their children. The deep value of fairness plays a central role for everyone.

 

Control for Biases to Enhance Fairness and Honesty

 

Being biased means being influenced by things evaluators should not be influenced by. Biases might be technical, like sampling error, or social, like racial and sexual framing, or situational, like conflict of interest, or psychological, like inappropriate anchoring. Whatever the biases, they can result in distorted findings. Stakeholders are justified in seeing biased studies as unfair. Taking care to mitigate bias is critical in conducting fair evaluations. Fortunately, there are analyses about how to protect against different biases (Scriven, 1976, Shadish, Cook, and Campbell, 2002, Kahneman, 2011, House, 2011).

 

Focus on Racial and Sexual Framing

 

Racial framing is a key mechanism through which racist beliefs are perpetuated. They play a huge role in American history (Feagin, 2011, DuBois, 1986). The white racial frame asserts that African Americans are violent, criminal, unintelligent, lazy, and oversexed (Feagin, 2013, p. 101). Whites are superior. Whites are immersed in racial framing in childhood and often act on it unconsciously (House, 2017). The effects are pernicious. Something similar happens with stereotyping females. The tacit nature of racial and sexual framing makes such biases extremely difficult to eradicate. Evaluators should look carefully for social biases in programs and evaluations, including program effects (House, 2017). They should check their own predispositions, those of colleagues, and expect them to check theirs.

 

Be Transparent

 

Trust is at a premium in fragmented societies. Transparency engenders trust; lack of transparency engenders mistrust. In climates of distrust, people imagine bad things are happening. Evaluators should be transparent about what they’re doing. In a Denver bilingual education program beset with years of distrust between the school administration and Latino community, in my role as federal court monitor, I made clear to each group what I was doing. I was open to discussion and recommendations from them. I shared the data we collected and solicited advice as to what to collect next to determine whether the program was being implemented properly. Over time transparency helped establish trust in the evaluation and improved trust among groups. Transparency is no panacea, but it helps.

 

Implementing the Practices

 

The purpose of these practices is to enhance the effectiveness of evaluations and address societal problems. The practices, singly or collectively, could strengthen many evaluations.

Acting as a moral fiduciary would be most useful in projects where there is a steep hierarchy. Inequalities in the larger society are manifested in evaluations. This can occur in large or small, formative or summative evaluations. An early evaluation task would be to carefully consider the hierarchy. Race, gender, and social class are key. The criterion for success is how well the evaluation protects and prioritizes the interests of those least advantaged.

For example, in the “Women Affirming Motherhood” program, two women developed a program to provide pre and postnatal care to minority mothers (Alkin and Christie, 2019). Later, a large NYC foundation offered funding if the program became one of several demonstrating interagency cooperation, a foundation goal. There evolved a steep socioeconomic hierarchy extending from the mothers and infants to the chair of the foundation. This hierarchy included the mothers and infants, those working directly with the mothers and infants, the program founders, the coordinating agencies, the foundation officers, and the foundation board. The hierarchy itself presented problems of understanding and communication.

In such a hierarchy, the interests of the mothers and infants might well be neglected. Those in each position have different perspectives, values, and interests, as well as age, race, gender, and class differences. Since the program was developing, I proposed a formative evaluation focused on helping the mothers and infants foremost, those who were least advantaged. A central evaluation focus was to determine whether they were getting the services they needed. Even with the best intentions, their welfare might be overlooked amidst the micropolitics and machinations of a large endeavor. We have seen the value of patient advocates who help patients in large hospitals. I envisioned the program founders, mothers, and core workers as prime audiences for the evaluation since they had most effect on the infants and most urgent need for information. (For details about how this might play out, see House, 2019).

Cultivating cognitive empathy can be complementary to any evaluation strategy, large or small, perhaps more useful in formative studies. There are several ways to accomplish this, depending on evaluation resources. The most likely path is for the evaluator to grasp other viewpoints through extensive discussions. Cognitive empathy should make a difference in how the evaluation is designed and received. If not, the evaluator isn’t delving deeply enough. Attaining cultural empathy can be built into the evaluation design by including regular interactions with stakeholders as on-going data collection in the study.

Focusing on deep stories and deep values provides a sense of direction about what to look for. What are these people about? What do they want? How do they see the world? There is a logic to their actions from their perspective. Some glimpse of this logic is invaluable in designing and carrying out an effective evaluation. Evaluators don’t need polished studies, such as those constructed by sociologists like Hochschild (2016), Kramer (2016), and Desmond (2016). Generally, the more interactions evaluators have with stakeholders, the better the possibility of understanding.

There are a few deep values that bind us as a community, including family, fairness, and in-group loyalty, as well as sexual and racial attitudes. How these are woven into deep stories marks us and leads us to act in certain ways. Since people share deep values, there is usually room for some cooperation, even amidst fragmentation.

Controlling for biases is obligatory for all evaluations. By biases I mean mistakes that can invalidate the study. If an evaluator draws an inappropriate sample, employs the wrong statistic, or uses racial or sexual stereotypes, observers might rightfully claim that such a study is biased, unfair, and even invalid. Over decades evaluators and others have assembled lists of actions and conditions that can cause people to arrive at incorrect findings in certain conditions. We call these biases. Knowledge of biases is core knowledge in evaluation. All biases have been demonstrated empirically to cause errors when drawing conclusions.

How transparent should evaluators be? There should be no secret side deals with various stakeholders. Even if innocuous, these destroy trust. Explaining the study may be enough, but if some stakeholders are more interested in particular aspects of the study, evaluators should accommodate them. There is no need to explain every detail. Rather, tell them what they need to know and what they believe they need to know. Interact with them frequently. Keep them informed periodically. Face to face meetings are most useful. People can “read” how credible you are through non-verbal cues. Also, you can learn much important information you never suspected. Transparency establishes trust in times of distrust.

 

Connection to Theory

 

The ethic of the moral fiduciary fits a long tradition of moral philosophy. Rawls (1971) theory of justice as fairness changed the dominant utilitarian conception of justice. With utilitarianism, you could justify fighting inflation by inducing a recession that forced large numbers of workers out of jobs. They would suffer, but the larger society benefited--greatest good for the greatest number—but without regard for how benefits and suffering were distributed within society. In elaborate arguments, Rawls said this was unfair. We should attend to those less advantaged as a moral duty.

Later critics of Rawls said those left out of the decision processes should also have some voice in making decisions that affect them. They could not always rely on decision makers to make the right decisions without being included in the discussions. Deliberative democratic advocates recommended inclusion, discussion, and deliberation of stakeholders in decision making and evaluation (Gutmann and Thompson, 1996, House and Howe, 1999).

However, democratic participation alone doesn’t seem potent enough in a society so unequal. Evaluators should take stronger actions to protect the interests of those less advantaged. Acting as moral fiduciaries is one way to help. Instead of balancing the interests of different stakeholders equally or putting the interests of sponsors first, evaluators might give priority to those who need help most. The core value is fairness, fairness in both society and evaluation.

 

References

 

Alkin, M. and C. A. Christie. (Eds.). (2017) Theorists’ Models in Action. New Directions in Evaluation, 163, Fall, 161-72.

Bebchuk, L and Fried, J. (2004). Pay without performance. Cambridge: Harvard University Press.

Campbell, R., Shaw, J., and Fehler-Cabral. (2015). Shelving justice: The discovery of thousands of untested rape kits in Detroit. City and Community, 14:2, June, 151-166.

Cramer, K. (2016). The politics of resentment. Chicago: University of Chicago Press.

Desmond, M. (2016). Evicted. New York: Crown.

DuBois, W. E. B. (1986). Writings. Library of America. New York.

Feagin, J. R. (2013). The white racial frame. (2^nd ed.). New York: Routledge.

Guttmann, A. and Thompson, D. Democracy and disagreement. Cambridge MA: Belknap.

Hacker, J. S. and Pierson, P. (2020). Let them eat tweets. New York: Liveright.

Hochschild, A. R. (2016). Strangers in their own land. New York: New Press.

House, E. R. and Howe, K. R. (1999). Values in evaluation and social research. Thousand Oaks, CA: Sage.

House, E. R. (2011). Conflict of interest and Campbellian validity. New Directions for Evaluation, 2011, 69-80.

House, E. R. (2015). Evaluating: Values, biases, and practical wisdom Charlotte NC: Information Age.

House, E. R. (2017). Evaluation and the framing of race. American Journal of Evaluation, 38(2), 167-189.

House, E. R. (2019). Evaluation with a focus on justice. In M. Alkin and C. A. Christie. (Eds). Theorists’ Models in Action. New Directions in Evaluation, 163, Fall, 161-72.

House, E. R. (2020). Evaluating in a fragmented society.  Journal of MultiDisciplinary Evaluation. 16(36), 26-36.

Kahneman, D. (2011). Thinking, fast and slow. New York: Farrar, Strauss, Giroux.

Kennedy, D. M. (1999). Freedom from fear. New York: Oxford University.

Kramer, K, (2016). The politics of resentment. Chicago: University of Chicago.

Krugman, P. (2017). The gilded age. In H. Boushey, J.B. DeLong, and M. Steinbaum (Eds). Beyond Piketty. (2014). Cambridge MA: Harvard University Press. 60-71.

Lessig, L. (2018). America, compromised. Chicago: University of Chicago Press.

Piketty, T. (2014). Capital in the twenty-first century. Cambridge MA: Belknap.

Rawls, J. (1971). A theory of justice. Cambridge MA: Belknap.

Rothstein. B. (2018). How the trust trap perpetuates inequality. Scientific American. 319, 5, 7.

Saez, E. and Zucman, G. (2020). The triumph of injustice. NY: W.W. Norton.

Schwandt, T. A. (2019). Post-normal evaluation? Evaluation. Vol 25(3), 317-329.

Small, M. L. (2019). Rhetoric and social science in a polarized society. 2019 Spencer Lecture, American Educational Research Association, Toronto, Canada, May 22.

Scriven, M. (1976). Evaluation bias and its control. In G. V Glass (Ed.). Evaluation studies review annual. Pp. 119-139. Beverley Hills: Sage.

Shadish, W. R., Cook, T. D., and Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.

Stiglitz, J. E. (2018). A rigged economy. Scientific American, 319:5, 57-61.