Evaluating Consensus Building Efforts: According To Whom? And Based On What?

These questions are certainly worth answering. But after twenty years, attempts to evaluate consensus building efforts in the public sector have failed to produce agreement on even the right criteria to use for evaluation.

Take, for example, the debate over attempts to evaluate negotiated rule-making efforts at the federal level that reached its apex in the Duke Law Journal in 1997.

Assistant Professor Cary Coglianese of Harvard University's John F. Kennedy School of Government carried out what most quantitatively inclined political scientists would consider to be a methodologically sound study. He reviewed a dozen US Environmental Protection Agency (EPA) negotiated rule makings - known as "reg-negs" in an attempt to determine whether these efforts saved time or reduced litigation costs compared to traditional EPA rulemaking efforts.

Coglianese found that, on average, reg negs had not saved time and that the rules produced through the reg neg process ran a higher risk of legal challenge than did EPA rules produced under traditional procedures. Based on these findings, Coglianese recommended that the federal government stop encouraging the use of negotiated rulemaking.

Philip Harter of the Washington-based Mediation Consortium responded that Coglianese had missed the point, that his methods were faulty, and as a result his findings and conclusion were invalid. Harter, one of the leading practitioners of regulatory negotiation, argued that Coglianese had not looked closely enough at the specific circumstances surrounding each negotiation.

Qualitatively, Harter said, the EPA rules produced through reg negs were not comparable to most EPA regulations. EPA had chosen particularly controversial issues for regulatory negotiation, hoping that the reg neg process could resolve conflicts that had stymied the traditional rulemaking process.

Harter also pointed out that some of the reg negs studied by Coglianese were not representative of reg neg's full potential because there were not conducted according to the process guidelines that dispute resolution professionals had advocated since the early 1980s. In Harter's view, Coglianese unfairly condemned a procedure whose outcomes differ significantly depending on how well it is implemented.

Harter also questioned Coglianese's evaluation criteria as well as his methods. Though saving time and reducing litigation are valid goals, other goals may be even more important: making it possible for more stakeholders to participate more directly in the policy making process; taking advantage of stakeholder's knowledge and experience to create more effective rules that meet stakeholder interests; and doing a better job of taking scientific and technical information into account.

Negotiated rulemaking is in its "adolescence," Harter argued, and will continue to evolve as agencies and interest groups gain experience and confidence in using it to reach consensus on public policy issues.

Coglianese and Harter are both sophisticated and skilled evaluators of consensus building efforts. Their sharp disagreement raises several issues relevant to the evaluation not only of negotiated rulemaking, but of public consensus building in general.

Evaluating what?

First, what should we be trying to evaluate? There are few unambiguous indicators of a "good" process or a "good" outcome. Process management frequently requires convenors, participants and neutrals to make procedural trade-offs between in-depth exploration of options and timely decision-making.

At first glance, evaluating outcomes would seem to be easier. Two basic criteria - fairness and efficiency - are widely accepted as valid measures of success, at least at a conceptual level. But evaluating the "fairness" of outcomes immediately raises the questions: according to whom and as compared to what?

If stakeholders, neutrals and disinterested evaluators don't all agree, it's probably because they are using different standards of fairness, such as accord with precedent, scientific or technical merit and distributive justice. How should these competing standards be weighed? There is no easy answer.

Evaluating the efficiency of outcomes isn't any easier. Let's say that stakeholders judge the outcome to be "efficient" - in the sense that all stakeholders believe that they could not have gotten more without making at least one other stakeholder worse-off. We still do not know whether these stakeholders could have created more joint gains if they had more information or were able to work together more effectively.

Perhaps focusing on "real world" comparisons will move us forward. Even if the consensus building process was not unambiguously fair or efficient in some absolute sense, maybe it was fairer or more efficient than the stakeholders' next best alternative. But as the Coglianese-Harter debate illustrates, the answer "as compared to what?" is rarely clear-cut, even in settings such as regulatory negotiations where there are well-established decision-making procedures.

Given the thicket of conceptual and methodological problems as well as the difficulty of defending any one set of evaluation criteria, it is no surprise that there is no agreement on how to evaluate public dispute resolution or consensus building efforts.

Nevertheless, some guidelines are worth noting. First, evaluators should not only lay out their evaluative criteria, but also explain why they have chosen those particular criteria. For example, if evaluators choose to assess the time and costs it takes to reach agreement, they should also examine other, less quantifiable costs and benefits that have been claimed for the process (such as the impact of relations and the level of organizational learning) and explain why they are not also examining them.

Second, evaluators need to acknowledge the imperfections of the methods they use. Whether their primary method is participant interviews, review of written documentation or statistical analysis of outcomes, evaluators must highlight the limitations of the methods they have chosen.

For example, it is not enough to say that all the process participants were evaluated using a standard interview questionnaire. Differences in the level of participants' involvement in the process, their recollection of events and their satisfaction with the process and the outcome may all color their responses to questionnaires and should be reviewed in the evaluators' presentation of the methods selected.

On a more positive note, the wide range of legitimate evaluative criteria presents a tremendous opportunity - as well as a formidable challenge - for would-be evaluators. To this writer's knowledge, there not yet been a peer-reviewed, published evaluation of a set of public dispute resolution cases using:

• Multiple process and outcome criteria;

• Qualitative and quantitative indicators and methods well-tailored to those criteria;

• A well-chosen control group of cases from the same arena that were resolved using traditional administrative, political and/or judicial methods

The field would benefit greatly from such studies. Practitioners and evaluators should work together to assemble their findings in a way that could improve both the theory and practice of consensus building in the public sector.

If such studies already exist, Consensus would like to know about them. If you have authored or know of one, please let us know.

Biography

The views expressed by authors are their own and do not necessarily reflect the views of Resourceful Internet Solutions, Inc., Mediate.com or of reviewing editors.