How might the quality of an action research initiative be evaluated?
— begin paste —
We have linked our five validity criteria (outcome, process, democratic, catalytic, and dialogic) to the goals of action research. Most traditions of action research agree on the following goals: (a) the generation of new knowledge, (b) the achievement of action-oriented outcomes, (c) the education of both researcher and participants, (d) results that are relevant to the local setting, and (e) a sound and appropriate research methodology. Based on these goals, we have identified indicators of quality for action research studies. In Table 4.1 we show how these goals are linked to validity criteria. [p. 67]
| Goals of Action Research | Quality / Validity / Criteria |
| 1. The generation of new knowledge | Dialogic and process validity |
| 2. The achievement of action-oriented outcomes | Outcome validity |
| 3. The education of both researcher and participants | Catalytic validity |
| 4. Results that are relevant to the local setting | Democratic validity |
| 5. A sound and appropriate research methodology | Process validity |
Outcome Validity. One test of the validity of action research is the extent to which actions occur, which leads to a resolution of the problem that led to the study. Greenwood and Levin (2006) call this criteria workability and link it to John Dewey’s notion of pragmatism. Watkins (1991) points out that “many Action Research studies abort at the stage of diagnosis of a problem or the implementation of a single solution strategy, irrespective of whether or not it resolves the presenting problem” (p. 8). Brooks and Watkins (1994) suggest skillfulness as action research’s equivalent to credibility for naturalistic inquiry or validity for positivist research. Action researchers must be competent at both research procedures and moving participants toward successful action outcomes. [pp. 67-68]
Jacobson (1998) uses the term integrity to discuss his criteria for good action research. Integrity must rest on “the quality of action which emerges from it, and the quality of data on which the action is based” (p. 130). Thus, outcome validity is synonymous with the “successful” outcome of the research project. This, of course, begs the question raised below under democratic validity, that is, successful for whom? Outcome validity also acknowledges the fact that rigorous action research, rather than simply solving a problem, forces the researcher to reframe the problem in a more complex way, often leading to a new set of questions or problems. This ongoing reframing of problems leads to the spiraling dynamic that characterizes the process of most action research over a sustained period of inquiry.
Process Validity. This asks to what extent problems are framed and solved in a manner that permits ongoing learning of the individual or system. In this sense, outcome validity is dependent on process validity in that, if the process is superficial or flawed, the outcome will reflect it. Are the “findings” a result of a series of reflective cycles that include the ongoing problematization of the practices under study? Such a process of reflection should include looping back to reexamine underlying assumptions behind problem definition (Argyris et al., 1985). Process validity must also deal with the much-debated problem of what counts as evidence to sustain assertions, as well as the quality of the relationships that are developed with participants. [p. 68]
Here, some criteria might be borrowed from naturalistic inquiry, depending on how evidence is defined. The notion of triangulation, or the inclusion of multiple perspectives, guards against viewing events in a simplistic or self-serving way. Triangulation also can refer to using a variety of methods—for example, observation and interviews—so that one is not limited to only one kind of data source. Process is not, however, limited to method. In narrative and essayist forms of inquiry, there are distinct criteria for what makes a good empirical narrative (as opposed to fiction). Connelly and Clandinin (1990) warn that “not only may one ‘fake the data’ and write a fiction but one may also use the data to tell a deception as easily as a truth” (p. 10). (For an elaboration of validity criteria for narrative research, see Connelly & Clandinin, 1990; Heikkinen, et al., 2007.) [pp. 68-69]
Democratic Validity. This refers to the extent to which research is done in collaboration with all parties who have a stake in the problem under investigation. If not done collaboratively, how are multiple perspectives and material interests taken into account in the study? For example, are teachers, nurses, social workers, or CEOs using action research to find solutions to problems that benefit them at the expense of other stakeholders? Are patients, clients, students, and community members seen as part of the insider community that undertakes this type of research, or are they viewed as outsiders by action researchers? Even when collaboration takes place, how deep does it go and how wide does it extend? While process validity depends on the inclusion of multiple voices for triangulation, democratic validity views it as an ethical and social justice issue.
Another version of democratic validity is what Cunningham (1983) calls local validity, in which the problems emerge from a particular context and in which solutions are appropriate to that context. Watkins (1991) calls this relevancy or applicability criteria for validity (i.e., how do we determine the relevance of findings to the needs of the problem context?) (p. 15). Drawing on Bronfenbrenner (1979), Tandon et al. (2001) use the term ecological validity, or the degree to which the constructs and products of the research are relevant to the participating group.
Catalytic Validity. This is “the degree to which the research process reorients, focuses, and energizes participants toward knowing reality in order to transform it” (Lather, 1986, p. 272). In the case of action research, not only the participants, but the researchers/practitioners themselves must be open to reorienting their view of reality as well as their view of their role. All involved in the research should deepen their understanding of the social reality under study and should be moved to some action to change it (or to reaffirm their support of it). The most powerful action research studies are those in which the researchers recount a spiraling change in their own and their participants’ understandings. This reinforces the importance of keeping a research journal in which action researchers can monitor their own change process and consequent changes in the dynamics of the setting. While this criteria overlaps with process and democratic validity, it highlights the transformative potential of action research, which makes it so appealing to many critical pedagogues, organization and staff developers, and change agents.
Dialogic Validity. In academic research, the “goodness” of research is monitored through a form of peer review. Research reports must pass through the process of peer review to be disseminated through academic journals. Many academic journals even provide opportunities for researchers to engage in point–counterpoint debates about research. A similar form of peer review is beginning to develop within and among action research communities. Many action research groups are forming throughout North America, as action researchers seek dialogue with peers. In addition, refereed publishing venues for action research have increased dramatically in the last decade. This is particularly important for those doing action research dissertations, since those who take academic jobs will need to publish their work. [p. 69]
Dialogic validity is similar to democratic validity but differs in that the focus is less on broad inclusion than on the validation—both during and after the study—that methods, evidence, and findings resonate with a community of practice. To promote both democratic and dialogic validity, some have insisted that action research should only be done as collaborative inquiry (Torbert, 1981; Carr & Kemmis, 1986). Others simply suggest that action researchers participate in critical and reflective dialogue with other action researchers (Martin, 1987) or work with a critical friend who is familiar with the setting and can serve as devil’s advocate for alternative explanations of research data. When the dialogic nature of practitioner inquiry is stressed, then studies can achieve what Myers (1985) calls “goodness-of-fit with the intuitions of the practitioner community, both in its definition of problems and in its findings” (p. 5).1
- 1 Bray, Lee, Smith, and Yorks (2000) provide another approach to doing collaborative research while carving out individual dimensions of the research.
All of the above validity criteria for action research are tentative and in flux. [p. 70]
— end paste —
Reference
Herr, Kathryn, and Gary L. Anderson. 2015. “Quality Criteria for Action Research: An Ongoing Conversation.” In The Action Research Dissertation, 2nd ed., 63–79. Thousand Oaks, CA: SAGE Publications Inc. https://us.sagepub.com/en-us/nam/the-action-research-dissertation/book239688.
In brief. David Ing. 