Culturally respectful evaluation
December 2004, Vol. 9, No. 3
ISSN 1540 5273
Daniel A. McDonald, Pamela B. C. Kutara, Lucinda S. Richmond, Sherry C. Betts
Abstract
Traditionally, researchers have used the term “cultural competence” when referring to the expertise needed when working with diverse populations. We believe it is important for practitioners to consider how competent evaluators can apply the ideals of cultural competence and in doing so, to view culture more broadly, including social and economic status, literacy levels, primary language, local political climate, and youth culture. This article highlights processes that evaluators can employ to successfully conduct respectful evaluations using a variety of strategies gleaned from our experiences. While the article describes specific situations to illustrate our conception of cultural respect, it is the authors’ hope that readers will be able to identify ways in which these strategies can apply in their own settings. The common thread running through all the examples is the development of a respectful relationship between evaluators and members of the group being evaluated.
Keywords: evaluation, cultural respect, diverse populations, cultural competence
Introduction
Traditionally, researchers have used the term “cultural competence” when referring to the expertise needed to work with diverse populations. As such, cultural competence has been defined in many different ways. Some define cultural competence as “a set of congruent behaviors, attitudes, and policies that come together in a system, agency, or among professionals and enables that system, agency, or those professionals to work effectively in cross-cultural situations” (Cross, Bazron, Dennis, and Isaacs 1989; Isaacs and Benjamin 1991). Others have operationally defined cultural competence as “the integration and transformation of knowledge about individuals and groups of people into specific standards, policies, practices, and attitudes used in appropriate cultural settings to increase the quality of services; thereby producing better outcomes” (Davis 1997). SenGupta, Hopson, and Thompson-Robinson (2004) have defined cultural competence in evaluation as “a systematic, responsive inquiry that is actively cognizant, understanding, and appreciative of the cultural context in which the evaluation takes place” (p. 13).
From our experience working with diverse cultures, we believe that it is important for practitioners to consider how evaluators can apply the ideals set forth in the definitions of cultural competence. Furthermore, there is a need in the field of evaluation to stimulate discourse and provide examples of what it means in practical terms to conduct culturally competent evaluations (SenGupta et al. 2004). Therefore, we have started thinking about the work we do more pragmatically within the context of cultural respect. Our conceptualization of cultural respect incorporates the notions of cultural knowledge, cultural sensitivity, and cultural awareness. However, cultural respect also involves the development of cross-cultural capacity through mutual understanding and appreciation of the different cultures in which we work. Simply put, culturally respectful methodologies consist of programming, evaluation, and research that are comfortable for both the members of the community and for the researchers.
With that in mind, we have found it useful to think about culture more broadly as we conduct our evaluation research. We continue to think about race and/or ethnicity, but we also consider contexts such as social and economic status, literacy levels, primary language, local political climate, and youth culture. Accordingly, the purpose of this paper is to highlight processes that evaluators can employ and roles that they can play to successfully conduct respectful evaluations. As we present specific examples from our own experiences to illustrate our points, we hope readers will recognize that the lessons learned can be applied to their own work in research and evaluation. Although these cases cover a broad spectrum of diverse populations, there are many commonalities in strategies, not the least of which is the development of a continuous feedback loop involving stakeholders and the evaluator. The article is divided into the following four areas:
- consideration of youth culture in the design and implementation of programs
- challenges presented by low literacy/low income situations in Hawaii
- fostering culturally respectful data collection among Native American populations and
- navigating the labyrinth of local politics when reporting results in a sub-Saharan African country
Engaging stakeholders and community
The inclusion of stakeholders and community members in the process of the evaluation is a demonstration of respectful evaluation, and it is essential from the beginning of the project. Involving stakeholders in the process, such as developing survey questions and determining an appropriate methodology for data collection, is critical to the success of an evaluation or the community program being assessed. The term “stakeholders” often refers to program directors or staff members, representatives of the funding institution, and program participants. However, when planning a program evaluation among diverse populations, it is essential to work with a broader coalition of stakeholders. Having broad representation helps improve the relevance of the evaluation and enables the evaluator to be more culturally respectful each step of the way.
Culturally respectful evaluation in program design within the youth culture
Discussions of cultural competence have traditionally focused on ethnicity and/or race. More recently, some have begun to expand the concept of culture to include additional components such as sexual orientation, religion, and socioeconomic status (Messina 1994). However, one element that is rarely discussed is the idea that people who occupy certain age groups share a unique culture. Specifically, the notion of “youth culture” or “teen culture” is oftentimes ignored when conducting research and/or evaluation with youth. In our work with youth development projects we have found that to truly conduct culturally respectful research, careful attention must be paid to the age group(s) of the population. When working with youth, evaluators must make a concerted effort to understand and respect youth culture. For our purposes, youth culture is defined very broadly as the culture associated with a generational unit or cohort (e.g., Generation X, Generation Y). As such, youth culture includes shared values, beliefs, and experiences and their associated patterns of behavior. It is important to recognize that youth culture is constantly evolving and that youth culture includes various subcultures that are also continually in flux. Even though there is a great deal of variation within youth culture, there are some basic lessons we have learned from our experiences evaluating youth programs.
Talking with youth
One of the most important ways to gain respect for youth culture is to simply spend time with young people. Ask them questions. Explore what is important to them. Try to find out what is going on in their lives. Immerse yourself in their culture by reading their magazines, listening to their music, and watching their movies and television shows. Spending time with youth may occur in formal or informal settings. A formal setting such a focus group, provides the researchers with a planned agenda and structured time to converse with youth about specific topics in detail. In a dating violence prevention project in Arizona, focus groups of youth were conducted before the implementation of the dating violence prevention curriculum. During the focus groups, youth were asked specific questions about dating and relationships. What terms do they use for “dating?” What types of activities are involved when they “go out?” And, how often does violence occur in dating relationships in their community?
Local adults can contribute to the understanding of youth culture. Gaining respect for the local youth culture does not always occur through interactions with youth. In the same dating violence prevention project, key informant interviews were conducted with school personnel. Through these interviews, we discovered important information about violence among youth in a particular community. Specifically, we found out that girl-on-girl violence was quite common, much more common than violence between boys. Girls in this community tend to keep grudges against each other, sometimes for many years. These grudges frequently lead to physical fights. This information was extremely useful in implementing a violence prevention program that was tailored to the needs of the youth culture within the community.
Culturally respectful survey instruments among low-income/low-literacy populations in Hawaii
Evaluation efforts often focus primarily on white, middle-class participants or secondary data from middle-class agency staff regarding their clientele. Limited efforts have attempted to conduct evaluation with low-income, low-literacy and/or non-native English speaking populations. One plausible explanation for this situation is that the development of survey instruments to be used with diverse populations may be challenging, time consuming, and inconvenient. This situation is especially prominent in Hawaii where its diverse, multi-cultural, multi-ethnic population consists of no majority ethnicities (Hawaiian/Part Hawaiian: 22.8 percent; Caucasian: 22.2 percent; Japanese: 19.9 percent; Filipino: 15.9 percent; Chinese: 4.9 percent; and the remainder consisting of over 20 different groups) (Hawaii Health Survey 2001; The State of Hawaii Data Book 2001). In addition, nearly 25 percent of Hawaii’s population speaks a language other than English at home and approximately 18 percent of the population scores at the lowest level of literacy (National Institute for Literacy ).
Not only is the construction of appropriate instruments a deterrent, but collection of the data can take time away from valuable workshop/classroom instruction. When the participants are low literate or English is a second language (or the translated native languages are not available), the staff may need to orally conduct the evaluation with individualized assistance. In these types of programs, such as EFNEP (Expanded Foods and Nutrition Education Program), which often serve participants of lower literacy levels, this is such a common occurrence that staff are specifically trained to answer or clarify questions without providing the answers.
Challenges with using existing instruments
Using existing standardized survey instruments among populations as diverse as those found in Hawaii can prove difficult. Braun and Karel (2000) found that some common English words and phrases caused confusion for some Hawaii residents, including words such as “thorough,” and “friend,” which are not used in daily conversation or are not part of the cultural norm for a service provider. In order to overcome this obstacle and obtain the same information for a nationwide comparison, the researchers collaborated with the original author of the survey instrument to provide further clarification for the respondents.
When working with lower-literacy groups where English may be the dominant language, the evaluator may be able to focus on standard evaluation safeguards such as readability, reliability, and validity of survey items. However, in a recent non-credit community class, the eight participants conversed in English, but upon further exploration it was revealed that the native languages for three of the adults were Chinese, Filipino, and Korean. When working with non-traditional audiences, a balance between instruction and relationship building is one of the most important initial steps (Bairstow, Berry, and Driscoll 2002). In this situation, a respectful approach entailed establishing a non-threatening environment and encouraging the participants to stop the lessons at any point for clarification. Additionally, the instructor also tried to be more alert to nonverbal clues. Demonstration or interactive activities increased, while lecturing was limited. As participants understood the concepts they became more involved in sharing their own insights.
Strategies for piloting internally constructed survey instruments
The limitations of using an existing evaluation instrument are magnified in a multicultural, diversely literate setting. Often it is necessary to develop internal evaluation instruments to assess specific characteristics of a community program. In some communities it is difficult to predict in advance the makeup of the group prior to the survey administration. Therefore, as in the case with an evaluation of a community program in Hawaii, instruments were developed using simple, upper elementary school level English with appropriate graphics inserted, ample white space, and interesting layouts with a variety of components such as Likert scales, connecting lines, checklists, and fill-in-the-blanks.
One effective strategy recommended by Taylor-Powell (2001) is to ask colleagues of the cultures being studied, and stakeholders, such as agency personnel, to review the instrument for prejudices and cultural assumptions that may inadvertently be present and could be interpreted as disrespectful, including suitability of language and appropriateness of survey items for the target audience. Prior to any community pilot, available students and clerical staff can read and complete the initial instruments as if they were the intended survey respondents to provide feedback on layout, spacing, grammar, spelling, abbreviations, and acronyms.
Valuable feedback can be obtained by including stakeholders and agency personnel from the beginning of a new project. They can also assist with the development of the evaluation instruments by observing the participants throughout the process. Upon the conclusion of a workshop series, the staff of a low-income project met with the evaluator to discuss the new project. They suggested several changes to a new pre-post assessment form. It was felt that “resources” was a big word and could be replace with “things;” “household” could be substituted with “family;” and more pictures would simplify or clarify the words. For this particular combination of a multi-cultural audience (Samoan, Chuukese, Laotian, Chinese, and English native language speakers), input from the participants was greatly appreciated.
Several pilots may be necessary as illustrated in the following example. In early trials of a new survey instrument, many respondents in one group asked for clarification concerning a particular phrase used in the instrument, whereas another group, of similar makeup, had very few concerns about the same phrase. This example demonstrated the importance of obtaining feedback from various groups on all aspects of the survey instrument including introductory comments, instructions or directions, and what to do with the completed questionnaire (Earthman, Richmond, Peterson, Marczak, and Betts 1999) .
Culturally respectful data collection among Native American populations in the Southwest
As with other communities, vast differences of opinion may exist in Native American communities over beliefs, priorities, and values (Duran and Duran 1999). It is helpful for the evaluator working within Native American communities to recognize the various models that delineate old and new ways of life within Native American cultures. (See Stubben 2001 for a review of the models.)
It would be erroneous to assume a homogeneous adherence to traditions among Native Americans, even concerning those activities that an outsider might consider a social norm. In preparing questions for a community assessment to be used on a reservation located in the Southwestern United States, one item asked whether or not a certain tradition was practiced by the respondent’s family. A member of the committee constructing the survey, a Native American who grew up on the reservation, was not familiar with the tradition referred to in the question and cautioned that others might have difficulty understanding the question as well. Clearly, agreement does not always exist on what constitutes a certain “culture,” even within a culture.
To design and implement a culturally respectful evaluation, given the lack of consensus on what constitutes the culture, Fisher and Ball (2002) recommend the Tribal Participatory Research model, which consists of the following four components:
- tribal oversight
- use of a facilitator (to act as an intermediary between staff and oversight committee)
- training and employing community members as project staff
- developing culturally specific interventions and assessments.
By being as inclusive as possible in the evaluation process, an evaluator can be respectful of the community and its multiple cultural perspectives. It is advisable for evaluators to obtain broad support from various entities including both formal and informal community leaders (Beauvais & Trimble 1992). Respecting and valuing the input of many will help reduce suspicion and increase trust (Duran and Duran 1999).
Evaluation by committee
The Cooperative Extension System, with Extension educators located within counties and communities, lends itself nicely to establishing local oversight and providing intermediaries. Such a mechanism can prove useful in the implementation of a simple evaluation. For instance, a straightforward set of questions was devised by an outside evaluator to assess the facilitator, materials, and location of a community program from “excellent” to “poor.” Community stakeholders and a local liaison informed the evaluator that participants in the program would not report anything negative about the program or the facilitator and that it was futile to ask the questions as they were posed. Directness and criticism are not part of the communication repertoire of most Native Americans (Fleming 1992; Stubben 2001). Consequently, the survey was scrapped and replaced with a simple summary of what participants felt were the “take home” messages from the session.
Hiring from within
Employing community members with cultural backgrounds similar to those in the community to recruit participants, conduct interviews, or facilitate focus groups is an effective approach to ensuring culturally respectful data collection (Fisher & Ball 2002; Stubben 2001). An example of this is the design and implementation of a community assessment on the reservation previously mentioned. Going door-to-door to recruit and interview residents was not only culturally appropriate, but pragmatically necessary. Homes in this rural community are located in isolated areas, often with no telephone. Mail is retrieved only periodically from the Post Office box located many miles from recipients’ residences. Some letters sit unopened for a long time before a family member or friend reads them. Therefore, Native American members of the community assessment committee were hired to recruit and interview participants.
Stakeholder involvement in the formulation of response sets. As with the formulation of survey questions, care must be taken when deciding the appropriate response set to those questions. Evaluators may find it necessary to relax the requirements of rigor in evaluating a program in favor of cultural appropriateness. A case in point involves the design of a community assessment. Local committee members insisted that response sets that provided a series of gradations from “strongly agree” to “strongly disagree,” would not translate correctly and thus would not be understood by respondents. Committee members anticipated that interviewers would receive stories in response to questions rather than a direct answer. Since the budget did not support the input and analysis of numerous open-ended responses, it was agreed that the interviewers, in such cases, would interpret the stories as either a “yes” or a “no” response.
Culturally respectful data collection and reporting in sub-Saharan Africa
Getting embroiled in local and regional politics and disputes can put the evaluator in a very difficult position. This is especially true when one is working with not only another cultural group, but in a country different from one’s own. A committee of educators from schools, district, and ministry offices, worked with the evaluator to adapt a 160-item, multiple-choice, U.S. survey, for use with secondary students in a sub-Saharan African country. The official language was English, but the spelling, usage, and vernacular differed from one country to the other, in addition to the issues, problems, supports, and opportunities available to young people. The survey adaptation and approval process took close to two years and was reviewed at local, district, and national levels resulting in an official letter of approval and endorsement by the Minister of Culture and Education. The evaluator thought that cultural relevance had been addressed in a respectful manner and that data collection and reporting would follow smoothly. However, a couple of bumps in the road were encountered, literally and figuratively.
Local politics and data collection
After a cadre of district education advisors and the evaluator trained teachers in the administration of the survey at nine secondary schools in four different communities, some interesting findings revealed unexpected implications for cultural relevancy. One noteworthy surprise was the poor quality of data from a secondary school in which students gave inconsistent responses that we knew were not possible. As an example, one secondary student indicated an age of 12 years, studying at “A” level, drinking local beer daily, getting high grades, and being homeless. Inquiry of some teachers at the school uncovered a dispute between the headmaster and district education adviser that resulted in the headmaster undermining the data collection process. This led to some very delicate negotiations between the evaluator and the local community of educators about the usefulness of the data. However, the problem was resolved by focusing on the goal of hearing the voices of young people, which was commonly shared by the headmaster, local community, and the district. Consequently, it was agreed that the data did not represent the real concerns and issues of the community’s youth and should not be reported at all.
Local politics and reporting result
When all other data were analyzed and a draft report was written by the evaluator, a copy was sent to the in-country project coordinator to read for any “red flags” or inadvertent culturally insensitive implications. The only comments by the project coordinator requested changes in the data reported, not the narrative. This dilemma was resolved after many communications, not by changing the data, but by reporting it differently. In the U.S. it is common to examine school data by gender and by grade level, however, in this sub-Saharan African country different cross tabulations were appropriate, using more and varied categories normally not considered by Western evaluators.
Conclusion
In this article we have provided a variety of practical examples for ways that evaluators can become more culturally competent by applying culturally respectful strategies in their work. While we used specific situations from our own experiences to illustrate our points, we hope that readers are able to identify ways in which the strategies offered can be put into action in their own settings. An examination of our approaches and suggestions reveals that the common thread running through all of our examples is the development of a relationship between evaluators and members of the cultural group studied. Furthermore, you will also note an openness to what constitutes a “culture” and what constitutes “good” evaluation. Taking the time to develop trust, to listen to the advice of stakeholders, and to interact with positive regard and appropriate deference, will help the evaluator gain competence and design, implement, and report culturally respectful evaluations.
Tips for conducting respectful evaluations
- Work with a broad coalition of stakeholders by being as inclusive as possible in the evaluation process.
- Obtain support from formal and informal leaders.
- Build and maintain a trusting relationship with stakeholders.
- Think broadly about what constitutes a “culture” such as age groups and literacy levels.
- Employ standard evaluation safeguards such as readability, reliability, and validity.
- Ask colleagues or agency personnel who are members of the culture to review materials for prejudice and cultural assumptions.
- Identify common goals for the evaluation that are shared by researchers and stakeholders.
- Be flexible.
Authors
Daniel A. McDonald
Research Specialist
University of Arizona
Division of Family Studies and Human Development
P.O. Box 210033
Tucson, AZ 85721-0033
Phone: 520-626-8496
Fax: 520-621-9445
mcdonald@ag.arizona.edu
Pamela B. C. Kutara
Extension Educator
University of Hawaii at Manoa
2515 Campus Road, #201E
Honolulu, HI 96822
Phone: 808-956-7212
Fax: 808-956-2241
kutara@hawaii.edu
Lucinda S. Richmond
Research Specialist
University of Arizona
Division of Family Studies and Human Development
P.O. Box 210033
Tucson, AZ 85721-0033
Phone: 520-626-5382
Fax: 520-621-9445
lucindas@ag.arizona.edu
Sherry C. Betts
Extension Specialist and Professor
University of Arizona
School of Family and Consumer Sciences
P.O. Box 210033
Tucson, AZ 85721-0033
Phone: 520-621-3399
Fax: 520-621-9445
sbetts@ag.arizona.edu
References
Bairstow, R., H. Berry, and D.M. Driscoll. 2002. Tips for Teaching Non-Traditional Audiences. Journal of Extension. On-line: http://www.joe.org/joe/2002december/tt1.shtml
Beauvais, F., and J.E. Trimble. 1992. The role of the researcher in evaluating American Indian alcohol and other drug abuse prevention programs. In M.A. Orlandi (ed.) Cultural Competence for Evaluators: A Guide for Alcohol and Other Drug Abuse Prevention Practitioners Working with Ethnic/Racial Communities. U.S. Department of Health and Human Services.
Braun, K.L., and H.S. Karel. 2000. Hawaii adapts a standardized assessment measure to fit its unique community. Diversity Currents.
Cross, T., B. Bazron, K. Dennis, and M. Issacs. 1989. Towards a culturally competent system of care (Vol. I). Washington, D.C.: Georgetown University Child Development Center, CASSP Technical Assistance Center.
Davis, K. (1997). Exploring the intersection between cultural competency and managed behavioral health care policy: Implications for state and county mental health agencies.Alexandria, Virginia: National Technical Assistance Center for Stage Mental Health Planning.
Duran, B.M., and E.F. Duran. 1999. Assessment, program planning, and evaluation in Indian Country: Toward a postcolonial practice. In R. M. Huff and M. V. Kline (eds.) Promoting Health in Multicultural Populations: A Handbook for Practitioners. Thousand Oaks, CA: Sage Publications, Inc.
Earthman, E., L.S. Richmond, D.J. Peterson, M.S. Marczak, and S.C. Betts. 1999. Adapting evaluation measures for ‘hard to reach’ audiences. CYFERnet. On-line:
http://ag.arizona.edu/fcs/cyfernet/evaluation/adapeval.pdf. (Retrieved November 23, 2004)
Fisher, P.A., and T.J. Ball. 2002. The Indian family wellness project: An application of the tribal participatory research model. Prevention Science 3: 235-240.
Fleming, C.M. 1992. American Indians and Alaska Natives: Changing Societies Past and Present. In M.A. Orlandi (ed.) Cultural Competence for Evaluators: A Guide for Alcohol and Other Drug Abuse Prevention Practitioners Working with Ethnic/Racial Communities. U.S. Department of Health and Human Services.
Hawaii Health Survey 2001. Gender, age, ethnicity, and poverty by county — Population of Hawaii. On-line:
http://www.state.hi.us/doh/stats/hhs/hhs01.html. (Retrieved August 2003.)
Isaacs, M., and M. Benjamin. 1991. Towards a culturally competent system of care (Vol. 2): Programs which utilize culturally competent principles. Washington, DC: Georgetown University Child Development Center, CASSP Technical Assistance Center.
Messina, S.A. 1994. A youth leader’s guide to building cultural competence. Washington, DC: Advocates for Youth.
National Institute for Literacy. On-line: http://www.nifl.gov/reders/reder.htm (Retrieved November, 2004).
SenGupta, A., R. Hopson, and M. Thompson-Robinson. 2004. Cultural competence in evaluation: An overview. New Directions for Evaluation, 102: 5-19.
State of Hawaii Data Book, The 2001. Language spoken at home by persons five years old and over, by age: 2000. On-line: http://www.hawaii.gov/dbedt/db01/ (Retrieved August 2003.)
State of Hawaii. 2000. Population by major race categories alone or in combination for the State of Hawaii. On-line: http://www.hawaii.gov/dbedt/census2k/hsdc2000-2.pdf (Retrieved August 2003.)
Stubben, J.D. 2001. Working with and conducting research among American Indian families. American Behavioral Scientist, 44: 1466-1481.
Taylor-Powell, E. 2001. Is translating the questionnaire good enough? Hear It From the Board. On-line: http://danr.ucop.edu/eee-aea. (Retrieved March 2003.)
Back to table of contents ->https://www.theforumjournal.org/2017/09/03/december-2004-vol-9-no-3/