Measuring the impact of youth development programs: A national on-line youth life skills evaluation system

October 2005, Vol. 10, No. 2
ISSN 1540 5273

Claudia C. Mincemoyer, Ph.D., The Pennsylvania State University,
Daniel F. Perkins, Ph.D., The Pennsylvania State University

Abstract

The National On-line Youth Life Skills Evaluation System allows youth development professionals to effectively design and determine the impact of their educational programs. The evaluation system is intended to disseminate quantitative evaluation tools through a high-quality, cost-effective, user-friendly, web-based system that links research, program design, and evaluation. The evaluation tools were developed using synthesis and meta-analysis of existing research to determine the skill set for each life skill studied. By establishing a national system, Cooperative Extension and other youth-serving organizations will be able to aggregate data at the county, state, and national levels that can show the impact of their youth development programs to various stakeholders. The system was developed jointly by Penn State University and Purdue University with a grant from the American Distance Education Consortium.

Keywords: life skills, evaluation, program evaluation, youth development

Introduction

In the past several years, the need for accountability in youth development programs has increased dramatically. The Government Performance and Results Act (GPRA) enacted by Congress requires all governmental entities (including Cooperative Extension) to be accountable for their use of federal funds. State and county governments are also asking for more impact data showing that the funds invested in youth development programs are beneficial. Private funding sources also require that impact evaluation strategies be included in any proposal. In short, they are all asking, “What is the impact on youth and families for the dollars invested?”

Community programs designed to promote youth development attempt to provide youth with opportunities to develop personal (life skills) and social assets through participation in a variety of projects and activities. Many of these programs claim to foster life skills and social assets; however, few have been scientifically evaluated (Eccles and Gootman 2002) and few instruments are available to assist youth development professionals in assessing youths’ personal and social assets. The importance of such skills/competencies in adolescence, combined with the fact that these skills can be taught and practiced, provides strong rationale for the development of assessment measures. As noted in Community Programs to Promote Youth Development, published by the National Academy of Sciences (Eccles and Gootman 2002), this type of applied, scientifically driven research in non-formal educational environments, such as 4-H, is needed to determine whether these environments and the curricula designed to enhance life skills are effective and successful.

Therefore, it is essential that youth and family educators are able to measure and verbalize the impact of their programs and/or curricula on youth involved in their organizations. However, most youth and family educators have no formal training in program evaluation, and few instruments are available to assist youth development professionals in assessing life skills. If evaluations occur at the program level, most focus on knowledge gained, attitudes changed, or increased awareness.

Because of the need for impact evaluation data and the limited skills, time, and resources of youth and family educators, an evaluation system to assist with the collection and analysis of impact data was developed by applied researchers at Penn State and Purdue Universities. Because of the passing of the co-principal investigator at Purdue University, the co-principal investigators from Penn State University assumed the leadership for the development and pilot-testing of the Web-based evaluation system. The evaluation system is designed so that youth and family educators can collect valid and reliable impact data at the local level to be shared with community decision makers, legislators, parents, and internal stakeholders.

Why life skills are important

In the 4-H youth development program, educational activities and programs center on projects that teach youth skills and content knowledge in a broad range of curriculum areas. 4-H curricula, like many other youth-serving organizations’ programs, are also designed to teach life skills. Common life skills across most of the youth curricula used in 4-H programming are those that young people need to make decisions, solve problems, communicate effectively, think critically, and set goals.

When designing programs for youth, it is important to understand how youth grow and change and identify those life skills that contribute to their healthy development. Life skills comprise the majority of competencies within the personal and social assets categories identified in the National Research Council and Institute of Medicine book on youth programs (Eccles and Gootman 2002). According to Eccles and Gootman (2002), youth need opportunities and support from their communities to develop these important personal and social assets to become competent, contributing adult members of society. Programs designed to support young people with positive experiences, environments, and people assist young people in the acquisition of these skills (Perkins, Borden, et al. 2003).

Through curricula, program activities, and positive relationships between adults and youth, 4-H and other youth development programs provide opportunities to gain and refine life skills that comprise personal or social assets. Barkman and Machtmes (2000) developed a life skills model that clearly links life skills development within the 4-H program.

Indeed, structured environments that help guide children’s development have been a major focus of public attention, policy, and research over the last decade (Roth and Brooks-Gunn 2003, Eccles and Gootman 2002, Tolman and Pittman 2002). However, youths’ needs grow more complex as they approach adolescence, and public commitment to supporting their development tends to weaken as they grow older.

Programs that have documented effectiveness are needed within the youth development field. The synthesis of research summarized by Eccles and Gootman (2002) and the recent findings by Lerner and colleagues (Lerner et al. 2005) clearly links life skills to the outcome of youth becoming contributing members of society as adults. Thus, while longitudinal studies are still needed to advance our understanding of how the process works, evaluation of any youth program’s ability to improve life skills over the short-term is an appropriate evaluative goal.

The on-line National Life Skills Evaluation System

Components of the systems

The National Life Skills Evaluation System hosts instruments for evaluating the life skills of decision making, critical thinking, problem solving, goal setting, communication, and leadership. Each evaluation instrument was developed by conducting a comprehensive review of research from social sciences and education journals and books. The emphasis of the review was the identification of specific sub-skills needed to successfully master a certain life skill. Any sub-skill contributing to the development of a certain life skill was placed into a matrix. Those sub-skills that were noted by more than 50 percent of the research articles were included as comprising the sub-skill set for a specific life skill. For example, the sub-skill set for decision-making is (1) define the problem, (2) generate alternatives, (3) check risks and consequences of choices, (4) select an alternative, and (5) evaluate the decision (Mincemoyer and Perkins 2003).

Once the sub-skill set was identified, survey questions were developed that addressed each of these sub-skills to comprise an evaluation instrument for the specific life skill. The evaluation instruments for each life skill (i.e., decision making, critical thinking, communication, problem solving, and goal setting) were pilot tested and item analysis was completed to ensure reliability and validity. After the analysis, revisions to specific items were made to increase reliability and validity of the construct.

The authors then developed a combined life skill evaluation tool that used selected items (3-5) from each of the following life skill areas: decision making, problem solving, critical thinking, communication, and goal setting. This instrument was pilot tested with more than 200 youth (Table 1). The life skills (i.e., decision making, critical thinking, communication, problem solving, and goal setting) selected for this overall measure were thought to be the most commonly identified life skills emphasized in 4-H youth development curricula and program activities.

Table 1. Reliability of Skills for Everyday Living Survey

*The Cronbach Alpha for this scale is within acceptable range, albeit at the lower end of the range.

The National Life Skills Evaluation System is an on-line resource that provides users with downloadable copies of each of the five life skills surveys — one for each skill area (i.e., decision making, critical thinking, communication, problem solving, and goal setting), and the sixth survey is a combination of the five previous surveys. Abstracts of the research studies that were reviewed and the sub-skill matrix used to develop the “sub-skill set” are also on the web site. In addition, there are educational activities and curriculum references that can be used to supplement or enhance curricula within youth programs to increase intentionality of teaching and practicing the sub-skill set.

Administering the evaluations with youth

The surveys are completed by youth individually either in written format or on-line. The pretest is administered at the start of the program year or the beginning of the program. A posttest is administered at the last program meeting of the year or at the end of the program. At minimum, a program is defined as meeting at least five times. This definition was put in place to guard against inappropriate assessments of one-shot programs. Attendance is recorded by the adult leader, and only those youth who completed more than half (and at least five) of the sessions of the program complete a posttest.

Data analysis

The evaluation system provides data analysis in real time for the user. Summary reports provided immediately after data entry include frequencies for demographics (i.e., age, gender, and ethnicity), club type, project content area, audience type; matched paired t-test results from the pre- and post surveys, and matched paired t-test results from the post-and post-post surveys (see Table 2 for sample summary report).

Table 2. Sample t-test summary report

*Youth’s decision making and communication ability was significantly greater after the program (p >.01) than before the program. Participation in this program may be associated with this increase.

** Youth’s critical thinking ability did not change significantly after the program when compared to before the program. Thus, there is no immediate effect; however, there may be a sleeper effect. A sleeper effect will appear at a later time period and could be positive or negative. Thus, whenever possible, it is important to conduct a post-posttest approximately three to nine months following the program.

***Youth’s goal-setting ability decreased after the program when compared to before the program. You may want to closely examine the sub-skills to determine areas where youth decreased the most. As youth increase in age, there is a normal decrease associated with developmental skills. It is important to conduct a post-posttest approximately three to nine months following the program.

In addition, because some youth professionals are not skilled in interpreting data analysis, a text report is generated with each data table depending on the condition of the difference. For example, if there was a significant increase in the practice of a particular life skill, the following would be included in the report generated for the user: “Youth’s (skill, i.e., decision making or critical thinking) ability was significantly greater after the program than before the program. Participation in this program may be associated with this increase.”

During system training and in subsequent communication with extension educators using the system, the authors emphasize the limitations of the system. Program participation and increased life skills are not a causal relationship. Rather, program participation may be associated with the increase in life skill development. However, with similar findings across a variety of youth programs, increased confidence in the relationship is possible.

Moreover, when a decrease in life skill development from pretest to posttest is found or when no change is found, the system discusses the dip associated with normal adolescent development. There is strong evidence from research about the negative psychological, social, and academic changes associated with adolescents making the transition to junior high school (Simmons and Blyth 1987). Thus, the authors encourage youth program staff to conduct a post-posttest approximately three to nine months following the program to see if the decrease continues or if there is a change in direction. This is known as a sleeper effect.

Pilot implementation and plans for the future

Currently, the on-line evaluation system is being piloted in Pennsylvania. 4-H Youth professionals are required to use the Skills for Everyday Living Survey with at least 10 percent of their 4-H club membership. The educators are using the data in their yearly accomplishment reports, requests to funders, reports to stakeholders, and with volunteer leaders for program improvement purposes. All the current life skills measures are designed for youth ages 12-18. However, a large percentage of youth enrolled in the 4-H youth development program are between the ages of 8-12. The authors are currently working on a measure for younger youth (ages 8-12) that parallels the Skills for Everyday Living survey.

The evaluation tools, research abstracts and teaching resources are available free for educational uses. These resources and information about the system are available at
http://www.humanserviceresearch.com/youthlifeskillsevaluation/

Authors

Claudia C. Mincemoyer, Ph.D.
Assistant Professor
The Pennsylvania State University
323 Ag. Admin. Bldg.
University Park, PA 16802
814-863-7851
fax: 814-863-4753
cxm324@psu.edu

Daniel F. Perkins, Ph.D.
Associate Professor
The Pennsylvania University
323 Ag. Admin. Bldg.
University Park, PA 16802
814-865-6988
fax: 814-863-4753
dfp101@psu.edu

References

Barkman, S.J., and K.L. Machtmes. 2000. Measuring youth development programs: The four-fold youth development model. Community Youth Development Journal 1: 4.

Eccles, J., and J. Gootman. 2002. Community programs to promote youth development.Committee on Community-Level Programs for Youth. Board on Children, Youth, and Families, Commission on Behavioral and Social Sciences Education, National Research Council and Institute of Medicine. Washington, DC: Sage Publications.

Lerner, R.M., J.V. Lerner, J.B. Almerigi, C. Theokas, E. Phelps, S. Gestsdottir, S. Naudeau, H. Jelicic, A. Alberts, L. Ma, L.M. Smith, D.L. Bobek, D. Richman-Raphael, I. Simpson, E.D. Christiansen, and A. von Eye. 2005. Positive youth development, participation in community youth development programs, and community contributions of fifth-grade adolescents: Findings from the first wave of the 4-H study of positive youth development. Journal of Early Adolescence 25:17-71.

Mincemoyer, C., and D. Perkins. 2003. Assessing decision-making skills of youth. The Forum for Family and Consumer Issues 8: 1.
http://www.ces.ncsu.edu/depts/fcs/pub/8(1)/mincemoyer.html

Perkins, D.F., L.M. Borden, J.G. Keith, T. Hoppe-Rooney, and F.A. Villarruel. 2003. Community youth development: A partnership for creating a positive world. In F. A. Villarruel, D.F. Perkins, L.M. Borden, and J.G. Keith (eds.), Community youth development: Practice, policy, and research 1-23. Thousand Oaks, California: Sage.

Roth, J., and J. Brooks-Gunn. 2003. What exactly is a youth development program? Answers from Research and Practice. Applied Developmental Science 7: 94-111.

Simmons, R.G., and D.A. Blyth. 1987. Moving into adolescence: The impact of pubertal change and school context. New York: Aldine de Gruyter.

Tolman, J., and K. Pittman. 2002. Toward a common vision: Naming and framingthe developmental imperative. Washington, DC: Academy for Educational Development.

Back to table of contents ->https://www.theforumjournal.org/2017/09/03/october-2005-vol-10-no-2/