We often hear different terms used when it comes to assessing how well we are doing in terms of our strategies, programs and services. Sometimes the term evaluation is common, other times we speak about performance measurement. So what are the differences and similarities between the two? What is appropriate for your work?
Performance measurement refers to the ongoing monitoring and reporting of program accomplishments, particularly progress towards pre-established goals. It focuses on whether a program has achieved its objectives, expressed as measurable performance standards. It is typically conducted by internal program or agency staff (www.gao.gov/special.pubs/gg98026.pdf).
Performance measures may address:
Note the similarities with program evaluation in terms of the types of things that can be assessed.
The main benefit of performance measurement is that it can serve as an early warning system to management and as a vehicle for improving accountability to the public because of its ongoing nature.
A "program" may be any activity, strategy, project, function, or policy that has an identifiable purpose or set of objectives.
Evaluation is the systematic assessment of the processes and outcomes of a program with the intent of furthering its development and improvement. Evaluations are conducted periodically or ad hoc to assess how well a program is working and are typically conducted by expert evaluators external to the program or agency for purposes of remaining objective (www.gao.gov/special.pubs/gg98026.pdf).
The main benefit of evaluation is that its in-depth examination of program performance and context allows for an overall assessment of whether the program works and identification of adjustments that may improve its results.
Comparing Performance Measurement and Evaluation
Performance Measurement | Evaluation |
---|---|
|
|
Performance measures are selected as part of the planning process. Performance measures are ideally attached to each outcome, enabler and strategy in your plan to track your progress against that specific piece.
When selecting measures, ask yourself and your team:
What types of measures can I use?
Here are some common types of performance measures:
Do you have a good set of measures? Do your measures help you manage?
Use this checklist to help formulate measures:
Performance Measurement Resources
National Health Services Institute for Innovation and Improvement This website provides examples and walks you through several easy-to-use tools for performance measurement and quality improvement. Although it is heavily focused on acute care settings, the tools are transferable and good to learn from.
Transforming Performance Measurement - Rethinking the Way We Measure and Drive Organizational Success. By Dean R. Spitzer This book focuses on the social aspects of measurement, what can happen when measurement goes wrong, and tips for maximizing the value of organizational performance measurement - a must-read for those involved in implementing performance measurement.
Program Evaluation - Tips
Similar to program planning, using a logic model framework for evaluation can be very helpful for prompting questions and developing outcomes and indicators to address your evaluation questions.
For a guide to conducting program evaluation, check out this resource on Evaluating Programs from the former Health Communications Unit.
This Evaluation Plan Template (MS Word Doc) can help you think through all of the components required to develop an evaluation plan.
Types of Evaluation
Program evaluation can be categorized based on when the evaluation is being conducted and the type of information collected: formative, process, summative or outcome evaluation. Each form is described below:
1. Formative Evaluation
Formative evaluation focuses on programs that are under development. It is used in the planning stages of a program to ensure the program is developed based on stakeholders needs and that programs are using effective and appropriate resources and procedures.
Formative evaluation can include activities such as:
Source: (THCU, 2007)
2. Process Evaluation
Process evaluation focuses on programs that are already underway. It examines the procedures and tasks involved in providing a program. It seeks to answer the question, "What services are actually being delivered and to whom?"
Source: (THCU, 2007)
3. Summative Evaluation
Summative evaluation focuses on programs that are already underway or completed. It investigates the effects of the program, both intended and unintended. It seeks to answer the questions "Did the program make a difference?"(impact evaluation) and "Did the program meet its stated goals and objectives?"(outcome evaluation).
In its most rigorous form the design of an outcome evaluation can become very complex in order to rule out any other plausible explanations for the results. Outcome evaluation can assess both short term outcomes, immediate changes in individuals or participants (such as participation rates, awareness, knowledge, or behaviour) and long term outcomes (sometimes referred to as impact evaluation) which look at the larger impacts of a program on a community. An outcome evaluation can also analyze the results in relation to the costs of the program.
Summative evaluation can include the following types of analyses:
Different types of evaluations can sometimes be referred to by different names. We encourage you not to get too stuck on terminology but to describe your evaluations in a way that is understandable to you and your stakeholders.
Here are a few definitions that may help to distinguish between the different types of summative evaluation.
Impact - Evaluates the impact your program had on the participants or other stakeholders of the project. Impact evaluation goes a little further than outcome. It measures outcomes but also measures what changes occurred as a result of those outcomes.
Outcome - Evaluates what occurred as a result of your program. It determines whether you achieved the programs short-term and/or long term objectives.
Cost-benefit - Evaluates the program in terms of costs. It measures both the program costs and the results (benefits) in monetary terms. This means that the results of the program or benefits must be translated into a dollar value.
Cost-effectiveness - In this type of evaluation only program costs are expressed in monetary terms. Benefits are expressed only in terms of the impacts or outcomes themselves (they are not given a dollar value). Interpretation of this type of analysis requires stakeholders to decide if the benefit received is worth the cost of the program or if there are other less expensive programs that would result in a similar or greater benefit. Source: (THCU, 2007)
Evaluation Resources
Evaluation Training and Education
The following organizations offer training and education about evaluation: