Why, How, and When to Get Help with SEL Program Evaluation
Most of us want to know if what we’re doing is making a difference. It’s human nature. In the world of school-based social and emotional learning, there are other reasons, in addition to human nature, to want to know if efforts to teach social and emotional competencies are making a difference.
Why Evaluate SEL Program Impact
Here are some of the more common reasons for wanting to know if efforts to nurture student social and emotional competencies are working:
- An SEL program provider needs to show prospective users that the program works, or at least, can work if done well.
- A district is implementing an SEL program and needs to know if it is working to decide whether to stay the course or make changes to increase impact.
- A researcher developing an SEL program or intervention wants to know whether it works and for whom.
- A funder requires a school or other organization to provide evidence of impact.
All of these situations involve questions of impact—does the activity that is intended to grow social and emotional competence result in that growth?
How to Evaluate SEL Program Impact
It seems like it would be a relatively straightforward matter to know whether an SEL program is making a difference. Just measure competencies before and after the program and see if things moved in the desired direction… right?
Pre-Post SEL Program Evaluation
Not exactly. If you want to know if X (SEL program) had an impact on Y (student social and emotional competence), you need to consider the evaluation design.
Measuring before and after a program is called a pre-post design. The problem with pre-post designs is that you can’t know if any change in competence level is the result of the program. Maybe students with no exposure to the program would show equal or greater growth on that measure. Maybe there was some other factor besides the program that resulted in the improvement.
So are pre-post designs worthless? No. While pre-post evaluations won’t get published in peer reviewed journals because of their flaws, they can provide at least some indication of whether things are going in the right direction. If outcomes are not improving or are getting worse, that’s not a great sign. If they are getting better, the program may be working. Because they don’t require a control group, pre-post designs are highly feasible and, in the absence of other kinds of data, they at least provide some basis in reasonably objective facts for a conversation about program effectiveness.
In addition, measuring over multiple time points can lead to a stronger, though still imperfect, sense of whether things are working. Imagine you have been implementing an SEL program for two years. Each year, you assessed social and emotional competence in the fall and the spring, and you noticed things weren’t really moving the way you’d hoped. You introduce an initiative to strengthen program implementation that you think will amplify the benefits of your program. If, after students have been exposed to that initiative, you see an acceleration in growth, it is likely to have something to do with the new initiative and program implementation.
Control Group SEL Program Evaluation
The strongest evaluation designs include a control group. In simple terms, a control group is group of comparable students (or classrooms or schools) who do not receive the SEL program whose impact you are trying to assess. The group of students (or classrooms or schools) who do receive the SEL program is the “treatment” group. If the students in the treatment group do better after the program implementation than the students in the control group, this provides stronger evidence that it was the program itself that produced the result.
There are many considerations in this kind of evaluation study, including these:
- Can you randomly assign students, classrooms, or schools to treatment and control groups? That permits the strongest causal inference, but is sometimes difficult to pull off.
- If you cannot, can you select a “control” group that is reasonably similar in their student composition and other characteristics to the “treatment” group? That is a “quasi-experimental design” and can yield pretty good evidence of impact if student social and emotional competence is assessed before and after the program.
- Is the unit of grouping the student, the classroom, or the school? There are pros and cons of each.
- How many students, classrooms or schools do you need in each group to stand a reasonable chance of detecting an effect if one is present? That is an issue of statistical “power.” How will you measure the outcomes you care about?
- Does your outcome assessment measure the same competencies that your program is designed to teach?
- Is the design that you are contemplating feasible in your setting?
- Once you collect the data, do you know how to analyze it to see whether the program had an impact?
What’s the Best SEL Program Evaluation Design
My feeling? There is no one best design. I take that back: The best design is the one that will accomplish your specific program evaluation goals as economically as possible. That involves weighing the pros and cons of evaluation design decisions against your specific goals.
Each of the considerations described above has implications for the resources it takes to pull off a good program evaluation, and the likelihood that, in the end, you’ll be able to say anything meaningful about program impact. For better or for worse, there are no wrong answers; there are only tradeoffs. What is important is that you design the evaluation study to meet your specific set of goals. If the goal is to establish a program as “evidence-based” a randomized field trial or a strong quasi-experiment is a must. If the goal is to see if measured competence is trending in the right direction after the introduction of a program, a pre-post design might be enough to at least give you a rough idea. As with most things that involve assessment, being clear about your goals is absolutely necessary (though not sufficient) if you want to accomplish that goal.
When to Get Help With SEL Program Evaluation
Designing and executing an effective program evaluation is no a simple matter. Most people who want to do so are not experts in research design. If your goal is anything more ambitious than a rough sense of the effectiveness of a program, I recommend consulting with someone with expertise in program evaluation to sort through design decisions, the resources required to pull off the evaluation, and to help with execution.
The xSEL Labs team has worked with leading research groups who use SELweb and Networker as research tools in the context of evaluation studies. In those partnerships, other research groups use our assessments to measure outcomes. We love knowing that leading scientists trust our assessments to measure the outcomes they care about.
We also have extensive expertise in research design and increasingly are advising organizations on the design and execution of program evaluations. We are pleased to announce that we have added a research service to our portfolio of offerings. We can help you design and execute your program evaluation, measure outcomes, and analyze the data.
Reach out if we can help you with your program evaluation.