Education research has been in the news, as the Trump administration has dramatically reduced funding for it.
Nevertheless, there’s still tons of it around, and more still will be forthcoming.
How can teachers best use research to help their students?
Today’s post is the first in a three-part series designed to walk educators through the process of interpreting research to help us all “sift the wheat from the chaff.”
What Does ‘Evidence-Based’ Mean?
Editor’s Note: As of this writing, all of the links to federal government sites are live.
Cara Jackson currently serves as the president of the Association for Education Finance & Policy. She previously taught in the New York City public schools and conducted program evaluations for the Montgomery County public schools in Maryland.
Norma Ming serves as the managing director of the Impact Lab at the Carnegie Foundation for the Advancement of Teaching and previously worked at the San Francisco Unified school district as manager of research and evaluation.
Liz Farley-Ripple is a professor in the School of Education at the University of Delaware, co-directs the Center for Research Use in Education, and is the director of the Partnership for Public Education.
Mentions of “evidence-based” have increased so dramatically over the past 25 years that educators may find it overwhelming. The field has undergone rapid changes over the past few weeks. Many educators are committed to using evidence to improve educational opportunities and outcomes.
In this post, we discuss how educators can assess current programs against evidence, explore the role of local context in implementation, and provide resources to support educators’ use of evidence. We use the term “program” throughout as shorthand for “program, policy, practice, intervention, or strategy.”
What are you hoping to accomplish, and what do you know about progress to date?
Educators should start by reflecting on what current programs are expected to accomplish, such as improving educational attainment, achievement, and social and emotional well-being. The federal guidance intended to strengthen the use of evidence offers a helpful list of questions to consider as you identify local needs and seek to understand the root causes of those needs.
Are current programs aligned with evidence?
Educators can next examine whether current programs align with evidence, consistent with the call for identifying evidence-based program components in the federal guidance. The Institute of Education Sciences (IES) practice guides and the Education Endowment Foundation resources reflect evidence-informed practices, while IES intervention reports and Evidence for ESSA offer summaries of research evidence for specific programs.
Though the answer to this question is unlikely to be as simple as “yes” or “no,” a general sense of how well aligned the current program is with the evidence can be helpful in thinking through whether to adapt the program or adopt a new one. The innovation domain of the Consolidated Framework for Implementation Research (CFIR) calls for examining the degree to which the program has evidence supporting effectiveness and is better than other available alternatives.
Figure by Cara Jackson
If the program is aligned with evidence, would changes improve outcomes?
A program’s success or failure depends on many contextual factors. If a current program is aligned with evidence but not achieving intended outcomes, consider what is needed to strengthen implementation. Additional training, staff, or resources may be necessary, including time for educators to collaboratively troubleshoot implementation challenges.
Educators’ time and energy is finite, so it’s important to consider whether reducing other demands on their time could improve implementation of current programs. The CFIR includes infrastructure, relational connections, communications and culture among the features of the inner setting to consider.
Is it worth replacing the program with a new one?
Though any new program incurs some opportunity cost in the time and energy invested in the change, if your answers to the questions above are “no,” it may be worth changing course. Following the principle that “a policy should be adopted only if there is no other way to create the same effect at a lower cost” (Harris, 2009), here are some questions to consider:
- How similar is your current program to the alternative? If an alternative program is similar to the current program, it’s unlikely to improve outcomes much.
- How similar is the local context to the context of the study or studies that demonstrate effectiveness?
Education researcher Dylan Wiliam has said “everything works somewhere, and nothing works everywhere.” As noted earlier, program success or failure may be related to contextual factors, so results from a study in a setting similar to yours may be more compelling than results from dissimilar settings. The CFIR lists local attitudes and conditions, policies and laws, and finance among the features of the outer setting to consider.
If your setting differs from the study setting, here are some questions to ask to determine if it’s likely that you would see similar results in your context.
- What features of the context matter for the program to be successful? Whether similarity in features such as geographic locale matters likely depends on what problem you’re trying to solve. Other contextual features, such as state or local policies, teachers’ union contracts, and qualifications of the staff implementing the program may be important to consider.
- To what extent is this program appropriate? Is it suitable for your teachers, parents, and students?
- Will the program be acceptable in your district, school, or classroom? Does it align well with other initiatives? Lack of alignment may create tension between priorities, and implementing a new program may divert teachers’ time and energy from other priorities.
- What resources are needed to implement the program, and are these resources available? While teachers’ time and energy is a key resource to consider,
the Center for Research and Reform in Education’s Amanda Neitzel also offers questions to consider regarding training needs, access to resources, and monitoring effectiveness. If you don’t have ALL the resources you need, would the program be compromised and in what ways? Would the likely benefits still outweigh those costs?
Key takeaways:
- Start with your goals and an assessment of whether current programs are meeting those goals.
- If the current program is aligned with evidence, consider whether it can be modified or supplemented to achieve goals.
- If adopting a new program, ask whether a program would be well-situated in your context.
The next post in this series will discuss how to be a critical consumer of research claims.
Thanks to Cara, Norma, and LIz for contributing their thoughts!
Consider contributing a question to be answered in a future post. You can send one to me at [email protected]. When you send it in, let me know if I can use your real name if it’s selected or if you’d prefer remaining anonymous and have a pseudonym in mind.
You can also contact me on Twitter at @Larryferlazzo.
Just a reminder; you can subscribe and receive updates from this blog via email. And if you missed any of the highlights from the first 13 years of this blog, you can see a categorized list here.
