Do you have any experience managing a program? Have you contributed to the planning and design of a program?
Have you been involved in the evaluation of a program's success?
If you answered yes to any of those questions, then we need your help! Matthew Gallagher is a PhD Candidate at Arizona State University. He is also a professional program evaluator, which means that his job is to collect and use quantitative and qualitative data to aid program leaders and their teams in developing knowledge about, making data-informed decision on, and managing their programs.
Program leadership’s decision to include an evaluator during the planning and design phase of their programs is the critical first step necessary for evaluators to provide the benefits associated with the discipline of evaluation. Within the evaluation field, federal legislation, including the recently passed Evidence Act of 2019, promotes the inclusion of evaluators in the planning and design phase of federal programs. Additionally, evaluation academics and practitioners advocate for this level of inclusion within all types of programs.
However, despite legislation and advocacy efforts, the evaluation literature depicts a situation where evaluators are not consistently included in the planning and design phase of programs due to numerous barriers, including:
1) program funding announcements which request the inclusion of evaluation activities in proposed programs, but do not require evidence of any collaboration with a program evaluator;
2) training programs that offer little or no training on the benefits of hiring an evaluator during the program planning and design phase.
Overall, the literature portrays programs as plagued by rushed and/or symbolic evaluations that are of limited use for program management, thereby frustrating program leaders and demoralizing evaluation practitioners.
To date, academic studies have examined approaches on how to incorporate an evaluator into a program’s planning and design phase (Fitzpatrick, 1988) and multiple researchers have advocated on behalf of the inclusion of the evaluator throughout all the phases of the project cycle (see: Patton, 1978; Stufflebeam, 2001; Preskill & Torres 2001; Mark, 2012; Scheirer, 2012). However, no studies have examined the extent to which evaluator inclusion occurs, or explored the reasons why a program leader decides whether to include an evaluator in a program’s planning and design phase. His PhD research aims to study this topic from the perspectives of both program evaluators and program leaders (who include program directors and program managers).
From the program evaluation side, he is currently collaborating with the American Evaluation Association to procure perspectives on this topic from 1,000 of their members, who were selected at random. To capture the perspectives of program leaders on this topic, he is also collaborating with local chapters of the Project Management Institute.
The new knowledge generated from this study has the potential to affect how program leaders and program evaluators are trained. When we have a mutual understanding of the extent to which evaluators are not included in the program planning and design phase, as well as what influences program leaders’ decisions to include or exclude evaluators during this phase, then we can develop a roadmap for how training content should be augmented to meet the federal mandate, fulfill advocacy efforts, and be proactive on exploring the topic more broadly.
How You Can Help
Let's help Matthew with his research by taking 10 minutes to complete his questionnaire. Your viewpoint and experience on the topic of evaluator inclusion in the program planning and design phase will be beneficial to his research and the field of program management. This survey will be open through December 15th. Additional information about the study, your voluntary participation, and data privacy are on the first page of the link before you enter the survey.
Thank you in advance for your thoughtful and honest responses!
Take the Survey: https://asu.co1.qualtrics.com/jfe/form/SV_1S8gMVSCOJOSBKe
Matthew Gallagher (right) collaborating with program leaders on program evaluation activities in Dili, Timor Leste.