Step # 3 Construct a program theory.
By implementing and evaluating the not-for-profit social program entitled, The Brain Fit Super Powers, we are better able to see how this program will benefit student learning. Before we begin identifying the particular program theory, let us delve into what is program theory and when we develop a program theory.
According to Sedani and Sechrest in A Review of Program Theory and Theory-Based Evaluations, “a program theory consists of a set of statements that describe a particular program, explain why, how, and under what conditions the program effects occur, predict the outcomes of the program, and specify the requirements necessary to bring about the desired program effects” (Sharpe, 2011, 72). In many occasions, programs functions with very little evaluation or program evaluation will occur during the onset of these programs. The latter can be more important as stakeholders will not have to entirely revise programs after evaluation begins if there are glaring errors and very little change occurring. Sharpe states, “the development of a program theory is necessary when hoping to determine why a program is succeeding or failing and if and where program improvement should be focused.
Additionally, a rationale may be vital to the success of the program and evaluation. A rationale is essentially the identification that transformation must occur and even a blueprint may follow. Chen explains, “the form the program rationale takes is that of systematic argument in which stakeholders assert the potential of a specific intervention’s potential to achieve a stated goal” (Chen, 2005, 3). Program rationales can be visual representations much like that of a logic model, almost a conceptual framework.
For any program evaluation to be successful, the program must have intentions or program goals. Program theories often use the “if-then” statements created to identify the action or intervention task and the following result or short- and long-term outcomes. According to Wilder Research, “a program theory should also spell out why you expect the changes to happen. Between the “if” and the “then’, there should be some solid evidence or some well-established connection supporting the idea that your service package will accomplish your program goals” (Wilder Research, 2009, 2).
Each program theory will be composed of 3 complete evaluand notions that will assist in indicating success and creating a structure. Chen identifies,
• “Comprehensive evaluation: studying the design and conceptualization of an intervention, its implementation, and its utility.
• Tailored evaluation: Evaluation questions and research procedures depend on whether the program is an innovative intervention, a modification or expansion of an existing effort, or a well-establish, stable activity.
• Theory-driven evaluation: Constructing models of how programs work, using the models to guide question formulation and data gathering; similar to what econometricians call model specification” (Donaldson & Lipsey, 2011, 14).
With all of this in mind, the particular context in which we will be applying a program theory will be that mentioned in the module 1 conditions. To review, The Brain Fit Super Powers is a social emotional program based on supporting students in a variety of areas. According to Natalie Hunniford, program creator, “the purpose of this program is to enhance each student’s mental health, which will result in improved overall wellbeing, emotional and physical health, and academic achievement” (Hunniford, 2019, 2). However, as the program is intended for school year long development, the immediate purpose has been identified by staff and administration; teaching students to use calming strategies to manage escalation and/or relieve anxiety. In this particular context, students are coming from an array of upbringings often connected to challenging circumstances. After completing a “scan” prior to beginning School Planning as an Inquiry Process, staff noticed through collaboration, surveys, staff meeting discussion and observational checklists, that students were consistently becoming escalated and lashing out before utilizing coping strategies. These students would utilize inappropriate language, become physical or escape these situations often putting themselves in danger. Therefore, it was through staff input and collaboration an intervention was visibly required. According to Kaser and Halbert, authors of The Spirals Playbook “teamwork is essential, and so is building a wider network of inquiry-minded people to deepen and spread the learning (Kaser & Halbert, 2017, 17).
As our program focuses on more than one element of social and emotional learning, we must provide a specific “If-Then” program goal to better allow us to evaluate the program. Throughout this process, we will be using evaluation of learning and use model as it applies greatly to our context. For this model, as stated by Dr. Green, this approach to evaluation has the purposes of “contribut[ing] to the quality of the program and to maximize the usefulness of the evaluation” (Education at Illinois, 2017b).
To begin developing a program theory, we shall utilize the “if-then connection” and the narrowing a focus developed by Simon Breakspear. This metaphor of “boulder-pebble-sand” will allow us to begin with a broader focus of the program theory and eventually develop evaluative program goals that are much more specific to our action and learners.
IF we use (specific instructional program) to improve student ability, THEN the intended outcome (specific skill as per the current curriculum) will occur.
If we implement The Brain Fit Super Powers program in our school, then students will have improved mental health.
If we have daily morning meetings using lessons based on calming down strategies, then students will de-escalate in moments of frustration, anxiety, or conflict.
If we have morning announcements based on the calming down strategies, then students will have improved knowledge to support in challenging situations.
If we have monthly assemblies with skits and videos based on each month’s social and emotional focus, then students will improve their understanding mental health and well-being.
If we create monthly bulletin boards based on each moth’s social and emotional focus, then student will develop a more consistent aware and language of mental health and well-being.
As we focus more on theory of action and theory of change, there are reports that this can occur at the onset of program implementation and evaluation or can be used throughout. According to Action Evaluation, “it is a powerful tool to surface assumptions and support critical thinking and reflection while also strengthening planning processes and sharpening strategies. It also provides a framework for monitoring and evaluation, by focusing efforts on the outcomes that need to be tracked and documented” (2013). Therefore, a series of evaluative steps must follow with long term goals at the crux of the program intentions: long-term vision, outcome pathways, testing assumptions, context and external factors, activities/inputs, and testing the logic. Action evaluation states, “The Theory of Action or Theory of Change is a framework for monitoring and evaluating your work (an outcomes framework) as well as an important tool to support planning, organizational learning, and strategic planning and adaptation” (2013). The format below will represent the goals and the orientations in which the program will be evaluated. Throughout the program evaluation we should be observing change predetermined in our assumptions prior to implementing the program. As mention in Better Education, “A theory of change explains how the activities undertaken by an intervention (such as a project, program or policy) contribute to a chain of results that lead to the intended or observed impacts” (i.e., pipeline results), (Education at Illinois, 2017b).
Please see attached Program Theory Development - Theory of Action and Theory of Change Format
Step #4: Identify, describe, and rationalize your evaluation approach.
As the program theory is based on an intervention that is designed to improve student social and emotional learning, there are many factors to consider. The inputs are far reaching, but the true foci are based on blended learning teachers, administration, and the activities put forth to benefit students. As previously mentioned, much of the behavioural issues we are seeing in our particular context are based on students having difficulty self-regulating during moments of duress. Chen states, a rationale is developed, “when determinants have been identified adequately for the purposes of the program, the evaluator’s work shifts to facilitating the working group’s selection of an intervention that can activate those determinants” (Chen, 2011, 12). Therefore, the Brain Fit Super Powers has been chosen and implemented to best support these visible needs of our students. To best evaluate this program, I will be using an evaluation of learning and use approach. According to Dr. Green, “this type of evaluation is typically oriented to program leadership and staff, people on the ground in leadership positions: the principals in a school . . .” (Education at Illinois, 2017b).
Evaluation of learning and use is a natural choice within this context as mixed methods of assessment occur and no particular design in this approach is utilized. Green states that this evaluation approach is for, “people interested in a variety of questions based on outcomes and processes and implementation and context. The purpose is to contribute to the quality of the program and usefulness of evaluation for the program” (Education at Illinois, 2017b). As this approach focuses on a variety of questions, they must be dependent on practical applications, contextual needs and acquiring data for participants.
Dr Green identifies,
“How well does the program’s theory of change work in practice?
How well does the program address the priority needs of diverse participants?
What are the priority information needs of key stakeholders?” (Education at Illinois, 2017b).
As this form of evaluation is used at a grassroots level of implementation, it is entirely appropriate for this particular program. It will be used with staff and administration in schools. There will be a variety of methods used to evaluate. The focus of the evaluation will be to ensure the efficacy of the program and practicality of the evaluation for the Brain Fit Super Powers. Staff will teach new strategies and concepts to students, with a monthly change. Administration will develop morning announcements for the entire school and community. Additionally, staff and administration will provide monthly assemblies in which these new concepts are introduced and demonstrated by student leaders. Bulletin boards will be strategically placed throughout the school emphasizing each monthly focus. And finally, monthly parent newsletters will be sent out informing the community of monthly focus and continuity of language and strategies being highlighted.
Evaluative Questions:
To what extent was the programme purpose met?
Implementation Questions
-To what extent will daily morning meetings influence students’ coping strategies?
-To what extent will morning announcements with embedded Brain Fit Super Powers influence student recognition of social and emotional learning?
-To what extent will monthly assemblies focusing on social and emotional learning stimulate awareness?
-To what extent will establishing monthly Brain Fit Super Powers bulletin boards impact our community of learners’ language use?
What data are needed?
How could the programme be improved for teachers?
By implementing and evaluating the not-for-profit social program entitled, The Brain Fit Super Powers, we are better able to see how this program will benefit student learning. Before we begin identifying the particular program theory, let us delve into what is program theory and when we develop a program theory.
According to Sedani and Sechrest in A Review of Program Theory and Theory-Based Evaluations, “a program theory consists of a set of statements that describe a particular program, explain why, how, and under what conditions the program effects occur, predict the outcomes of the program, and specify the requirements necessary to bring about the desired program effects” (Sharpe, 2011, 72). In many occasions, programs functions with very little evaluation or program evaluation will occur during the onset of these programs. The latter can be more important as stakeholders will not have to entirely revise programs after evaluation begins if there are glaring errors and very little change occurring. Sharpe states, “the development of a program theory is necessary when hoping to determine why a program is succeeding or failing and if and where program improvement should be focused.
Additionally, a rationale may be vital to the success of the program and evaluation. A rationale is essentially the identification that transformation must occur and even a blueprint may follow. Chen explains, “the form the program rationale takes is that of systematic argument in which stakeholders assert the potential of a specific intervention’s potential to achieve a stated goal” (Chen, 2005, 3). Program rationales can be visual representations much like that of a logic model, almost a conceptual framework.
For any program evaluation to be successful, the program must have intentions or program goals. Program theories often use the “if-then” statements created to identify the action or intervention task and the following result or short- and long-term outcomes. According to Wilder Research, “a program theory should also spell out why you expect the changes to happen. Between the “if” and the “then’, there should be some solid evidence or some well-established connection supporting the idea that your service package will accomplish your program goals” (Wilder Research, 2009, 2).
Each program theory will be composed of 3 complete evaluand notions that will assist in indicating success and creating a structure. Chen identifies,
• “Comprehensive evaluation: studying the design and conceptualization of an intervention, its implementation, and its utility.
• Tailored evaluation: Evaluation questions and research procedures depend on whether the program is an innovative intervention, a modification or expansion of an existing effort, or a well-establish, stable activity.
• Theory-driven evaluation: Constructing models of how programs work, using the models to guide question formulation and data gathering; similar to what econometricians call model specification” (Donaldson & Lipsey, 2011, 14).
With all of this in mind, the particular context in which we will be applying a program theory will be that mentioned in the module 1 conditions. To review, The Brain Fit Super Powers is a social emotional program based on supporting students in a variety of areas. According to Natalie Hunniford, program creator, “the purpose of this program is to enhance each student’s mental health, which will result in improved overall wellbeing, emotional and physical health, and academic achievement” (Hunniford, 2019, 2). However, as the program is intended for school year long development, the immediate purpose has been identified by staff and administration; teaching students to use calming strategies to manage escalation and/or relieve anxiety. In this particular context, students are coming from an array of upbringings often connected to challenging circumstances. After completing a “scan” prior to beginning School Planning as an Inquiry Process, staff noticed through collaboration, surveys, staff meeting discussion and observational checklists, that students were consistently becoming escalated and lashing out before utilizing coping strategies. These students would utilize inappropriate language, become physical or escape these situations often putting themselves in danger. Therefore, it was through staff input and collaboration an intervention was visibly required. According to Kaser and Halbert, authors of The Spirals Playbook “teamwork is essential, and so is building a wider network of inquiry-minded people to deepen and spread the learning (Kaser & Halbert, 2017, 17).
As our program focuses on more than one element of social and emotional learning, we must provide a specific “If-Then” program goal to better allow us to evaluate the program. Throughout this process, we will be using evaluation of learning and use model as it applies greatly to our context. For this model, as stated by Dr. Green, this approach to evaluation has the purposes of “contribut[ing] to the quality of the program and to maximize the usefulness of the evaluation” (Education at Illinois, 2017b).
To begin developing a program theory, we shall utilize the “if-then connection” and the narrowing a focus developed by Simon Breakspear. This metaphor of “boulder-pebble-sand” will allow us to begin with a broader focus of the program theory and eventually develop evaluative program goals that are much more specific to our action and learners.
IF we use (specific instructional program) to improve student ability, THEN the intended outcome (specific skill as per the current curriculum) will occur.
If we implement The Brain Fit Super Powers program in our school, then students will have improved mental health.
If we have daily morning meetings using lessons based on calming down strategies, then students will de-escalate in moments of frustration, anxiety, or conflict.
If we have morning announcements based on the calming down strategies, then students will have improved knowledge to support in challenging situations.
If we have monthly assemblies with skits and videos based on each month’s social and emotional focus, then students will improve their understanding mental health and well-being.
If we create monthly bulletin boards based on each moth’s social and emotional focus, then student will develop a more consistent aware and language of mental health and well-being.
As we focus more on theory of action and theory of change, there are reports that this can occur at the onset of program implementation and evaluation or can be used throughout. According to Action Evaluation, “it is a powerful tool to surface assumptions and support critical thinking and reflection while also strengthening planning processes and sharpening strategies. It also provides a framework for monitoring and evaluation, by focusing efforts on the outcomes that need to be tracked and documented” (2013). Therefore, a series of evaluative steps must follow with long term goals at the crux of the program intentions: long-term vision, outcome pathways, testing assumptions, context and external factors, activities/inputs, and testing the logic. Action evaluation states, “The Theory of Action or Theory of Change is a framework for monitoring and evaluating your work (an outcomes framework) as well as an important tool to support planning, organizational learning, and strategic planning and adaptation” (2013). The format below will represent the goals and the orientations in which the program will be evaluated. Throughout the program evaluation we should be observing change predetermined in our assumptions prior to implementing the program. As mention in Better Education, “A theory of change explains how the activities undertaken by an intervention (such as a project, program or policy) contribute to a chain of results that lead to the intended or observed impacts” (i.e., pipeline results), (Education at Illinois, 2017b).
Please see attached Program Theory Development - Theory of Action and Theory of Change Format
Step #4: Identify, describe, and rationalize your evaluation approach.
As the program theory is based on an intervention that is designed to improve student social and emotional learning, there are many factors to consider. The inputs are far reaching, but the true foci are based on blended learning teachers, administration, and the activities put forth to benefit students. As previously mentioned, much of the behavioural issues we are seeing in our particular context are based on students having difficulty self-regulating during moments of duress. Chen states, a rationale is developed, “when determinants have been identified adequately for the purposes of the program, the evaluator’s work shifts to facilitating the working group’s selection of an intervention that can activate those determinants” (Chen, 2011, 12). Therefore, the Brain Fit Super Powers has been chosen and implemented to best support these visible needs of our students. To best evaluate this program, I will be using an evaluation of learning and use approach. According to Dr. Green, “this type of evaluation is typically oriented to program leadership and staff, people on the ground in leadership positions: the principals in a school . . .” (Education at Illinois, 2017b).
Evaluation of learning and use is a natural choice within this context as mixed methods of assessment occur and no particular design in this approach is utilized. Green states that this evaluation approach is for, “people interested in a variety of questions based on outcomes and processes and implementation and context. The purpose is to contribute to the quality of the program and usefulness of evaluation for the program” (Education at Illinois, 2017b). As this approach focuses on a variety of questions, they must be dependent on practical applications, contextual needs and acquiring data for participants.
Dr Green identifies,
“How well does the program’s theory of change work in practice?
How well does the program address the priority needs of diverse participants?
What are the priority information needs of key stakeholders?” (Education at Illinois, 2017b).
As this form of evaluation is used at a grassroots level of implementation, it is entirely appropriate for this particular program. It will be used with staff and administration in schools. There will be a variety of methods used to evaluate. The focus of the evaluation will be to ensure the efficacy of the program and practicality of the evaluation for the Brain Fit Super Powers. Staff will teach new strategies and concepts to students, with a monthly change. Administration will develop morning announcements for the entire school and community. Additionally, staff and administration will provide monthly assemblies in which these new concepts are introduced and demonstrated by student leaders. Bulletin boards will be strategically placed throughout the school emphasizing each monthly focus. And finally, monthly parent newsletters will be sent out informing the community of monthly focus and continuity of language and strategies being highlighted.
Evaluative Questions:
To what extent was the programme purpose met?
Implementation Questions
-To what extent will daily morning meetings influence students’ coping strategies?
-To what extent will morning announcements with embedded Brain Fit Super Powers influence student recognition of social and emotional learning?
-To what extent will monthly assemblies focusing on social and emotional learning stimulate awareness?
-To what extent will establishing monthly Brain Fit Super Powers bulletin boards impact our community of learners’ language use?
What data are needed?
How could the programme be improved for teachers?