Program Evaluation

The Program Evaluation consists of the following instruments:

  • Pre-test survey
  • Phase 1 formative assessment
  • Phase 2 formative assessment
  • Phase 3 formative assessment
  • Post-test survey

We drafted the questions in Google Docs, created the survey in QuestionPro, and disseminated the survey through the fellowship materials. This was through email (for the pre and post-tests) and through the fellowship materials at the end of each Phase (sessions 3, 6, and 8). All questions required a response unless we indicated otherwise. Participants did not get access to the next set of materials until they completed the formative assessments at the end of each Phase.

Here is the link to make a copy of the Program Evaluation plan: https://docs.google.com/document/d/1zsj-89hb6HPhWn0J9RGscYhu1cxLT0tVyK_pLQXiBtk/edit?usp=sharing

Research questions

We obtained IRB approval from our institution in order to research the following questions:

  1. How do faculty perceive their progress towards identified program learning outcomes
  2. What’s the language faculty are using to describe their approach to open pedagogy
  3. How does the experience of open pedagogical practices shape the dynamics of a classroom for faculty
  4. Does the use of open pedagogical practices impact faculty success toward identified teaching objectives?

If you have further questions, please reach out to the authors.

Pre-test survey

Q # Question Response
1 Please complete this pre-program questionnaire for the Opening Up Fellowship. As part of our faculty development programs we regularly gather data to improve the design and implementation of the program. This survey should take between 10-20 min to complete and must be completed prior to accessing the program content. We appreciate hearing from you and look forward to working together! No response
2 Prior to engaging with content from the fellowship please review the outcomes below and rate your confidence for each statement:
  1. I feel confident in my own experience with Open Pedagogy and OER in order to identify my own path of engagement.
  2. I feel confident in my ability to critically analyze my pedagogical, curricular, and assessment choices through a social justice lens.
  1. I feel confident in my ability to improve equity in the classroom through my pedagogical, curricular, and assessment choices.
  2. I feel confident in my ability to create pedagogical, curricular, and/or assessment materials to share with my community.
 For each statement

0=not confident, 1=slightly, 2=unsure 3=somewhat confident, 4=confident
3 How would you describe your current understanding of Open Pedagogy? Q3-5 get at attitude toward open pedagogy

Open response

Keep in a question block
4 What does it mean, to you, for educational materials to be Open? Open
5 How would you describe what having a  “world view of open” means to  you personally, as an educator, and/or as a scholar? open
6 Please rate your confidence for the following statements regarding classroom dynamics

I am confident in my ability to…

  1. Explain the purpose of an open pedagogy assignment/project to students.
  2. Integrate open pedagogy into my current teaching practices.
  3. Explain to students their role as knowledge creators through an open pedagogy assignment/project
  4. Guide students through the process of an open pedagogy assignment/project.
 How open impacts classroom dynamics

0=not confident, 1=slightly, 2=unsure 3=somewhat confident, 4=confident

 
7 I’m clear on how to integrate my open pedagogical work into my annual faculty goals. 7-10 in one question block

0=strongly disagree 1=disagree 2=unsure 3=agree 4=strongly agree
8 I believe my departmental colleagues value the work of Open Pedagogy to improve equity. 0=strongly disagree 1=disagree 2=unsure 3=agree 4=strongly agree
9 I believe my departmental structures (eg. leadership, policies, culture) reward the work of Open Pedagogy to improve equity. 0=strongly disagree 1=disagree 2=unsure 3=agree 4=strongly agree
10  

Please share any examples you have of how the work of Open Pedagogy to improve equity has/has not been rewarded in your department.

 *not required question
11 Please provide any other thoughts, questions, or concerns you have regarding open pedagogy. Last question block

*not required

Open-ended

Phases 1-3 formative assessment

Q # Question Response
1 Please complete this reflection survey for the Opening Up Fellowship. This activity is a formative assessment reflection for you to consider your own progress in the program as well as to provide feedback for facilitators. Completing these formative assessment surveys is considered a critical component to fulfilling the requirements of this program (as outlined in the initial MOU). This survey should take between 15-30 minutes to complete. We appreciate you taking the time to reflect on your experience thus far in this formative assessment and for all of your contributions to the program. No response
2 Name
3 These are the learning objectives for Phase 1 of the Fellowship (which you have completed). Please rate your confidence for each statement:
  1. I feel confident in my ability to explore the landscape of open educational practices and the values of open in order to reflect on where I am in my open journey.
  2. I feel confident in my ability to Engage with global and local conversations around the role of social justice and open.
  3. I feel confident in my ability to critically question traditional measures of authority in the scholarly landscape.
 For each statement

0=not confident, 1=slightly, 2=unsure 3=somewhat confident, 4=confident
4 Take some time to go back through your Evolve guide reflections and respond to the following questions. Text
5 How would describe your current understanding of Open Pedagogy? Open-ended
6 How did the materials and activities from Phase 1 of this fellowship (sessions 1, 2, and 3) contribute to your understanding of open pedagogy and social justice? Open-ended
7 Please share any other comments or questions that you would like the facilitators to know regarding your learning experience in this program thus far. Open-ended

Post-test Survey Questions

Q # Question Response
1 Please complete this pre-program questionnaire for the Opening Up Fellowship. As part of our faculty development programs, we regularly gather data to improve the design and implementation of the program. This survey should take between 10-20 min to complete and must be completed prior to accessing the program content. We appreciate hearing from you and look forward to working together! No response
2 Prior to engaging with content from the fellowship please review the outcomes below and rate your confidence for each statement:
  1. I feel confident in my own experience with Open Pedagogy and OER in order to identify my own path of engagement.
  2. I feel confident in my ability to critically analyze my pedagogical, curricular, and assessment choices through a social justice lens.
  1. I feel confident in my ability to improve equity in the classroom through my pedagogical, curricular, and assessment choices.
  2. I feel confident in my ability to create pedagogical, curricular, and/or assessment materials to share with my community.
 For each statement

0=not confident, 1=slightly, 2=unsure 3=somewhat confident, 4=confident
3 How would you describe your current understanding of Open Pedagogy? Q3-5 get at attitude toward open pedagogy

Open response

Keep in a question block
4 What does it mean, to you, for educational materials to be Open? Open
5 How would you describe what having a  “world view of open” means to  you personally, as an educator, and/or as a scholar? open
6 Please rate your confidence for the following statements regarding classroom dynamics

I am confident in my ability to…

  1. Explain the purpose of an open pedagogy assignment/project to students.
  2. Integrate open pedagogy into my current teaching practices.
  3. Explain to students their role as knowledge creators through an open pedagogy assignment/project
  4. Guide students through the process of an open pedagogy assignment/project.
 How open impacts classroom dynamics

0=not confident, 1=slightly, 2=unsure 3=somewhat confident, 4=confident

 
7 I’m clear on how to integrate my open pedagogical work into my annual faculty goals. 7-10 in one question block

0=strongly disagree 1=disagree 2=unsure 3=agree 4=strongly agree
8 I believe my departmental colleagues value the work of Open Pedagogy to improve equity. 0=strongly disagree 1=disagree 2=unsure 3=agree 4=strongly agree
9 I believe my departmental structures (eg. leadership, policies, culture) reward the work of Open Pedagogy to improve equity. 0=strongly disagree 1=disagree 2=unsure 3=agree 4=strongly agree
10  

Please share any examples you have of how the work of Open Pedagogy to improve equity has/has not been rewarded in your department.

 *not required question
11 Please provide any other thoughts, questions, or concerns you have regarding open pedagogy. Last question block

*not required

Open-ended

* Additional post-test questions coming soon. 

 

 

License

Icon for the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Opening Up: A Fellowship for Open Pedagogy Copyright © 2023 by Elaine Kaye; Liz Chenevey; and Nicole Wilson is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted.

Share This Book