Chapter 5 | Research and Evaluation
The purpose of this chapter is to identify relevant factors for conducting research on open pedagogy, with particular consideration for open educational practices (OEP) that showcase diversity, equity, accessibility, and inclusion (DEAI). This chapter will highlight the need for continued research on open pedagogy, as well as examples from the existing literature on the subject. We will also point to resources on effective methodology and best practices. Finally, we will suggest questions and topics that may inspire future research on open pedagogy.
This chapter does not provide an exhaustive list of possible OEP research methodologies. Instead, we are featuring recent studies that have approached OEP from an interdisciplinary perspective and have appealed to us for employing relevant theoretical approaches and creative methodologies.
Research on OEP
OEP is an emerging topic of research and there are many unexplored areas. It is important for scholars working on open pedagogy to conduct research in order to identify and fill research gaps and continue to refine what we understand about open pedagogical practices. Emerging research on OEP comes from an interdisciplinary perspective, using theories from sociology, education, psychology, and related fields. There are various broad categories of research on OER and open pedagogy:
- Perception-based studies: these explore stakeholders’ perceptions of open pedagogy from the perspective of students, faculty, administrators, librarians, or other groups. Data sources include surveys, focus groups, structured and semi structured interviews, as well as self-reflection. The analysis is typically organized around emerging themes and may include thematic coding.
- Studies that consider student perceptions often use interdisciplinary methods and focus on student agency.
- Studies that look at faculty perceptions look at the lived experience of educators as they create or use OER
- Comparison studies: these compare some objective measure, such as student performance, in two groups, one which employed open educational practices and one that did not. The goal of these studies is to find statistically significant differences, or to explain the lack of differences.
- Review of literature: these articles compile existing research on particular themes and can be useful for developing future research questions
- Research Case studies: these provide analysis and examples of innovative approaches and practices at a specific institution
Example | Theory/Approach | Method | Mode | Analysis |
1 | Transformative/emancipatory approach to learning
Tualaulelei, E., & Green, N. C. (2022) |
2 rounds of focus groups. Round one informs the questions in round 2 | Face-to-face or online | Thematic coding |
2 | Investigating how OER is used in informal settings, where learners are self-directed
Kim, D., Lee, I., & Park, J. (2019) |
Secondary data from OER Hub project | N/A | Identifying latent classes, covariates, and distal outcomes |
3 | Learning design theory
Hallam, S., Willingham, P., & Baranovic, K. (2021) |
Conducting a lesson in real-time and evaluating its success | Face-to-face or online | Pre and post surveys |
4 | McLaughlin’s theory of mutual adaptation
Hare, S., Frye, J. M., & Samuelson, B. L. (2020) |
Designing collaborative open pedagogy assignments | Face-to-face or online | student discussion forum posts and student feedback on final course evaluations |
A Primer on Methodology and Best Practices
Choosing the methodology or design of a study is highly dependent on its objectives. Generally, methodologies fall into the categories of qualitative, quantitative, or mixed methods. Ethical research methods are necessary for conducting any study, but they are particularly important in research that considers the implications of DEAI on OEP. As such, care and considerations should be taken in how subjects are recruited, as well as how gender, socioeconomics, race, and ethnicity are analyzed and represented. In addition, authors should also think about how their experiences, backgrounds, interests, and potential biases may impact both the study design and its outcomes. Authors are sometimes encouraged to include a statement of researcher positionality, in order to acknowledge their perspective in designing the study. Information on creating positionality statements can be found in A Note on Reflexivity and Positionality in Practicing and Presenting Social Research.
In addition, a key aspect of OEP is the importance of deconstructing traditional power structures (Cheng, 2023). It is important for OEP research to be aware of these and not unconsciously reproduce them. There are helpful resources available to gain more knowledge on the impact of traditional power structures in education (Morris and Friend; Gannon; & Fritzgerald)
The GO-GN Research Methods Handbook is a helpful resource to review when preparing to conduct research in OEP. The Handbook covers research methodology with a specific focus on open research.
Future research on OEP: Key Questions and Topics
Virginia Clinton-Lisell (2021) discusses several areas for future research on open pedagogy, including studies that relate to effectiveness, the need to explore OEP in more diverse international contexts (specifically low and middle income countries), as well as research that supplements student perceptions in various ways. While student perception research is essential, having more focused studies on different iterations of OEP practices could be helpful. As part of these efforts we suggest the importance of developing multi-institutional partnerships to support this type of research. In the Introduction, we provided a more detailed history and context about the formation and path of the group publishing this toolkit. What is relevant here are the challenges, successes and barriers we found when trying to form and maintain a multi-institutional research group. Explore this resource which provides an initial framework of research questions, and assessment items that we had planned to implement in multiple courses at multiple institutions. The challenges to our work included navigating multiple IRB systems and finding committed faculty partners (as levels of faculty load and burnout vary at different institutions, even though this problem was persistent across the group). The final challenge, which can become an opportunity, is to have clear and intentional conversations about the privileges held at certain institutions and how those can be leveraged to create equitable research projects. The idea is to continue to refine what “open” pedagogical practices mean, how they are categorized, and what differences may exist within them. It is important to also control for how the term “open” is used in various ways by both faculty and students. Intentionality in use of terminology will increase the effectiveness of OEP research.
Reflection Point
Research in OEP is key to supporting further development in this area. As you consider your own research and work, you may want to consider the following questions:
- How does scholarship fit into your job requirements?
- What are the expectations of faculty partners and how do we work together to bring in their disciplinary methods in OEP?
- Ready to get started? Consider returning to Chapter 1 as a place to begin your research and literature review.
References
Cheng, S. (2023). Leveraging OER and open pedagogy to promote EDI in the classroom. Publications and scholarship. 26. https://source.sheridancollege.ca/lls_publ/26
Clinton-Lisell, V. (2021). Open Pedagogy: A Systematic Review of Empirical Findings. Journal of Learning for Development 8(2), 255-268. https://files.eric.ed.gov/fulltext/EJ1314199.pdf
Farrow, R., Iniesto, F., Weller, M. & Pitt., R. (2020). The GO-GN research methods handbook. Open education research hub. The Open University, UK. https://go-gn.net/gogn_outputs/research-methods-handbook/
Fritzgerald, A. (2021). Honor as power: The practical keys to antiracist teaching. https://www.ascd.org/el/articles/honor-as-power-the-practical-keys-to-antiracist-teaching
Gannon, K. (2020). Radical hope: A Teaching manifesto. West Virginia University Press.
Hallam, S., Willingham, P., & Baranovic, K. (2021). A process of engagement: Using government documents in open pedagogy. The Journal of Academic Librarianship, 47(3), 102358–. https://doi.org/10.1016/j.acalib.2021.102358
Hare, S., Frye, J. M., & Samuelson, B. L. (2020). Open pedagogy as an approach to introducing doctoral students to open educational resources and information literacy concepts. Library Trends, 69(2), 435–468. https://doi.org/10.1353/lib.2020.0041
Hilton, J., Wiley, D., Chaffee, R., Darrow, J., Guilmett, J., Harper, S., & Hilton, B. (2019). Student perceptions of open pedagogy: An exploratory study. Open Praxis 11(3), 275–288. https://doi.org/10.5944/openpraxis.11.3.973
Kim, D., Lee, I., & Park, J. (2019). Latent class analysis of non‐formal learners’ self‐directed learning patterns in open educational resource repositories. British Journal of Educational Technology, 50(6), 3420–3436. https://doi.org/10.1111/bjet.12746
Morris, S. M., & Friend, C. “Listening for student voices.” Critical Digital Pedagogy: A Collection, Edited by Jesse Stommel, Chris Friend, and Sean Michael Morris. Hybrid Pedagogy Inc., 2020, pp. 127-131.
Robinson, O., & Wilson, A. (2022). Practicing and presenting social research. UBC Library. https://doi.org/10.14288/84SB-8T57
Tualaulelei, E., & Green, N. C. (2022). Supporting educators’ professional learning for equity pedagogy: the promise of open educational practices. Journal for Multicultural Education, 16(5), 430–442. https://doi.org/10.1108/JME-12-2021-0225