6. METHODS AND LIMITS

Leadership is a complex phenomenon, which makes it challenging to study. While an increasing number of studies have been published in this field, this body of research faces many hurdles. One of the biggest challenges is simply appropriately measuring leadership (Stentz et al., 2012). Uncertainties about the nature of leadership have also provoked controversies about how to best study it (Yukl 1989).

Fortunately, social scientists are constantly improving methodologies to measure leadership effectively. Such methods help us understand how different leadership styles, combined or separated, apply in various environments and affect both leaders and followers. These approaches help elucidate how leaders influence followers and predict decision-making processes in specific situations. Surveys and questionnaires have and may continue to be the dominant quantitative methodologies used to study leadership. However, scholars interested in conducting leadership research should be aware of the limitations of nonexperimental designs and should consider exploring alternative or mixed methods approaches.

We also wish to highlight the importance of peer critique in advancing scholarship, particularly in leadership research. Several qualified scholars have aptly pointed out flaws in methodologies commonly used in leadership research. While this shows there is much room for improvement, it does not negate progress made in leadership research to date. Valid critiques must be taken seriously to expand knowledge in the field, and scholars must find inspiration rather than discouragement in those critiques.

Led by Leadership Quarterly (LQ) editors (see LQ special issue volume 34, issue 6), scholars have identified flaws in common methodologies, such as the overreliance on questionnaires and self-reports, which can be biased and fail to capture the complexity of leadership dynamics. Below is a list of key critiques:

Endogeneity

Lonati and Antonakis (2023) reported that a critical issue in leadership research is endogeneity—that is, when an independent variable is correlated with an unexplained error, or variation, in the dependent variable. Endogeneity can lead to problems like omitted variable bias, reverse causality, and measurement errors, which undermine causal claims. Researchers in the field of leadership, who have undoubtedly had many of their own experiences as followers and leaders, must be particularly diligent in using the scientific method so that preconceived notions do not bias study designs or analysis of results (Antonakis et al., 2004).

Construct Redundancy

Banks et al. (2018) argued that the field of leadership studies needs to prune its constructs, given that many constructs overlap heavily. Their meta-analysis revealed that task-oriented, relational, passive, inspirational, values-based, and moral leadership behaviors were susceptible to construct redundancy.

Conflation

Banks et al. (2023) pointed out that many studies in the field of leadership “conflate non-behavioral and behavioral concepts.” Their systematic review of 214 studies reported that only 3% of analyzed variables were behavioral in nature. In other words, most leadership research does not directly measure behavior, even though it is a critical component of leadership theory and even though studies often advance conclusions with behavioral implications.

Lack of Context

Studies of leadership often ignore contextual factors, despite well-established situational leadership theories. Research relying on one-time surveys may not reflect the process-oriented nature of leadership.

Assumptions about Subordinates and Leaders

Many leadership studies use pre-developed instruments that ask subordinates to assess leaders’ behaviors. As Hunter et al. (2007) point out, this assumes that all relevant leadership behaviors have been witnessed by followers, that employees are accurate raters of behaviors they observe, and that employees are equally impacted by leadership. Asking followers about their supervisor or even asking supervisors to self-report assumes that all supervisors are leaders and does not allow for a distinction between leaders and managers.

Lack of Multi-level Analysis

Hunter et al. (2007) argue that “leadership is an inherently multi-level phenomenon” and that more leadership research should recognize this (p. 441). While a good deal of leadership research is hierarchical in nature, leadership scholars only began to conceptualize leadership as a multi-level construct in the 1990s (Batistic et al., 2017). Leaders of workers within firms and firms within industries are examples of clusters that researchers may need to account for to keep from violating the assumption of independence. Statistical methods such as multilevel modeling can help with this. However, two-thirds of studies in a review by Antonakis et al. (2010) failed to include controls for clustering that may confound results. Indeed, multi-level research in the leadership field is fragmented and sometimes non-existent in the most popular leadership conversations (Batistic et al., 2017).

Batistic et al. (2017) acknowledge the fact that leadership researchers “tend to avoid multi-level work because it is ‘too difficult’” but add that this is not a valid excuse if researchers want to advance the field (p. 98). Arguments have also been made to promote an increase in the use of structural equation modeling (SEM). SEM has a great deal of utility in leadership research with capabilities of mapping complex relationships and quantifying unobserved latent factors (Batistic et al., 2017).

Still, many leadership scholars have not received training in the use statistical methods beyond multiple regression, which is a significant barrier to conducting research that leverages more advanced statistical techniques.

To overcome these issues, scholars have focused on revising methods, study designs, and theories. Solutions include experimental randomization, quasi-random variation, and comprehensive theoretical models to account for confounding variables. Readers are advised to consult peer-reviewed articles for each method. For experimental design studies, see Podsakoff and Podsakoff (2019). For mixed methods studies, see Stentz et al. (2012). For natural experiments, see Sieweke and Santoni (2020). For a comprehensive (if slightly dated) inventory of research in leadership see Dinh et al. (2014).

Recognizing the value of theory and the need for qualitative research to complement quantitative methods, LQ has called for two special issues focusing on the contribution of theory and qualitative methods. Overall, the field of leadership research needs to incorporate more rigorous and diverse scientific methods to avoid biased results and enhance the validity of findings.

Indeed, while much attention has been given to quantitative research, it is also valuable for leadership studies to continue using qualitative methods due to the complexity of the discipline—and not just at the beginning stages of theory development (Conger, 1998). Qualitative research that unpacks complex constructs has gained momentum in the past couple of decades (Parry, 2014). That said, qualitative research too has its own set of challenges. Conger (1998) argues that qualitative research relies too heavily on interviewing. Single-source data poses a threat to the rigor of qualitative research, just as it does with quantitative methods. More tools in the qualitative researcher toolbox, such as observation and life histories, must be utilized. The challenge then becomes organizing the volumes of data that multiple sources of qualitative data can create.

Despite the criticisms discussed here, we wish to emphasize that methods used should always be driven by the research question. Also, no research study is perfect, and there is a great deal of good research advancing the field both in content and methodological design. This section of our text highlights some of the trends that researchers should consider with the intent of making their work better. The difference between superior performance and exceptional performance is often not a big idea or quantum leap but making small gains in multiple areas.

When describing bad Management theory, Ghoshal (2005) stated that “if the theory is wrong, the truth is preserved for discovery by someone else” (p.77). This statement is also valid for any research method, including leadership instruments. By adopting inclusiveness in research design, the leadership field can better advance our understanding of this complex phenomenon.

References

Antonakis, J., Schriesheim, C. A., Donovan, J. A., Gopalakrishna-Pillai, K., Pellegrini, E. K., & Rossomme, J. L. (2004). Methods for studying leadership. In J. Antonakis, A. T. Cianciolo, & R. J. Sternberg (Eds.), The nature of leadership (pp. 48–70). Sage Publications.

Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2010). On making causal claims: A review and recommendations. The Leadership Quarterly, 21(6), 1086–1120. https://doi.org/10.1016/j.leaqua.2010.10.010

Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2024). Causality and endogeneity: Problems and solutions. In D.V. Day (Ed.), The Oxford handbook of leadership and organizations (pp. 93–117). Oxford University Press.

Banks, G., Gooty, J., Ross, R., Williams, C., &Harrington, N. (2018). Construct redundancy in leader behaviors: A review and agenda for the future. Leadership Quarterly, 29(1), 236–251. https://doi.org/10.1016/j.leaqua.2017.12.005

Banks, G. C., Woznyj, H. M., & Mansfield, C. A. (2023). Where is “behavior” in organizational behavior? A call for a revolution in leadership research and beyond. The Leadership Quarterly, 34(1), article 101581. https://doi.org/10.1016/j.leaqua.2021.101581

Fischer, T., Hambrick, D. C., Sajons, G. B., & Van Quaquebeke, N. (2020). Beyond the ritualized use of questionnaires: Toward a science of actual behaviors and psychological states. The Leadership Quarterly, 31(4), article 101449. https://doi.org/10.1016/S1048-20)30076-X

Fischer, T., & Sitkin, S. B. (2023). Leadership styles: A comprehensive assessment and way forward. Academy of Management Annals, 17(1). https://doi.org/10.5465/annals.2020.0340

Fischer, T., Dietz, J., & Antonakis, J. (2024). A fatal flaw: Positive leadership style research creates causal illusions. The Leadership Quarterly, 35(3), article 101771. https://doi.org/10.5465/annals.2020.0340

Batistič, S., Černe, M., & Vogel, B. (2017). Just how multi-level is leadership research? A document co-citation analysis 1980–2013 on leadership constructs and outcomes. The Leadership Quarterly, 28(1), 86–103. https://doi.org/10.1016/j.leaqua.2016.10.007

Conger, J. A. (1998). Qualitative research as the cornerstone methodology for understanding leadership. The Leadership Quarterly, 9(1), 107–121. https://doi.org/10.1016/S1048-9843(98)90044-3

Hunter, S. T., Bedell-Avers, K. E., & Mumford, M. D. (2007). The typical leadership study: Assumptions, implications, and potential remedies. The Leadership Quarterly, 18(5), 435–446. https://doi.org/10.1016/j.leaqua.2007.07.001

Lonati, J., & Antonakis, J. (2023). Endogeneity and Endogenous Theorizing. In Goethals, G. R., Allison, S. T., & Sorenson, G. J. (Eds.), The Sage encyclopedia of leadership studies (pp. 318-323). Sage Publications.

Parry, K., Mumford, M. D., Bower, I., & Watts, L. L. (2014). Qualitative and historiometric methods in leadership research: A review of the first 25 years of The Leadership Quarterly. The Leadership Quarterly, 25(1), 132–151. https://doi.org/10.1016/j.leaqua.2013.11.006

Podsakoff, P. M., & Podsakoff, N. P. (2019). Experimental designs in management and leadership research: Strengths, limitations, and recommendations for improving publishability. The Leadership Quarterly, 30(1), 11-33. https://doi.org/10.1016/j.leaqua.2018.11.002

Stentz, J. E., Plano Clark, V. L., & Matkin, G. S. (2016). Applying mixed methods to leadership research: A review of current practices: Corrigendum. The Leadership Quarterly, 27(4), 711. https://doi.org/10.1016/j.leaqua.2015.12.006

Sieweke, J., & Santoni, S. (2020). Natural experiments in leadership research: An introduction, review, and guidelines. The Leadership Quarterly, 31(1), article 101338. https://doi.org/10.1016/j.leaqua.2019.101338

Yukl, G. (1989). Managerial leadership: A review of theory and research. Journal of Management, 15(2), 251-289. https://doi.org/10.1177/014920638901500207