Mixed-methods evaluation designs for agricultural extension services: A systematic review of integration quality and methodological rigor

Introduction

Agricultural extension evaluation has long been dominated by quantitative approaches focused on measurable outcomes like technology adoption rates, yield increases, and income changes [1] . These methods produce generalizable findings but frequently fail to capture the relational dynamics, local knowledge systems, and institutional contexts that determine whether extension programs achieve lasting impact [2] . Qualitative approaches address these gaps through in-depth exploration of farmer experiences and contextual factors, but their findings are difficult to scale and may lack the statistical credibility that policymakers expect [3] . Mixedmethods research, which deliberately combines quantitative and qualitative data collection and analysis within a single study, has been promoted as a way to capture both breadth and depth in extension evaluation [4] . The appeal of mixed-methods in extension evaluation is clear, yet adoption does not guarantee quality. Poorly integrated mixed-methods studies amount to two separate investigations packaged together, with quantitative and qualitative components running in parallel without meaningful interaction [5] . The distinction between genuine mixed-methods research and parallel mono-method studies sharing a single report has important implications for the validity and usefulness of evaluation findings [6] . This research systematically reviewed published mixedmethods evaluations of extension services to assess how well the field is implementing these designs. The objectives were to map the distribution of design types, evaluate integration quality and methodological rigor, and identify factors associated with higher-quality mixed-methods practice in extension evaluation [7] .

Theoretical Framework

The research applied Creswell and Plano Clark's mixedmethods typology [8] to classify evaluation designs and used the Good Reporting of A Mixed Methods Study (GRAMMS) framework [9] to assess reporting quality. The analysis was additionally informed by Teddlie and Tashakkori's integrative framework [10] , which argues that the defining feature of mixed-methods research is not merely the use of both data types but rather the quality of inference achieved through their combination.

Material and Methods

Material

A systematic search was conducted across six databases (Web of Science, Scopus, CAB Abstracts, AGRIS, PubMed, and ERIC) for articles published between January 2010 and December 2022. Search terms combined extension-related vocabulary with mixed-methods terminology. Inclusion www.extensionjournal.com criteria required studies to explicitly describe both quantitative and qualitative data collection within an evaluation of agricultural extension services. After screening 1,847 initial records, 169 articles met all criteria and were included in the review. The search and screening followed PRISMA 2020 guidelines.

Pilot Research

The scoring rubrics were pilot-tested on a random sample of 15 articles by two independent reviewers. Initial inter-rater agreement was moderate (kappa = 0.68), leading to rubric refinement and additional calibration sessions. After revision, inter-rater agreement on the remaining pilot articles improved to substantial levels (kappa = 0.81). Disagreements were resolved through discussion and the refined rubric was applied to the full sample.

Methods

Each article was scored on two validated rubrics: an integration quality scale (1-10) assessing how well quantitative and qualitative components connected at design, analysis, and interpretation stages; and a methodological rigor scale (1-10) covering sampling, data collection, analysis procedures, and reporting transparency. Design types were classified using Creswell and Plano Clark's typology. Correlation analysis examined the relationship between integration quality and methodological rigor. Independent samples t-tests compared scores between studies with and without explicit mixed-methods frameworks. All analyses were conducted using R version 4.2. Sequential explanatory designs were most common (28.4%), but exploratory sequential and multi-phase designs achieved higher integration and rigor scores. Only 31.4% of all studies reached high integration quality. Studies citing a named mixed-methods framework scored significantly higher on integration (mean 6.1 vs 4.5, p < 0.001). Formative evaluations achieved the highest mean integration score (6.1), while needs assessments scored lowest (5.4). Impact evaluations, despite being most common, showed moderate integration quality, suggesting that the pressure for statistical rigor in impact evaluation may crowd out attention to qualitative integration. A positive correlation (r = 0.71, p < 0.001) was found between integration quality and methodological rigor, indicating that studies investing in genuine data integration also tend to demonstrate stronger overall methodological practice.

Results

Discussion

The finding that only 31.4% of studies achieved high integration quality exposes a significant gap between the promise and practice of mixed-methods in extension evaluation. Most studies combined quantitative and qualitative components at the data collection stage but failed to integrate them meaningfully during analysis and interpretation [5] . This pattern produces studies that are mixed-methods in name but parallel mono-method in practice. The strong positive correlation between integration quality and methodological rigor suggests these are not competing priorities. Studies that invested in integration planning also paid greater attention to sampling, analysis transparency, and reporting quality [11] . This finding counters the concern that mixed-methods complexity necessarily compromises rigor and instead supports the view that methodological thoughtfulness operates across all aspects of research design [12] . Studies citing named mixedmethods frameworks scored 34.8% higher on integration, indicating that explicit methodological grounding improves practice. This practical finding suggests that extension evaluation training should include specific instruction in mixed-methods frameworks rather than treating mixedmethods as simply combining surveys with interviews [13] .

The particularly strong performance of multi-phase designs (mean integration 6.8) reflects the iterative nature of these approaches, where each phase builds on and integrates with preceding phases [14] .

Limitations

The review was limited to published English-language articles, excluding evaluations published in other languages or in grey literature. Scoring integration quality involves subjective judgment despite rubric standardization. The review period (2010-2022) may not capture recent improvements in mixed-methods practice. Publication bias likely means that included studies represent above-average methodological quality compared to unpublished evaluations.

Conclusion

This systematic review assessed 169 mixed-methods extension evaluation studies and found that while adoption of mixed-methods designs has grown substantially, the quality of integration between quantitative and qualitative components remains a persistent weakness. Most studies combine data types at collection but fail to connect them meaningfully during analysis and interpretation. Extension evaluation practitioners should explicitly adopt named mixed-methods frameworks, plan integration strategies at the design stage rather than retrospectively, and report integration procedures transparently. Training programs for extension evaluators should include specific modules on mixed-methods integration techniques. Journal reviewers and editors can contribute by requiring explicit description of integration procedures in mixed-methods manuscripts.

References

1. Birner R, Davis K, Pender J, et al. From best practice to best fit: A framework for designing and analyzing pluralistic agricultural advisory services. Journal of Agricultural Education and Extension. 2009;15(4):341355.
2. Chambers R, Pacey A, Thrupp LA. Farmer First: Farmer Innovation and Agricultural Research. London: Intermediate Technology Publications; 1989.
3. Patton MQ. Qualitative Research and Evaluation Methods. 4th ed. Thousand Oaks: SAGE; 2015.
4. Johnson RB, Onwuegbuzie AJ. Mixed methods research: A research paradigm whose time has come. Educational Researcher. 2004;33(7):14-26.
5. Bryman A. Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research. 2007;1(1):8-22.
6. Bazeley P. Issues in mixing qualitative and quantitative approaches to research. Research in Marketing. 2004;1:141-156. www.extensionjournal.com
7. Bamberger M, Rugh J, Mabry L. Real World Evaluation: Working Under Budget, Time, Data, and Political Constraints. 2nd ed. Thousand Oaks: SAGE; 2012.
8. Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. 3rd ed. Thousand Oaks: SAGE; 2018.
9. O'Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. Journal of Health Services Research and Policy. 2008;13(2):92-98.
10. Teddlie C, Tashakkori A. Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches. Thousand Oaks: SAGE; 2009.
11. Creamer EG. An Introduction to Fully Integrated Mixed Methods Research. Thousand Oaks: SAGE; 2018.
12. Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs. Health Services Research. 2013;48(6):2134-2156.
13. Greene JC. Mixed Methods in Social Inquiry. San Francisco: Jossey-Bass; 2007.
14. Morse JM, Niehaus L. Mixed Method Design: Principles and Procedures. Walnut Creek: Left Coast Press; 2009.
15. Anderson JR, Feder G. Agricultural extension. Handbook of Agricultural Economics. 2007;3:23432378.
16. Ivankova NV, Creswell JW, Stick SL. Using mixedmethods sequential explanatory design. Field Methods. 2006;18(1):3-20.
17. Hesse-Biber SN. Mixed Methods Research: Merging Theory with Practice. New York: Guilford Press; 2010.
18. Plano Clark VL, Ivankova NV. Mixed Methods Research: A Guide to the Field. Thousand Oaks: SAGE; 2016.
19. Schoonenboom J, Johnson RB. How to construct a mixed methods research design. KZfSS Kolner Zeitschrift fur Soziologie und Sozialpsychologie. 2017;69(2):107-131.
20. Mertens DM. Transformative Mixed Methods Research. Qualitative Inquiry. 2010;16(6):469-474.