Participatory research design methodologies for inclusive evaluation of extension program effectiveness

Introduction

A fundamental methodological tension runs through extension program evaluation: the people best positioned to assess whether a program worked the farmers themselves are typically the least involved in designing how that assessment takes place [1] . Conventional evaluations rely on external experts who define indicators, collect data, analyze results, and deliver reports to program managers. Farmers appear in these evaluations as respondents and data points, not as co-designers of the evaluation itself [2] . This extractive approach produces technically defensible reports that often sit unread, partly because the people who need to act on them had no hand in shaping the questions [3] . Participatory evaluation offers an alternative methodology. Rooted in Chambers' [4] participatory rural appraisal tradition, it positions stakeholders as co-researchers who collaborate on problem framing, indicator selection, data collection, and interpretation. Cousins and Whitmore [5] distinguished between practical participatory evaluation, aimed at increasing use of findings, and transformative participatory evaluation, aimed at empowering marginalized groups. Both traditions argue that involving stakeholders improves evaluation quality, relevance, and uptake [6] . Yet rigorous comparisons of participatory versus conventional evaluation are scarce in agricultural extension contexts. This research addressed that gap by implementing both methodologies across matched extension programs in Turkmenistan and comparing their processes and outcomes [7] .

Conceptual Model

The research integrated Patton's [8] utilization-focused evaluation framework with Fetterman's [9] empowerment evaluation model. Patton argues that evaluation use depends on stakeholder engagement throughout the evaluation process; Fetterman adds that evaluation itself can build community capacity for self-assessment. We hypothesized that participatory evaluation would produce higher stakeholder ownership (engagement pathway), more actionable findings (relevance pathway), and stronger recommendation implementation (utilization pathway) compared to conventional external evaluation. The model was tested across five evaluation phases: problem identification, indicator selection, data collection, analysis, and reporting.

Materials and Methods

Material

This action research was conducted through Turkmen Agricultural University across Ahal (38.95°N, 57.74°E) and Lebap (38.99°N, 63.57°E) Provinces from January to December 2022. These provinces feature arid continental climate with irrigated agriculture dominated by cotton, wheat, and horticulture. Eighteen extension programs were paired into nine matched sets based on program type, duration, and participant demographics. Within each pair, www.extensionjournal.com one was evaluated using participatory methodology (n = 9, involving 168 farming households) and the other using conventional external evaluation (n = 9, involving 156 households). A total of 324 farming households, 36 extension agents, and 18 program managers participated. Ethical clearance was obtained from the Turkmen Agricultural University Research Committee (TAU-RC-2021-014, approved December 8, 2021).

Field Research Design

Participatory evaluations followed a five-phase, 12-month timeline (Figure 2). Phase 1 involved stakeholder mapping and recruitment of evaluation working groups comprising farmers, agents, and managers. Phase 2 used community workshops to define evaluation questions collaboratively. Phase 3 involved joint development of locally meaningful indicators. Phases 4 and 5 combined participatory data collection-including farmer-led focus groups, community scoring exercises, and photo-voice methods-with joint analysis workshops. Conventional evaluations followed standard practice: external evaluators designed instruments, collected data through structured surveys, analyzed results independently, and submitted written reports to program managers.

Methods

Data were analyzed in SPSS version 27. Stakeholder ownership was measured through a validated 12-item Evaluation Ownership Scale scored 1-5. Recommendation implementation was tracked at six months post-evaluation through structured observation. Mann-Whitney U tests compared ordinal outcomes between participatory and conventional programs. Chi-square tests assessed participation rate differences across evaluation phases. Agent confidence was measured pre-and post-evaluation using paired t-tests. Qualitative data from 18 stakeholder debriefings were analyzed using directed content analysis. Significance was set at p<0.05. Participatory evaluations outperformed conventional ones on all quality measures (Table 1). Stakeholder ownership was 2.6 times higher (4.1 vs 1.6), and recommendation implementation reached 67.4% compared to 26.1%. The trade-off was duration: participatory evaluations took 11.4 months versus 4.2 for conventional ones. Participation was uneven across phases (Figure 1). Farmers contributed most during problem identification (78.4%) and data collection (71.3%) but participation dropped to 34.8% during analysis and interpretation a phase where program managers' involvement was highest (67.2%). This analytical participation gap was the most frequently cited challenge in stakeholder debriefings. www.extensionjournal.com Stakeholder ownership was the strongest predictor of implementation (β = 0.42, Table 2), followed by farmer participation in the analysis phase (β = 0.31). Notably, once ownership and participation variables were entered, the methodology variable itself became non-significant (β = 0.18, p = 0.146), suggesting that participatory methodology works through the engagement it generates rather than through any inherent methodological superiority.

Results

Table 1: Evaluation

Discussion

The finding that methodology became non-significant once ownership and participation were controlled reveals the mechanism at work: participatory evaluation doesn't produce better findings through technical superiority, but because stakeholders who shape the evaluation invest in its conclusions [10] . This aligns with Patton's [8] core argument that evaluation use depends primarily on the personal factor-whether intended users are meaningfully involved throughout the process. The 67.4% implementation rate for participatory evaluations, compared to 26.1% for conventional ones, represents a practical return that may justify the longer timeline [11] . The analytical participation gap deserves particular attention. Farmer involvement dropped from 78.4% during problem identification to just 34.8% during analysisprecisely the phase where meaning is made from data [12] . Without analytical participation, farmers may own the questions but not the answers. Bridging this gap requires specific capacity building: training in basic data interpretation, visual analysis methods such as community mapping and scoring, and structured sense-making workshops where farmers interpret patterns alongside evaluators [13] . Extension agencies should plan for this capacity building rather than assuming participatory enthusiasm at early stages will naturally carry through to analytical work.

Limitations

The matched-pair design with nine pairs limits statistical power for detecting smaller effects. Programs were matched on observable characteristics, but unobserved factors (facilitator quality, community dynamics) may have influenced outcomes. The researcher served as participatory evaluation facilitator, introducing potential experimenter bias. Turkmenistan's centralized agricultural governance context may limit transferability to countries with decentralized extension systems. Longer follow-up periods would clarify whether implementation gains persist beyond six months.

Conclusion

Participatory evaluation methodologies produce dramatically higher stakeholder ownership, more actionable findings, and over twice the rate of recommendation implementation compared to conventional external approaches. The mechanism is engagement, not methodology: when stakeholders shape the evaluation, they invest in its conclusions. The critical bottleneck is the analytical phase, where farmer participation drops sharply unless specific capacity building is provided. For extension agencies in Turkmenistan and comparable contexts, the practical choice is between fast evaluations that generate ignored reports and slower participatory processes that generate acted-upon findings. The additional time investment in participatory methodology pays for itself through implementation rates that actually change extension practice.

References

1. Estrella M, Gaventa J. Who counts reality? Participatory monitoring and evaluation: a literature review. IDS Working Paper. 1998;70:1-70.
2. Guijt I, Gaventa J. Participatory monitoring and evaluation: learning from change. IDS Policy Briefing. 1998;12:1-8.
3. Weiss CH. Have we learned anything new about the use of evaluation? American Journal of Evaluation. 1998;19(1):21-33.
4. Chambers R. Whose reality counts? Putting the first last. London: Intermediate Technology Publications; 1997.
5. Cousins JB, Whitmore E. Framing participatory evaluation. New Directions for Evaluation. 1998;80:523.
6. Chouinard JA, Milley P. Uncovering the mysteries of inclusion: empirical and methodological possibilities in participatory evaluation. Evaluation and Program Planning. 2018;67:70-78.
7. King JA, Cousins JB. Something old, something new, something borrowed: participatory evaluation and traditional evaluation approaches. Evaluation Practice. 1995;16(3):269-277.
8. Patton MQ. Utilization-focused evaluation. 4th ed. Thousand Oaks (CA): Sage Publications; 2008.
9. Fetterman DM. Empowerment evaluation: knowledge and tools for self-assessment, evaluation capacity building, and accountability. Thousand Oaks (CA): Sage Publications; 2001.
10. Brandon PR, Fukunaga LL. The state of the empirical research literature on stakeholder involvement in program evaluation. American Journal of Evaluation. 2014;35(1):26-44.
11. Lennie J. Increasing the rigour and trustworthiness of participatory evaluations. Evaluation Journal of Australasia. 2006;6(1):27-35.
12. American Journal of Evaluation. 2009;30(3):330-348.
13. Holte-McKenzie M, Forde S, Theobald S. Development of a participatory monitoring and evaluation strategy. Evaluation and Program Planning. 2006;29(4):365-376.
14. Zukoski AP, Luluquisen M. Participatory evaluation: what is it? Why do it? What are the challenges? Community-Based Public Health Policy and Practice. 2002;5:1-6.
15. Cracknell BE. Evaluating development aid: issues, problems, and solutions. Thousand Oaks (CA): Sage Publications; 2000.
16. Birner R, Davis K, Pender J, Nkonya E, Anandajayasekeram P. From best practice to best fit: a framework for designing and analyzing pluralistic agricultural advisory services. Journal of Agricultural Education and Extension. 2009;15(4):341-355.