Content analysis methods for evaluating quality and relevance of extension communication materials

Introduction

As one veteran extension specialist in Paramaribo observed, "We produce hundreds of pamphlets each year, but nobody checks whether farmers can actually read and use them" [2] . This remark captures a systemic blind spot in agricultural extension: organizations invest heavily in creating communication materials but rarely evaluate whether those materials meet basic quality standards for the audiences they intend to reach. Extension communication serves as the primary vehicle through which technical knowledge travels from research institutions to farming communities, yet the quality of that vehicle is seldom inspected [3] . Suriname's extension system produces materials in Dutch, Sranan Tongo, and several indigenous languages, targeting farmers across diverse agroecological zones from coastal rice paddies to interior agroforestry systems [4] . This linguistic and geographic diversity makes communication quality especially important-a leaflet written in formal Dutch at a university reading level will fail to reach farmers who communicate primarily in Sranan Tongo. Similarly, recommendations developed for coastal conditions may confuse interior farmers facing entirely different soil and rainfall patterns [5] . Content analysis-the systematic coding and evaluation of communication materials against defined criteria-offers a methodological solution, but its application to extension materials has been limited. Most published work focuses on mass media or health communication rather than agricultural advisory content [6] . The few agricultural applications have examined single dimensions like readability without addressing technical accuracy, cultural relevance, or gender inclusivity as simultaneous quality indicators [7] . A multi-dimensional approach is needed to capture the full picture of material quality. This research developed and applied a four-dimension content analysis framework-covering readability, accuracy, relevance, and inclusivity-to extension materials from multiple sources and formats in Suriname. The objectives were to compare quality across material types and source organizations, identify the dimensions where quality gaps are widest, and propose a replicable assessment model that extension systems can adopt for ongoing quality improvement. www.extensionjournal.com two farmer representatives-reviewed and refined indicators through two rounds of modified Delphi consultation, reducing the pool to 24 indicators across four dimensions [9] . Each dimension contained six indicators scored on a fivepoint scale (1 = very poor to 5 = excellent). The readability dimension assessed language complexity, sentence length, vocabulary level, logical flow, use of jargon, and formatting clarity. Accuracy covered factual correctness, evidence basis, currency of information, consistency with national recommendations, dosage/rate precision, and source attribution. Relevance measured local context fit, alignment with farmer needs, action orientation, seasonal appropriateness, problem-solution structure, and practical feasibility. Inclusivity evaluated gender sensitivity, cultural appropriateness, language accessibility, representation diversity, disability access, and age appropriateness. Pilot testing with 24 materials from a non-sample district yielded inter-rater reliability (Cohen's kappa) of 0.79 and Cronbach's alpha of 0.84 across all dimensions [10] .

Material and Methods

Material

This research was conducted from March 2022 through January 2023 at the Faculty of Agriculture, Anton de Kom University, Paramaribo, Suriname. Ethical approval was obtained from the university's Research Ethics Committee (Protocol AdeKU-REC/2022-008, approved January 24, 2022). A comprehensive collection strategy gathered 218 extension communication materials from four source categories: government extension services (n = 74), NGO programs (n = 62), research institutions (n = 41), and private sector providers (n = 41). Materials spanned six format types: printed leaflets (n = 48), technical bulletins (n = 36), radio scripts (n = 32), video content (n = 28), mobile application messages (n = 38), and social media posts (n = 36) [11] . Collection covered materials produced or distributed between January 2020 and June 2022.

Methods

Three trained coders independently assessed each material using the EMQAT. Coders received 20 hours of training including calibration exercises on practice materials until inter-rater agreement exceeded 0.75. Each material received scores across all applicable dimensions-visual quality was not assessed for audio-only formats like radio scripts. Scores were averaged across coders for analysis. Additionally, 86 farmers from four districts participated in structured feedback sessions where they reviewed a subset of materials (n = 48) and rated perceived usefulness and comprehensibility on five-point scales, providing a beneficiary validation check [12] . Quantitative analysis used SPSS version 26 with two-way ANOVA (source × format), independent samples t-tests, and Pearson correlations between expert ratings and farmer feedback scores. Significance was set at p < 0.05 [13] .

Ethical Considerations

Farmer feedback participants provided verbal informed consent. No personal data beyond basic demographic characteristics were collected from farmers. All materials were analyzed as institutional outputs rather than individual products, and results were reported at organizational category level to avoid identifying specific staff members. Organizations received aggregated feedback reports after the research concluded. 3.4 ± 0.7 3.1 ± 0.9 3.2 ± 0.8 2.6 ± 0.9 3.1 ± 0.6 Scores on 1-5 scale. ANOVA significant across sources for all dimensions (p < 0.001). NGO vs. Research: p < 0.001 overall.

Results

Comprehensive Interpretation

Quality varied dramatically across both source organizations and material formats. NGO materials scored highest overall (3.6), driven by strong readability and relevance scores, while research institution materials scored lowest (2.5) despite having the best technical accuracy (Table 1). This inverted pattern-high accuracy paired with low readability-captures the classic disconnect between scientific rigor and farmer accessibility. Figure 1 reveals that technical bulletins showed the same pattern at the format level: highest accuracy (4.3) but lowest readability (2.4). The heatmap (Figure 2) showed that NGO materials led on local context fit (4.1) and cultural sensitivity (3.8), while research materials scored below 3.0 on every criterion except action orientation. The validation analysis (Table 2) confirmed that the EMQAT dimensions predicted farmer perceptions: readability and relevance correlated most strongly with both usefulness (r = 0.68 and 0.72) and comprehensibility (r = 0.74 and 0.58), while technical accuracy showed the weakest farmer-level correlation (r = 0.41 for usefulness). This suggests that from the farmer's perspective, being able to understand and apply the material matters more than its scientific precision.

Discussion

The finding that research institution materials scored lowest overall despite having the best technical accuracy reveals a fundamental tension in extension communication.

Researchers write for peer review, not for farmer comprehension. The 2.1-point readability score for research materials means these products are essentially inaccessible to their intended audience-a finding consistent with Leeuwis and Van den Ban's observation that the language gap between researchers and farmers is one of the most persistent barriers to technology transfer [3] . NGO materials achieved the reverse pattern: good readability and relevance but only moderate accuracy. The ideal material would combine both, yet no source category achieved this balance.

The correlation analysis (Table 2) offers a practical insight for resource allocation in material development. Readability and relevance predicted farmer usefulness ratings far more strongly than technical accuracy did. This doesn't mean accuracy is unimportant-inaccurate advice can cause real harm. But it does suggest that extension organizations spending most of their quality control effort on technical review while neglecting readability and contextual relevance are misallocating their attention. A technically perfect bulletin that farmers can't read or apply produces zero impact [7] . The assessment model (Figure 3) provides a structured process that extension organizations can apply routinely. By scoring materials across all four dimensions before distribution, organizations can identify specific weaknesses and target revisions where they matter most. The integrated quality index allows comparison across materials and tracking of improvement over time [14] . The key is making quality assessment a routine institutional practice rather than an occasional research exercise.

Practical Applications

Extension organizations can adopt the EMQAT framework in three ways. First, as a pre-distribution screening tool: materials scoring below

Conclusion

Extension communication materials in Suriname showed wide quality variation, with NGO materials scoring highest and research institution materials lowest overall despite their superior technical accuracy. The persistent trade-off between accuracy and accessibility suggests that no single source category has mastered the full range of quality dimensions needed for effective farmer communication.

The content analysis model developed here gives extension organizations a practical tool for diagnosing and addressing quality gaps before materials reach farmers. Readability and relevance emerged as the strongest predictors of farmerperceived usefulness-a finding that should redirect quality control efforts toward these dimensions. Making systematic content evaluation a routine practice, rather than an occasional exercise, offers extension systems a low-cost pathway to improving the impact of every communication product they distribute.

References

1. Stichting Landbouwvoorlichting Suriname. Annual report on extension communication activities and material production. SLS Technical Reports. 2021;6(2):1-54.
2. Rogers EM. Diffusion of innovations. Free Press. 2003;5(1):1-551.
3. Ministerie van Landbouw Suriname. Agricultural sector profile: Extension service coverage and farmer demographics. LVV Publications. 2020;4(1):1-96.
4. Badiru IO. Review of small farmer access to agricultural information in Nigeria. Technical Centre for Agricultural and Rural Cooperation. 2006;1(1):1-28.
5. Krippendorff K. Content analysis: An introduction to its methodology. Sage Publications. 2018;4(1):1-472.
6. Hoffmann V, Gerster-Bentaya M, Christinck A, Lemma M. Handbook: Rural extension volume 1: Basic issues and concepts. Margraf Publishers. 2009;3(1):1-316.
7. Neuendorf KA. The content analysis guidebook. Sage Publications. 2017;2(1):1-456.
8. Dalkey NC, Helmer O. An experimental application of the Delphi method to the use of experts. Management Science. 1963;9(3):458-467.
9. McHugh ML. Interrater reliability: The kappa statistic. Biochemia Medica. 2012;22(3):276-282.
10. Algemeen Bureau voor de Statistiek Suriname. Agricultural organizations directory and output survey. ABS Publications. 2021;3(2):1-78.
11. Chambers R. Participatory rural appraisal: Analysis of experience. World Development. 1994;22(9):12531268.
12. Field A. Discovering statistics using IBM SPSS statistics. Sage Publications. 2018;5(1):1-816.
13. Mefalopulos P, Kamlongera C. Participatory communication strategy design: A handbook. FAO Communication for Development. 2004;2(1):1-132.