Quantitative methods: Difference-in-differences and propensity score matching

Introduction

Impact assessment is a structured and systematic approach used to examine the social, economic, and behavioral changes that result from a specific program, policy, or project affecting individuals and communities (OECD, 2002) [10] . It enables policymakers, researchers, and development professionals to determine whether an intervention has successfully met its intended objectives while also identifying any unexpected or unintended effects that may have emerged during implementation (Becker, 2001) [4] . Furthermore, impact assessment strengthens accountability by ensuring that resources are used effectively and transparently. It promotes evidence-based decision-making, allowing governments and organizations to rely on empirical findings rather than assumptions when designing or modifying policies (Taylor, Bryan, & Goodrich, 2002). By systematically analyzing outcomes, it also contributes to refining the planning and implementation of future interventions. Significance of impact assessment extends beyond measuring results; it provides deeper insights into social consequences, enhances policy formulation and strategic planning, encourages active stakeholder participation, and supports long-term sustainability and social equity (Gertler et al., 2016) [8] .

Theoretical Background of Impact Assessment

Impact evaluation techniques are generally grouped into quantitative, qualitative, and mixed-method approaches. Quantitative approaches use numerical data and statistical tools to estimate the magnitude of change resulting from an intervention. In contrast, qualitative approaches focus on understanding people's perceptions, experiences, and social contexts through methods such as interviews, focus group discussions (FGDs), and participatory rural appraisal (PRA). Mixed-method designs integrate both quantitative and qualitative techniques to provide a more comprehensive and balanced understanding of program outcomes (Gertler et al., 2016) [8] . While experimental designs like Randomized Controlled Trials (RCTs) are considered the strongest method for establishing cause-and-effect relationships, they are not always practical or feasible in real-world development settings. In such situations, quasi-experimental methods particularly Difference-in-Differences (DiD) and Propensity Score Matching (PSM) offer reliable alternatives by enabling researchers to estimate causal impacts even in the absence of random assignment.

Difference-in-Differences (DiD) Method

The Difference-in-Differences (DiD) method estimates the causal impact of an intervention by comparing changes in outcomes over time between a treatment group (affected) and a control group (unaffected). It isolates the treatment effect by removing time-invariant unobserved heterogeneity (Angrist & Pischke, 2009) [2] . The method assumes that, in the absence of treatment, both groups would have followed parallel trends in outcomes. This assumption is critical for unbiased estimation. Mathematically, the DiD estimator is expressed as: where Y_T and Y_C denote average outcomes for treatment and control groups, respectively (Bertrand et al., 2004) [5] .

Key Features 1. Examines Pre-and Post-Intervention Changes:

Evaluates how outcomes in the treatment group differ before and after the implementation of the program.

Incorporates a Comparison Group:

Assesses the difference in changes between the treatment group and a control group that did not receive the intervention. 3. Determines Causal Effect: Aims to estimate the actual impact of a policy or program by isolating its specific contribution to observed changes. 4. Assumes Similar Trend Patterns: Relies on the assumption that, without the intervention, both treatment and control groups would have experienced comparable trends over time.

Accounts for Constant Unobserved Characteristics:

Reduces bias by controlling for factors that remain unchanged across time, such as inherent abilities, land quality, or other fixed attributes. 6. Requires Data from Multiple Time Periods: Depends on panel data or repeated cross-sectional data collected at least before and after the intervention.

Suitable When Random Assignment Is Not Feasible:

Particularly valuable in practical settings where conducting randomized controlled trials (RCTs) is not possible. Policy Implication: This result suggests that continuous farmer capacity-building and extension interventions can substantially improve dairy output and income. However, ensuring similar pre-intervention conditions and monitoring for external factors such as feed prices or disease outbreaks is crucial for accuracy.

Case Study 1: Application of DiD in Livestock Extension

Advantages of

Limitations of Difference-in-Differences (DiD) 1. Depends on the Parallel Trends Assumption:

The validity of results relies on both groups exhibiting similar trends before the intervention. If this assumption is violated, the estimates may be biased.

Does Not Address Time-Varying Unobserved Factors:

If unexpected changes affect one group differently over time, the estimated impact may be distorted.

Propensity Score Matching (PSM) Method

Propensity Score Matching (PSM), first developed by Rosenbaum and Rubin (1983) [11] , is a quantitative method used to estimate causal relationships in studies where random assignment is not possible. The technique aims to create comparability between treated and untreated groups by matching individuals who share similar characteristics based on a calculated propensity score. This score reflects the likelihood that a unit receives the treatment, given a set of observed variables, and is commonly estimated through logistic regression analysis. Several matching procedures are applied within the PSM framework, including Nearest Neighbor, Caliper, Radius, and Kernel matching methods (Caliendo & Kopeinig, 2008) [6] . Once matching is completed, researchers conduct balance tests to ensure that the covariates are similarly distributed across groups. The treatment effect is then determined by comparing the outcomes of the matched treated and control units (Austin, 2011) [3] . Therefore, direct comparison of participants vs. nonparticipants would be biased.

Key

To remove this bias, PSM was applied.

Research

Step 2 -Matching Technique

Farmers were matched using:

Nearest-Neighbor Matching (1:1, with replacement) Each trained farmer was matched with one untrained farmer with the closest propensity score. Matching caliper = 0.05 to avoid poor matches.

Step 3 -Balance Checking

• Standardized mean differences between groups were reduced (<10%). • Covariates became statistically similar. This confirmed good balance and reduced selection bias.

Step 4 -Average Treatment Effect on the Treated (ATT)

After matching Average Milk Yield (liters/day)

References

1. Abadie A. Semiparametric difference-in-differences estimators. Review of Economic Studies. 2005;72(1):119.
2. Angrist JD, Pischke JS. Mostly harmless econometrics: an empiricist's companion. Princeton: Princeton University Press; 2009.
3. Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research. 2011;46(3):399-424.
4. Becker H. Social impact assessment field handbook. USA; 2001.
5. Bertrand M, Duflo E, Mullainathan S. How much should we trust differences-in-differences estimates? Quarterly Journal of Economics. 2004;119(1):249-275.
6. Caliendo M, Kopeinig S. Some practical guidance for the implementation of propensity score matching. Journal of Economic Surveys. 2008;22(1):31-72.
7. Card D, Krueger AB. Minimum wages and employment: a case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review. 1994;84(4):772-793.
8. Gertler PJ, Martinez S, Premand P, Rawlings LB, Vermeersch CMJ. Impact evaluation in practice. 2nd ed. Washington (DC): World Bank; 2016.
9. Guo S, Fraser MW. Propensity score analysis: statistical methods and applications. 2nd ed. Thousand Oaks (CA): SAGE Publications; 2015.
10. OECD. Impact assessment: an overview. Paris: Organisation for Economic Co-operation and Development; 2002.
11. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41-55.
12. Rosenbaum PR, Rubin DB. Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. American Statistician. 1985;39(1):33-38.
13. Stuart EA. Matching methods for causal inference: a review and a look forward. Statistical Science. 2010;25(1):1-21.
14. Thoemmes F, Kim ES. A systematic review of propensity score methods in the social sciences. Multivariate Behavioral Research. 2011;46(1):90-118.
15. Wooldridge JM. Econometric analysis of cross-section and panel data. 2nd ed. Cambridge (MA): MIT Press; 2010.