Bayesian estimation of impact in experimental interventions with continuous outcome

Impact evaluation in experimental intervention is an estimation of how design and implementation strategies affect the overall outcome and effectiveness of interventions by establishing causation. It provides the approach for determining the “Average treatment effect” and the “Average effect of treatment on the treated” otherwise known as impact. One of several approaches to estimating impact in experimental intervention is the use of the Difference-in-Differences (DID) estimation procedure based on classical regression approach which often overstates reality and rely on assumptions that are often violated in real life. Bayesian estimation is popular in literature but not widely applied in evaluation of experimental intervention. This study developed an approach for estimating impact based on the Bayesian estimation procedure. It derived a distribution for the difference of difference variable when outcome is continuous and normally distributed using the convolution procedure. The likelihood of distribution of the difference of difference variable was combined with the normal prior, at a specific value of the prior hyperparameter obtained from previous study, to estimate the posterior distribution. Using data from computer simulation and secondary data, the posterior mean was estimated. Also, classical regression approach was used to estimate the impact. Results from the Bayesian approach produced lower impact estimate and lower mean square error compared with the classical approach. This study provides a better alternative to estimating impact in experimental interventions.

Illustrations explain the paper; charts are replotted from the paper's own table values.

AI-generated illustration

↗ Enlarge

Bayesian Posterior Estimation FlowClose ×

AI-generated conceptual illustration (Abstract mathematical probability distributions). It explains the idea and does not show study data.

Open full-size image ↗

Replotted from source data

↗ Enlarge

𝜽̂ and MSE: Classical regression... vs Bayesian estimation...Close ×

Replotted from Table 2 (page 7) of the original paper. Every bar is the paper's own table value.

Open full-size image ↗

Introduction

Interventions are targeted towards desired outcomes such as improvement in state of health, nutrition status, behavior change, improvement in school performance, reduction in susceptibility and efficacy of new treatments among others. Impact evaluation in experimental intervention is an estimation of how design and implementation strategies affect the overall outcome and effectiveness of interventions by establishing causation [1,2] . It provides the approach for determining the "Average treatment effect" and the "Average effect of treatment on the treated" otherwise known as "impact". The average treatment effect is the average impact of the intervention across all the individuals in the population of interest. It is defined as the average impact of individual in that population across the population of the intervention and the "Average effect of treatment on the treated" is the impact of the intervention on those exposed to it [2] . One of several approaches to estimating impact in experimental intervention is the use of the Difference-in-Differences (DID) estimation procedure. The DID provide the average impact by comparing the average changes in outcomes before and after implementation between the group who received the intervention and those who did not [2] . In estimating impact in experimental intervention, the classical approach for the obtaining the Difference in Difference estimator is based on the regression analysis [2] . The type of regression depends on the type of the outcome variable. A multiple regression model, with three covariates including exposure variable, time variable and the interaction variable for intervention and time, is developed, with the assumption that effects of possible confounders are differenced out under randomization [2] . Model parameters are estimated using Maximum Likelihood Estimation or Least Squares procedures.

Significance of each parameter is tested using the conventional t test and decision is based on p-value and confidence interval. The coefficient of the interaction variable is the impact parameter that is, the "Average impact of the intervention on the exposed to it" [2] . Bayesian estimation is popular in literature but not widely applied in experimental intervention. Bayesian statistics is generally dependent on two principal elements which are the prior distribution and posterior probability distributions. Prior probability distribution of an unknown quantity represents the information about an unknown parameter that is incorporated with new data to estimate the posterior distribution. That is, the probability that explains beliefs about a quantity. This belief can either be from previous studies with evidence or subjective judgments which may include expert opinion or researcher's perception. The unknown quantity can either be a parameter, latent variable or missing value which is basically random. An approach that treats parameters as random and not fixed has been shown to produce a more robust estimate of population parameters [3] . Results from Bayesian analysis can be transformed into probabilities that can help policy makers determine the effectiveness of a program or policy compared to the classical approach which utilizes hypothesis and P-values. Some of the first works on application of Bayesian approach to experimental intervention can be traced to Lindley and Smith in 1972, Jackson, Novic and Thayer in 1971, Novick, Lewis and Jackson in 1973, and Wang in 1977 [4,5,6,7] . More recent work is the proposition by Finucane in 2019 which highlighted the benefit of Bayesian approach to social evaluation such as the challenges working with probability, priors, point estimate, interpretation, and sensitivity analysis [8] . None of these works provided theoretical framework, rather they identified the benefit of Bayesian approach and highlighted common challenges in Bayesian Statistical Inference. Matti et al., in 2017 applied the principle of Bayesian estimation in evaluation of behavior change intervention. The work provided basic introduction to Bayesian procedures and identified several advantages of the Bayesian over classical approach such as how both interpret probability. The works provided a step by step framework for Bayesian estimation using computer aided procedures rather than theoretical procedures. It did not emphasize on the difference in difference approach of impact estimation [9] . In inferential statistics, correct decision is highly likely when the researcher uses all the information available especially when making decisions under uncertainty. Classical statistical analysis restricts the information used in data analysis to those obtained from a set of clearly relevant and current data. Prior knowledge is not used except to suggest the choice of a population model to fit to the data. In experimental intervention, determination of the efficacy of a new drug, idea, procedure, or intervention are largely based on classical approaches. The widely used method of estimating numerical value of such claims do not incorporate prior knowledge from existing studies. For estimation of impact to confirm claims of improvement of new methods or procedures in experimental intervention, adopting method of analysis that incorporate prior knowledge becomes very important and helps to regulate such claims. Findings from this study will contribute to existing body of knowledge.

Materials and Methods

Materials and Methods

Classical approach to impact estimation

Regression analysis is one of the applicable approaches for estimating impact, when there are more than two variables under study. The parameters of the regression model are easily obtained using any of the statistical software with only the outcome of interest (Y), the Intervention status (treatment/Comparison) and the time (baseline, follow up). The DID estimate is the coefficient of the interaction between location and time. Test of significance is conducted to ascertain the estimates are significantly different from zero using the t test and confidence interval. The hypothesis to test is that Ho: 𝜃 3 = 0 vs Hi: 𝜃 3 ≠ 0. Decisions are based on the p value and confidence interval [2] .

Proposed Bayesian approach to impact evaluation

A Bayesian approach for estimation of impact in experimental intervention is proposed. First, a probability distribution was developed for the difference of difference and combined with the and the impact is estimated using Bayesian approach. Theoretically, the Bayesian impact estimate is the mean of the posterior distribution [10,11] . Therefore, this approach will derive the posterior distribution of the difference in difference when the outcome of interest is continuous and normally distributed.

Define

Define

Y -Continuous normally distributed random variable describing the outcome of interest,

Where Y p0outcome of interest before intervention among those exposed to intervention (baseline) Y p1outcome of interest after intervention among those exposed to intervention (follow-up) Y c0outcome of interest before intervention among those not exposed to intervention (baseline) Y c1outcome of interest after intervention among those not exposed to intervention (follow-up The Difference in Difference diagram according to Lance et al., 2014 [2] can be relabeled below. [2] .

From figure 1, it follows that

↗ Enlarge

Figure 1Close ×

Relabeling of the DID diagram from Lance et al., 2014 [2].

Open full-size image ↗

Original paper figureSource PDF · page 3 ↗

Where d -Difference of Difference

Recall that Ycontinuous normally distributed random variable describing the outcome of interest Among those exposed to intervention At follow up, 𝑌 𝑝1 ∼ 𝑁(𝜃 𝑝1 , 𝜎 𝑝1

Therefore, the distribution of the differences in the intervention group denoted by 𝑑 𝑝 = (𝑌 𝑝1 -𝑌 𝑝0 )is given by,

Let 𝑌 𝑝0 = (𝑌 𝑝1 -𝑑 𝑝 ) and substitute it for 𝑌 𝑝0 , we obtain,

Which is the distribution of the difference in outcome for the intervention group at follow up and baseline respectively.

Among the comparison group

Given that the distribution of the outcome variable at follow up is, 𝑌 𝑐1 ∼ 𝑁(𝜃 𝑐1 , 𝜎 𝑐1

2 ) with a probability density function

…10

And the distribution of the outcome variable at baseline is, 𝑌 𝑐0 ∼ 𝑁(𝜃 𝑐0 , 𝜎 𝑐0

2 ) with a probability density function,

…11

The differences in the comparison group, 𝑑 𝑐 = (𝑌 𝑐1 -𝑌 𝑐0 )Is given as,

It follows from equation 9 that,

Which is the distribution of the difference in outcome for the comparison at follow up and baseline respectively.

To determine the distribution of the impact, The difference in difference 𝑑 = (𝑑 𝑝 -𝑑 𝑐 ), such that

…15

𝜃 𝑝 = 𝜃 𝑝1 -𝜃 𝑝0 , 𝜃 𝑐 = 𝜃 𝑐1 -𝜃 𝑐0 and 𝜃 𝑑 = (𝜃 𝑝1 -𝜃 𝑝0 ) -(𝜃 𝑐1 -𝜃 𝑐0 ) Also𝜎 𝑝 2 = 𝜎 𝑝0 2 + 𝜎 𝑝1

…20

Using the normal prior, Prior of 𝜃 𝑑 is given by

…22

And the joint prior follow Normal and Gamma distributions such that:

…23

Hence, from 17, the posterior distribution is given by

…32

…32

…33

Which can also be written as,

Estimation of the posterior distribution based on secondary data

The posterior distribution for the continuous normally distributed response data was estimated using data from an Infant and Young Child Feeding and Nutrition intervention. Data was available for weight of children treated for Severe Acute Malnutrition (SAM) over a six weeks period using "Ready to use Therapeutic Food" (RUTF). This data was available for 50 children who took part in the study. During the intervention phase, each child was provided with RUTF daily and monitored for the duration of the intervention. Weight of each child was measured using a digital scale before commencement of the intervention and at the end of the six weeks intervention period. Also, another community used as a control where only raw food items were provided to families and not the RUTF. Children in these communities were measured before and after six weeks program. The objective of the study was to determine the impact of feeding RUTF on weight gain compared with children fed with regular food. Literature search on previous study was conducted to identify values of the prior hyperparameter. The study by Isanaka et al., (2009) reported impact of RTUF for treatment of malnutrition to be 0.22kg [12] . Hence values of the Normal hyperparameter 𝛽 ̃ was set to 0.22. Estimation of Posterior distributions was carried out using R.

Results and Discussions 3.1 Posterior estimation using simulated data

For continuous outcome, using simulated data and a normal prior, the mean of the distribution of the impact parameter at suggested value of θ ̃ were computed. Table 1 presents the result of analysis. Decision is based on Minimum Mean Square Error. Furthermore, classical regression analysis was conducted with the simulated data and results were compared. The Bayesian estimation approach provided a more conservative impact estimate and lower Mean Square Error compared with classical regression method as provided in table 1.

Method	𝜽̂	MSE
Classical Approach	2.9689	0.0723
Bayesian estimation approach with normal prior	2.1767	0.0015

Table text extracted from the original PDF; superscript letters are shown inline. The original table image is below.

View original table image

↗ Enlarge

Table 1Close ×

Result of Bayesian and Classical approach based on simulated data

Open full-size image ↗

Original paper tableSource PDF · page 7 ↗

Posterior estimation using secondary data

The hyperparameter θ ̃ for the normal prior distribution was set to 0.22, which returned an impact of 2.2669. Classical approach produced an impact of 2.5698 with higher MSE. Hence Bayesian approach showed that RUTF is about 2.3 times better than regular food for treatment of acute malnutrition in children compared with the classical approach which reported that RUTF is about 2.6 times better than regular food for management of Severe Acute Malnutrition in Children under 5 years.

Conclusion

The study developed a Bayesian procedure for estimating impact in experimental intervention for continuous outcome. Result from estimation based on Bayesian approach appears more reliable than those obtained from classical procedure.

Conflict of interest

The authors declare no conflict of interest.

Figures and tables from the paper

Method	𝜽̂	MSE
Classical regression Approach	2.5698	1.5525
Bayesian estimation procedure	2.2669	0.9530

Table text extracted from the original PDF; superscript letters are shown inline. The original table image is below.

View original table image

↗ Enlarge

Table 2Close ×

Comparison of Bayesian and Classical approach based on secondary data

Open full-size image ↗

Original paper tableSource PDF · page 7 ↗

1. Gertler PJ, Martinez S, Premand P, Rawling BL, Vermeersch MC. Impact Evaluation in Practice. Washington: World Bank Group 2016. doi:10.1596/978-1-4648-0779-4
2. Lance P, Guilkey D, Hattori A, Angeles G. How do we know if a program made a difference? A guide to statistical methods for program impact evaluation, Chapel Hill, North Carolina: MEASURE Evaluation 2014.
3. Adebayo SB. Bayesian Geoadditive Modelling of Breastfeeding Initiation in Nigeria. Journal of Applied Econometrics 2004;19(2):267-281.
4. Lindley DV, Smith AFM. Bayes Estimate for the Linear Model (with Discussion) Part 1. Journal of the Royal Statistical Society, Ser B 1972;34:1-41.
5. Jackson PH, Novick MR, Thayer DT. Estimating Regressions in m groups. British Journal of Mathematical and Statistical Psychology 1971;24:129-153. https://doi:10.1111/j.2044-8317.1971.tb00462.x
6. Ming-Mei Wang, Melvin R, Novick Gerald L, Isaacs, Dan Ozenne. A Bayesian Data Analysis System for the Evaluation of Social Programs, Journal of the American Statistical Association 1977;72:360a:711-722, DOI: 10.1080/01621459.1977.10479947
7. Finucane Mariel, John Deke. Moving Beyond Statistical Significance: Moving Beyond Statistical Significance: THE BASIE (Bayesian Interpretation of Estimates) Framework for Interpreting Findings from Impact Evaluations, OPRE Report # 201935, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services 2019.
8. Matti TJ, Heino Matti Vuorre, Nelli Hankonen. Bayesian evaluation of behavior change interventions: a brief introduction and a practical example, Health Psychology and Behavioral Medicine 2018;6(1):49-78. DOI: 10.1080/21642850.2018.1428102
9. Gelman A, Tuerlinckx F. Type S Error Rates for Classical and Bayesian Single and Multiple Comparison Procedures. Columbia University Working Paper. New York: Columbia University 2000.
10. Spiegelhalter D, Freedman LS, Parmar M. Bayesian Approaches to Randomized Trials. Journal of the Royal Statistical Society, Series A 1994, 357-416.
11. Isanaka S et al., Effect of preventive supplementation with ready-to-use therapeutic food on the nutritional status, mortality, and morbidity of children aged 6 to 60 months in Nigeria: a cluster randomized trial," Jama 2009;301(3):277-285.