Opening Up the “Black Box” of Education Policy Research: Application of Theory-Based Impact Evaluation in Randomized Controlled Trial Studies

doi:10.16382/j.cnki.1000-5560.2020.08.003

Abstract

Abstract: Impact evalution aims to understand the impact of policies/interventions on the welfare of participants. The key is to establish a causal relationship between the two (what works) and to understand the mechanism of policies/interventions (why it works). Theory-based impact evaluation (TBIE) is widely believed to help answer the question of “why it works”. TBIE clarifies the intervention mechanisms by establishing a causal chain from input to output and then to impact, and using empirical data to test the validity of the theory and potential hypotheses that may play a role in the causal chain. This paper reviews and summarizes the core concepts and principles of TBIE, and uses examples to illustrate how to apply TBIE when conducting random controlled trials.

Key words: analysis of mechanism, theory of change, causal chain, impact evaluation

Tang Lei, Ma Jing, Liu Han, Yue Ai, Bai Yu, Meng Chun, Ma Chengjun. Opening Up the “Black Box” of Education Policy Research: Application of Theory-Based Impact Evaluation in Randomized Controlled Trial Studies[J]. Journal of East China Normal University(Educational Sciences), 2020, 38(8): 92-109.

References

白钰, 党瑞瑞, 郑丽娟, 刘国强, 王蕾. (2020). 中国农村中小学生自我控制力水平及其与教育产出的关系. 世界农业, (05), 12?19+38.
高玉娟, 白钰, 马跃, & 史耀疆. (2018). 正负效应的先来后到: 父母外出对留守儿童学业表现的影响研究. 劳动经济研究,6(3),98?114
甘雨, 岳爱, 高嘉琪, 汤蕾, & 罗仁福. (2019). 贫困农村地区婴幼儿抚养人抑郁症状的危险因素. 中国心理卫生杂志, 33(10), 751?755.
李英, 贾米琪, 郑文廷, 汤蕾, & 白钰. (2019). 中国农村贫困地区儿童早期认知发展现状及影响因素研究. 华东师范大学学报(教育科学版),37(3),21?36
李志艳. (2015). 公益的方法. 北京: 社会资源研究所.
史耀疆, 王欢, 罗仁福, 张林秀, 刘承芳, & 易红梅, 等. (2013). 营养干预对陕西贫困农村学生身心健康的影响研究. 中国软科学,4(10),53?63
史耀疆, 薛浩, 王欢, Sean, Sylvia, & Alexis, et al. (2016). 中国农村医生医疗服务质量的测量——基于标准化病人法的实验研究. 劳动经济研究,4(2),48?71
史耀疆, 张林秀, 常芳, 刘涵, 唐彬, 高秋风, 关宏宇, 聂景春, 杨洁, 白钰, 李英, 汤蕾, 岳爱, 茹彤. (2020). 教育精准扶贫中随机干预实验的中国实践与经验. 华东师范大学学报(教育科学版), 38(8), 1—67.
岳爱, 蔡建华, 白钰, 汤蕾, 史耀疆, & 罗仁福, 等. (2019). 中国农村贫困地区0—3岁婴幼儿面临的挑战及可能的解决方案. 华东师范大学学报(教育科学版),37(3),5?20
张林秀, 刘承芳, 罗仁福, 史耀疆, 岳爱, & Rozelle, S. (2009). 农村小学生的贫血状况及其对身体发育和学业表现的影响: 对陕西省贫困县的实证研究. 中国教育经济学学术年会,1756?1766
张林秀, 易红梅, 罗仁福, 刘承芳, & 史耀疆. (2014). 中等收入陷阱的人力资本根源: 中国案例. 中国人民大学学报,28(3),8?18
Attanasio, O., Cattan, S., Fitzsimons, E., Meghir, C., & Rubio-Codina, M. (2018). Estimating the production function for human capital: Results from a randomized control trial in Colombia. Cowles Foundation Discussion Papers.
Attanasio, O., Fernandez, C., Fitzsimons, E., Grantham-McGregor, S., Meghir, C., and Rubio-Codina, M. (2014). Using the infrastructure of a conditional cash transfer program to deliver a scalable integrated early child development program in Colombia: Cluster randomized controlled trial. BMJ, 349(oct08 9), 6126
Bai, Y., Neubauer, M., Ru, T., Shi, Y., Kenny, K., & Rozelle, S. (2019). Impact of second-parent migration on student academic performance in northwest China and its implications. The Journal of Development Studies, 1?18
Banerjee, A. V., & Duflo, E. (2009). The experimental approach to development economics. Annual Review of Economics, 1(1), 151?178
Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173?1182
Bauer, D. J., Preacher, K. J., & Gil, K. M. (2006). Conceptualizing and testing random indirect effects and moderated mediation in multilevel models: New procedures and recommendations. Psychological Methods, 11, 142?163
Black, R. E., Victora, C. G., Walker, S. P., Bhutta, Z. A., Christian, P., De Onis, M., .. & Uauy, R. (2013). Maternal and child undernutrition and overweight in low-income and middle-income countries. The Lancet, 382(9890), 427?451
Bonell, C., Fletcher, A., Morton, M., Lorenc, T., & Moore, L. (2012). Realist randomised controlled trials: A new approach to evaluating complex public health interventions. Social Science & Medicine, 75(12), 2299?2306
Britto, P. R., Lye, S. J., Proulx, K., Yousafzai, A. K., & Bhutta, Z. A. (2016). Nurturing care: Promoting early childhood development. The Lancet, 389(10064)
Brown, C. H., & Liao, J. (1999). Principles for designing randomized preventive trials in mental health: An emerging developmental epidemiology paradigm. American Journal of Community Psychology, 27(5), 673?710
Campbell, F. A., Pungello, E. P., Miller-Johnson, S., Burchinal, M., & Ramey, C. T. (2001). The development of cognitive and academic abilities: Growth curves from an early childhood educational experiment. Developmental Psychology, 37(2), 231
Campis, L. K., Lyman, R. D., & Prentice-Dunn, S. (1986). The parental locus of control scale: Development and validation. Journal of Clinical Child Psychology, 15(3), 260?267
Carvalho, S., & White, H. (2004). Theory-based evaluation: The case of social funds. American Journal of Evaluation, 25(2), 141?160
Carvalho, S., Perkins, G., White, H. (2004). Social funds: Participation, social capital and sustainability. Journal of International Development, 14, 611?625
Carvalho, S., Perkins, G., White, H., Bahnson, C., Kapoor, A. G., & Webervenghaus, S. (2002). Social funds: Assessing effectiveness. The World Bank.
Chen, H. T. (1990). Theory-driven evaluations. Thousand Oaks, CA: Sage.
Chen, H., Rossi, P. H., & Policy Studies Organization. (1992). Using theory to improve program and policy evaluations. Greenwood Press.
Cogill, S. R., Caplan, H. L., Alexandra, H., Robson, K. M., & Kumar, R. (1986). Impact of maternal postnatal depression on cognitive development of young children. Br Med J (Clin Res Ed), 292(6529), 1165?1167.
Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schroter, D. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199?226
Currie, J., & Almond, D. (2011). Human capital development before age five. Handbook of labor economics, 4, 1315?1486
Duflo, E. (2006). Field experiments in development economics. In R.Blundel et al (eds), Advances in economics and econometrics: Theory and applications. Cambridge University Press.
Elster, J. (1989). Nuts and bolts for the social sciences. Cambridge University Press.
Engle, P. L., Black, M. M., Behrman, J. R., De Mello, M. C., Gertler, P. J., Kapiriri, L., .. & International Child Development Steering Group. (2007). Strategies to avoid the loss of developmental potential in more than 200 million children in the developing world. The Lancet, 369(9557), 229?242
Fernald, L. C. H., Prado, E. L., & Kariger, P. K. (2017). Toolkit for measuring early childhood development in low- and middle-income countries. World Bank Publications.
Frazier, P. A., Tix, A. P., & Barron, K. E. (2004). Testing moderator and mediator effects in counseling psychology research. Journal of Counseling Psychology, 51(1), 115?134
Frongillo, E. A., Tofail, F., Hamadani, J. D., Warren, A. M., & Mehrin, S. F. (2014). Measures and indicators for assessing impact of interventions integrating nutrition, health, and early childhood development. Annals of the New York Academy of Sciences, 1308(1), 68?88
Gertler, P. (2016). Impact evaluation in practice (Second Edition). World Bank Group.
Gertler, P. J., Heckman, J. J., Pinto, R. R., Zanolini, A., Vermeerch, C., & Walker, S., et al. (2011). Labor market returns to early childhood stimulation: A 20-year followup to an experimental intervention in Jamaica. National Bureau of Economic Research, Inc.
Glennerster, R., & Takavarasha, K. (2013). Running randomized evaluations: A practical guide. Princeton University Press.
Grantham-McGregor, S., Cheung, Y. B., Cueto, S., Glewwe, P., Richter, L., & Strupp, B. (2007). Developmental potential in the first 5 years for children in developing countries. The Lancet, 369(9555), 60?70
Guan, H., Wang, H., Huang, J., Du, K., Zhao, J., Boswell, M., … Rozelle, S. (2018). Health seeking behavior among rural left-behind children: Evidence from Shaanxi and Gansu provinces in China. International Journal of Environmental Research and Public Health, 15(5), 883
Hamadani, J. D., Tofail, F., Hilaly, A., Hu da, S. N., & Grantham-Mcgregor, S. M. (2010). Use of family care indicators and their relationship with child development in Bangladesh. Journal of Health Population & Nutrition, 28(1), 23?33
Heckman, J. J., & Smith, J. A. (1995). Assessing the case for social experiments. The Journal of Economic Perspectives, 9(2), 85?110
Horton, R. (2008). Maternal and child undernutrition: An urgent opportunity. The Lancet, 371(9608), 179?179
Jamal, F., Fletcher, A., Shackleton, N., Elbourne, D., Viner, R., & Bonell, C. (2015). The three stages of building and testing mid-level theories in a realist RCT: A theoretical and methodological case-example. Trials, 16(1), 466
Jones, D. S., & Podolsky, S. H. (2015). The history and fate of the gold standard. The Lancet, 385(9977), 1502?1503
Justman, M. (2018). Randomized controlled trials informing public policy: Lessons from project STAR and class size reduction. European Journal of Political Economy, 54, 167?174
Khandker, S. R., Koolwal, G. B., & Samad, H. A. (2010). Handbook on impact evaluation: Quantitative methods and practices. The World Bank.
Lai, F., Zhang, L., Hu, X., Qu, Q., Shi, Y., & Qiao, Y., et al. (2013). Computer assisted learning as extracurricular tutor? Evidence from a randomised experiment in rural boarding schools in Shaanxi. Journal of Development Effectiveness, 5(2), 208?231
Lee, H., Herbert, R. D., Lamb, S. E., Moseley, A. M., & McAuley, J. H. (2019). Investigating causal mechanisms in randomised controlled trials. Trials, 20(1), 524
Lin, J., Song, F., Yao, P., Yang, X., & Liu, L. (2007). Effect of vitamin A supplementation on immune function of well-nourished children suffering from vitamin A deficiency in China. European Journal of Clinical Nutrition, 62(12), 1412?1418
Lovejoy, M. C., Graczyk, P. A., O'Hare, E., & Neuman, G. (2000). Maternal depression and parenting behavior: A meta-analytic review. Clinical Psychology Review, 20(5), 561?592.
Luo, R., Yue, A., Zhou, H., Shi, Y., Zhang, L., & Martorell, R., et al. (2017a). The effect of a micronutrient powder home fortification program on anemia and cognitive outcomes among young children in rural China: A cluster randomized trial. BMC Public Health, 17(1), 738
Luo, R., Jia, F., Yue, A., Zhang, L., Lyu, Q., Shi, Y., .. & Rozelle, S. (2017b). Passive parenting and its association with early child development. Early Child Development and Care, 189(10), 1709?1723
Ma, X., Zhou, Z., Yi, H., Pang, X., Shi, Y., Chen, Q., .. & Liu, Y. (2014). Effect of providing free glasses on children’s educational outcomes in China: Cluster randomized controlled trial. BMJ, 349, 5740
Mahoney, J. (2001). Beyond correlational analysis: Recent innovations in theory and method. Sociological Forum.
Marchal, B., Westhorp, G., Wong, G., Van Belle, S., Greenhalgh, T., Kegels, G., & Pawson, R. (2013). Realist RCTs of complex interventions–An oxymoron. Social Science & Medicine, 94, 124?128
Merton, R. K. (1967). Social theory and social structure. Free Press, a division of Macmillan Pub.
Mo, D., Luo, R., Liu, C., Zhang, H., Zhang, L., Medina, A., & Rozelle, S. (2014a). Text messaging and its impacts on the health and education of the poor: Evidence from a field experiment in rural China. World Development, 64, 766?780
Mo, D., Zhang, L., Luo, R., Qu, Q., Huang, W., & Wang, J., et al. (2014b). Integrating computer-assisted learning into a regular curriculum: Evidence from a randomised experiment in rural schools in Shaanxi. Journal of Development Effectiveness, 6(3), 300?323
Neiss, M., & Rowe, D. C. (2000). Parental education and child’s Verbal IQ in adoptive and biological families in the national longitudinal study of adolescent health. Behavior Genetics, 30(6), 487?495
Nobre-Lima, L., Vale-Dias, M. da L., Mendes, T. V., Mónico, L., & MacPhee, D. (2014). The Portuguese version of the Knowledge of Infant Development Inventory-P (KIDI-P). European Journal of Developmental Psychology, 11(6), 740?745
Paulsell, D., Avellar, S., Martin, E. S., & Del Grosso, P. (2010). Home visiting evidence of effectiveness review: Executive summary. Mathematica Policy Research.
Pawson, R., & Tilley, N. (1997). Realistic evaluation. Sage.
Pearl, J., & Mackenzie, D. (2018). The book of why: The new science of cause and effect. Basic Books.
Peterson, C. A., Luze, G. J., Eshbaugh, E. M., Jeon, H. J., & Kantz, K. R. (2007). Enhancing parent–child interactions through home visiting: Promising practice or unfulfilled promise?. Journal of Early Intervention, 29(2), 119?140
Reynolds, A. J., Temple, J. A., & Ou, S. R. (2010). Preschool education, educational attainment, and crime prevention: contributions of cognitive and noncognitive skills. Children & Youth Services Review, 32(8), 1054?1063
Riley, S., Brady, A. E., Goldberg, J., Jacobs, F., & Easterbrooks, M. A. (2008). Once the door closes: Understanding the parent-provider relationship. Children & Youth Services Review, 30(5), 597?612
Rogers, P. (2014). Theory of change, methodological briefs: Impact evaluation 2. UNICEF Office of Research, Florence.
Rogers, P. J. (2008). Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation the International Journal of Theory Research & Practice, 14(1), 29?48
Roggman, L. A., Cook, G. A., Peterson, C. A., & Raikes, H. H. (2008). Who drops out of early head start home visiting programs?. Early Education and Development, 19(4), 574?599
Sanson, A., Hemphill, S., Yagmurlu, B., & McClowry, S. G. (2002). Temperament and social development. In Smith, P. K. & Hart, C. H. (Eds.), Blackwell handbook of childhood social development. Oxford: Blackwell.
Shah, M. K., Kieffer, E. C., Choi, H., Schumann, C., & Heisler, M. (2015). Mediators and moderators of the effectiveness of a community health worker intervention that improved dietary outcomes in pregnant Latino women. Health Education & Behavior, 42(5), 593?603
Stanhope, L., Bell, R. Q., & Parker-Cohen, N. Y. (1987). Temperament and helping behavior in preschool children. Developmental Psychology, 23(3), 347?353
Stanley, Z., Paul, A., Claudia, S., Kojo Yeboah, A., George, Y., & Ana, P. (2003). Home-fortification with iron and zinc sprinkles or iron sprinkles alone successfully treats anemia in infants and young children. Journal of Nutrition, 133(4), 1075?1080
Sylvia, S., Warrinnier, N., Luo, R., Yue, A., Attanasio, O., Medina, A., & Rozelle, S. (2018). From quantity to quality: Delivering a home-based parenting intervention through China’s family planning cadres. Working paper. Rural Education Action Program, Stanford University.
Van Belle, S., Wong, G., Westhorp, G., Pearson, M., Emmel, N., Manzano, A., & Marchal, B. (2016). Can “realist” randomised controlled trials be genuinely realist?. Trials, 17(1), 313?330
Verch, K. (2017). Premeira Infancia Melhor: Transforming the attention towards the first years of life in Latin America: Challenges and achievements of a public policy in southern Brazil. Inter-American Development Bank.
Walker, S. P., Wachs, T. D., Grantham-Mcgregor, S., Black, M. M., Nelson, C. A., & Huffman, S. L., et al. (2011). Inequality in early childhood: Risk and protective factors for early child development. The Lancet, 378(9799), 1325?1338
Walker, S. P., Wachs, T. D., Meeks Gardner, J., Lozoff, B., Wasserman, G. A., Pollitt, E., & Carter, J. A. (2007). Child development: Risk factors for adverse outcomes in developing countries. The Lancet, 369(9556), 145?157
Wallander, J. L., Biasini, F. J., Thorsten, V., Dhaded, S. M., Jong, D. M. D., & Chomba, E., et al. (2014). Dose of early intervention treatment during children’s first 36 months of life is associated with developmental outcomes: An observational cohort study in three low/low-middle income countries. BMC Pediatrics, 14(1), 1?11
Weiss, C. H. (1995a). Nothing as practical as a good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families, in J. P. Connell, A. C. Kubisch, L. B. Schorr and C. H. Weiss (eds) New approaches to evaluating community initiatives, pp. 65?69. Washington, DC: Aspen Institute.
Weiss, C. H. (1995b). New approaches to evaluating comprehensive community initiatives. The Aspen Institute, Washington, DC.
Weiss, C. H. (1998). Evaluation: Methods for studying programs and policies. Upper Saddle River, NJ: Prentice-Hall.
White, H. (2009). Theory-based impact evaluation: Principles and practice. Journal of Development Effectiveness, 1(3), 271?284
Yue, A., Bai, Y., Shi, Y., Luo, R., Rozelle, S., Medina, A., & Sylvia, S. (2020). Parental Migration and Early Childhood Development in Rural China. Demography.
Yue, A., Shi, Y., Luo, R., Chen, J., Garth, J., Zhang, J.,.. & Rozelle, S. (2017). China’s invisible crisis: Cognitive delays among rural toddlers and the absence of modern parenting. The China Journal, 78(1), 50?80
Yue, A., Shi, Y., Luo, R., Wang, B., Weber, A., Medina, A., ... & Rozelle, S. (2019). Stimulation and early child development in China: Caregiving at arm’s length. Journal of Developmental & Behavioral Pediatrics.
Zellner, A., & Rossi, P. E. (1984). Bayesian analysis of dichotomous quantal response models. Journal of Econometrics, 25(3), 365?393
Zhou, H., Sun, S., Luo, R., Sylvia, S., & Rozelle, S. (2016). Impact of text message reminders on caregivers’ adherence to a home fortification program against child anemia in rural western China: A cluster-randomized controlled trial. American Journal of Public Health, 106(7), 1?7
Zhang, S., Dang, R., Yang, N., Bai, Y., Wang, L., Abbey, C., & Rozelle, S. (2018). Effect of caregiver’s mental health on early childhood development across different rural communities in China. International Journal of Environmental Research and Public Health, 15(11), 2341.