Identifying the “Best Evidence”: How to Use Meta-analysis to Conduct a Literature Review—A Case of STEM Education’s Effect on Students’ Academic Achievement

doi:10.16382/j.cnki.1000-5560.2020.06.005

Abstract

Abstract: Compared with the method of traditional subjective literature review, meta-analysis is more objective, normative, and can effectively sort out general conclusion from existing empirical researches. Therefore, it has become an important method to identify the “best evidence” in the process of evidence-based educational reform. So far there are few educational studies in China using the meta-analysis method to review the empirical literatures. In view of this, this research analyzes the impact of STEM education on students’ academic achievement as an example of how to use meta-analysis to synthesise existing empirical research. We found: 1) STEM education is conductive to students academic achievements (d=0.410); 2) factors like educational methods, students’ educational stage, location and sample size can affect the effects. The study provides the general evidence of effects of STEM education and explores how to obtain reliable evidence through a literature review to support educational reform.

Key words: evidence-based educational reform, STEM education, academic achievement, meta-analysis

Zeng Zhaobing, Yao Jijun. Identifying the “Best Evidence”: How to Use Meta-analysis to Conduct a Literature Review—A Case of STEM Education’s Effect on Students’ Academic Achievement[J]. Journal of East China Normal University(Educational Sciences), 2020, 38(6): 70-85.

References

卢谢峰, 唐源鸿, 曾凡梅. (2011). 效应量: 估计、报告和解释. 心理学探新,31(3),260-264
姚计海. (2017). “文献法”是研究方法吗?——兼谈研究整合法. 国家教育行政学院学报,19(7),89-94
张斌贤, 李曙光. (2015). 文献综述与教育学博士学位论文撰写. 学位与研究生教育,32(1),59-63
张天嵩, 董圣杰, 周支瑞. (2015). 高级Meta分析方法——基于Stata实现. 上海: 复旦大学出版社.
Acar, D., Tertemiz, N., & Tasdemir, A. (2018). The Effects of STEM Training on the Academic Achievement of 4th Graders in Science and Mathematics and their Views on STEM Training Teachers. International Electronic Journal of Elementary Education, 10(4), 505-513
Akinoglu, O., & Tandogan, R. O. (2007). The Effects of Problem-Based Active Learning in Science Education on Students’ Academic Achievement, Attitude and Concept Learning. Eurasia Journal of Mathematics, Science & Technology Education, 3(1), 71-81
An, D. (2012). A Meta-Analysis of the Effectiveness of STEM-Programs in the United States. Seoul: Sangmyung University.
Apugliese, A., & Lewis, S. T. (2017). Impact of instructional decisions on the effectiveness of cooperative learning in chemistry through meta-analysis. Chemistry Education Research and Practice, 18, 271-278
Araz, G. (2007). The Effect of Problem-Based Learning on the Elementary School Students' Achievement in Genetics. Department of Elementary Science and Mathematics Education.
Atkinson, R. D., & Mayo, M. J. (2010). Refueling the US Innovation Economy: Fresh Approaches to Science, Technology, Engineering and Mathematics (STEM) Education. The Information Technology & Innovation Foundation.
Barth, K. N. (2013). An Investigation of the Effects of Integrating Science and Engineering Content and Pedagogy in an Elementary School Classroom (Doctoral dissertation). Provo: Brigham Young University.
Becker, K., & Park, K. (2011). Integrative Approaches among Science, Technology, Engineering, and Mathematics (STEM) Subjects on Students' Learning: A Meta-Analysis. Journal of STEM Education: Innovations and Research, 12(5), 23-37
Bediou, B., Adams, D. M., Mayer, R. E., Tipton, E., & Green, C. S. (2018). Meta- Analysis of Action Video Game Impact on Perceptual, Attentional, and Cognitive Skills. Psychological Bulletin, 144(1), 77-110
Belland, B. R., Walker, A. E., Olsen, M. W., & Leary, H. (2015). A Pilot Meta-Analysis of Computer-Based Scaffolding in STEM Education. Educational Technology & Society, 18(1), 183-197
Belland, B. R., Walker, A. E., & Kim, N. J. (2017). A Bayesian Network Meta-Analysis to Synthesize the Influence of Contexts of Scaffolding Use on Cognitive Outcomes in STEM Education. Review of Educational Research, 87(6), 1042-1081
Bicer, A., Navruz, B., Capraro, R. M., Capraro, M. M., Oner, T., & Boedeker, P. (2015). STEM Schools VS. Non-STEM Schools: Comparing Students’ Mathematics Growth Rate on High-Stakes Test Performance. International Journal on New Trends in Education and Their Implications, 6(1), 138-150
Bicer, A., Capraro, R. M., & Capraro, M. M. (2017). Integrated STEM Assessment Model. EURASIA Journal of Mathematics, Science & Technology Education, 13(7), 3959-3968
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to Meta-Analysis. New Jersey: John Wiley & Sons, Ltd.
Cakici, Y., & Turkmen, N. (2013). An Investigation of the Effect of Project-Based Learning Approach on Children’s Achievement and Attitude in Science. The Online Journal of Science and Technology, 3(2), 10-17
Carbonneau, K. J., Marley, S. C., Selig, J. P. (2013). A Meta-Analysis of the Efficacy of Teaching Mathematics With Concrete Manipulatives. Journal of Educational Psychology, 105(2), 380-400
Carmichael, C. C. (2017). A State-By-State Policy Analysis of STEM Education for K-12 Public Schools (Ed.D. Dissertation). New Jersey: Seton Hall University.
Cervetii, G. N., Barber, J., Dorph, R., Pearson, D., & Goldschmidt, P. G. (2012). The Impact of an Integrated Approach to Science and Literacy in Elementary School Classrooms. Journal of Research in Science Teaching, 19(5), 631-658
Cheung, A., & Slavin, R. E. (2012). Effects of Educational Technology Applications on Reading Outcomes for Struggling Readers: A Best Evidence Synthesis. Center for Research and Reform in Education, Johns Hopkins University.
Cheung, A., & Slavin, R. E. (2013a). The Effectiveness of Educational Technology Applications on Mathematics Achievement in K-12 Classrooms: A meta-analysis. Educational Research Review, 9(1), 88-11
Cheung, A. C. K., & Slavin, R. E. (2013b). Effects of Educational Technology Applications on Reading Outcomes for Struggling Readers: A Best Evidence Synthesis. Reading Research Quarterly, 48(3), 277-299
Cheung, A., & Slavin, R. E. (2016). How Methodological Features Affect Effect Sizes in Education. Educational Researcher, 45(5), 283-292
Chow, S. L. (1988). Significance Test or Effect Size?. Psychological Bulletin, 103(1), 105-110
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
Cotabish, A., Dailey, D., Robinson, A., & Hughes, G. (2013). The Effects of a STEM Intervention on Elementary Students’ Science Knowledge and Skills. School Science and Mathematics, 113(5), 215-226
Denney, A. S., Tewksbury, R. (2013). How to Write a Literature Review. Journal of Criminal Justice Education, 24(2), 218-234
Duval, S., & Tweedie, R. (2000). Trim and Fill: A Simple Funnel-Plot-Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis. Biometrics, 56(2), 455-463
Fan, S. C., & Yu, K. C. (2015). How an integrative STEM curriculum can benefit students in engineering design practices. International Journal of Technology & Design Education, 27(1), 1-23
Fatade, A. O., Mogari, D., & Arigbabu, A. A. (2013). Effect of Problem-Based Learning on Senior Secondary School Students’ Achievements in Further Mathematics. Acta Didactica Napocensia, 6(3), 27-44
Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active Learning Increases Student Performance in Science, Engineering, and Mathematics. National Acad Science, 111(23), 8410-8415
Han, S., Rosli, R., Capraro, M. M., & Capraro, R. M. (2016). The Effect of Science, Technology, Engineering and Mathematics (STEM) Project Based Learning (PBL) on Students’ Achievement in Four Mathematics Topics. Journal of Turkish Science Education, 13, 3-29
Harris, C. J., Penuel, W. R., D’Angelo, C. M., DeBarger, A. H., Gallagher, L.P., Kennedy, C. A., Cheng, B. H., & Krajcik, J. S. (2015). Impact of Project-Based Curriculum Materials on Student Learning in Science: Results of a Randomized Controlled Trial. Journal of Research in Science Teaching, 52(10), 1362-1385
Holveck, S. E. (2012). Teaching for Conceptual Change in a Density Unit Taught to 7th Graders: Comparing Two Teaching Methodologies-Scientific Inquiry and a Traditional Approach. Oregon: University of Oregon.
Honey, M., Pearson, G., & Schweingruber, H. (2014). STEM Integration in K-12 Education Status, Prospects, and an Agenda for Research. Washington D. C.: National Academies Press.
James, J. S. (2014). Science, Technology, Engineering, and Mathematics (STEM) Curriculum and Seventh Grade Mathematics and Science Achievement. Michigan: ProQuest LLC.
Kalaian, S. A., & Kasim, R. M. (2014). A Meta-Analytic Review of Studies of the Effectiveness of Small-Group Learning Methods on Statistics Achievement. Journal of Statistics Education, 22(1), 1-20
Kala. (2018). Effectiveness of Small-Group Learning Pedagogies in Engineering and Technology Education: A Meta-Analysis. Journal of Technology Education, 29(2), 20-35
Kassir, H. A. N. (2013). The effectiveness of the Science-Inquiry Teaching Approach on the Students’ Achievement and Engagement in the UAE’ Public Schools. Dubai: The British University in Dubai.
Kelley, T. R., & Knowles, J. G. (2016). A conceptual framework for integrated STEM education. International Journal of STEM Education, 3, 1-11
Kim, K. H., Tassel-Baska, J. V., Bracken, B. A., Feng, A., Stambaugh, T., & Bland, L. (2012). Project Clarion: Three Years of Science Instruction in Title I Schools among K-Third Grade Students. Research in Science Education, 42(5), 813-829
Kim, N. A., Belland, B. R., & Walker, A. E. (2018). Effectiveness of Computer-Based Scaffolding in the Context of Problem-Based Learning for Stem Education: Bayesian Meta-analysis. Educational Psychology Review, 30(2), 397-429
Kine, M. S. (2017). Visual-Spatial Ability in STEM Education: Transforming Research into Practice. Switzerland: Springer International Publishing Switzerland, 2017.
Kizkapan, O., & Bektas, O. (2017). The Effect of Project Based Learning on Seventh Grade Students’ Academic Achievement. International Journal of Instruction, 10(1), 37-54
Korur, F., Efe, G., Erdogan, F., & Tunc, B. (2015). Effects of Toy Crane Design-Based Learning on Simple Machines. International Journal of Science and Mathematics Education, 15, 251-271
Lazowski, R., & Hulleman, C. (2015). Motivation Interventions in Education: A Meta-Analytic Review. Review of Educational Research, 86(2), 1-39
Li, Y., Huang, Z., Jiang, M., & Chang, T. W. (2016). The Effect on Pupils’ Science Performance and Problem-Solving Ability through Lego: An Engineering Design-based Modeling Approach. Educational Technology & Society, 19(3), 143-156
Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-Analysis. Thousand Oaks: Sage Publications, Inc.
Maxwell, D. O., Lambeth, D. T., & Cox, J. T. (2015). Effects of Using Inquiry-Based Learning on Science Achievement for Fifth-Grade Students. Asia-Pacific Forum on Science Learning and Teaching, 16(1), 1-31
McDonald, C. V. (2016). STEM Education: A Review of the Contribution of the Disciplines of Science, Technology, Engineering and Mathematics. Science Education International, 27(4), 530-569
Merrill, C. (2001). Integrated Technology, Mathematics, and Science Education: A Quasi-Experiment. Journal of Industrial Teacher Education, 38, 45-61
Michko, G. M. (2008). Meta-Analysis of Effectiveness of Technology Use in Undergraduate Engineering Education. https://www.researchgate.net/publication/224361059_Meta-analysis_of_effectiveness_of_technology_use_in_undergraduate_engineering_education.
Ojaleye, O., & Awofala, A. O. A. (2018). Blended Learning and Problem-Based Learning Instructional Strategies as Determinants of Senior Secondary School Students’ Achievement in Algebra. International Journal of Research in Education and Science, 4(2), 486-501
Olivarez, N. (2013). The Impact of a STEM Program on Academic Achievement of Eighth Grade Students in a South Texas Middle School. Corpus Christi, Texas: Texas A & M University - Corpus Christi.
Rehmat, A. P. (2015). Engineering the Path to Higher-Order Thinking in Elementary Education: A Problem-Based Learning Approach for STEM Integration. Las Vegas: University of Nevada.
Robinson, A., Dailey, D., Hughes, G., & Cotabish, A. (2014). The Effects of a Science- Focused STEM Intervention on Gifted Elementary Students’ Science Knowledge and Skills. Journal of Advanced Academics, 25(3), 189-213
Rosenthal, R. (1979). The “File Drawer Problem” and Tolerance for Null Results. Psychol Bull, 86, 638-641
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005). Publication Bias in Meta-Analysis Prevention, Assessment and Adjustments.Chichester: John Wiley & Sons Ltd.
Rozek, C. S., Ramirez, G., Fine, R. D., & Beilock, S. L. (2019). Reducing Socioeconomic Disparities in the STEM Pipeline Through Student Emotion Regulation. Proceeding of the National Academy of Sciences of the United States of America, 116(5), 1553-1558
Sanders, M. (2009). STEM, STEM Education, STEMmania. Technology Teacher, 68(4), 20-26
Sarac, H. (2018). The Effect of Science, Technology, Engineering and Mathematics-Stem Education Practices on Students’ Learning Outcomes: A meta-analysis. The Turkey Online Journal of Educational Technology, 17(2), 125-142
Slavin, R. E. (2008). Evidence-Based Reform in Education: What Will It Take?. European Educational Research Journal, 7(1), 124-128
Slavin, R. E. (2017). Evidence-Based Reform in Education. Journal of Education for Students Placed at Risk, 22(3), 178-184
Sokolowski, A. (2013). The Effects of Exploratory Learning Environments on Students’ Mathematics Achievement. Texas: Texas A&M University.
Sokolowski, A., Li, P., & Willson, V. (2015). The Effects of Using Exploratory Computerized Environments in Grades 1 to 8 Mathematics: A Meta-Analysis of Research. International Journal of STEM Education, 2(8), 1-17
Steenbergen-Hu, S., & Cooper, H. (2013). A Meta-Analysis of the Effectiveness of Intelligent Tutoring Systems on K-12 Students’ Mathematical Learning. Journal of Educational Psychology, 105(4), 970-987
Taylor, K. (2016). Collaborative Robotics, More Than Just Working in Groups: Effects of Student Collaboration on Learning Motivation, Collaborative Problem Solving, and Science Process Skills in Robotic Activities. Boise: Boise State University.
Valentine, J. C., & Cooper, H. (2003). What Works Clearinghouse Study Design and Implementation Assessment Device (Version 0.6). Washington, DC: U.S. Department of Education.
Yasin, R. M., & Yunus, N. S. (2014). A Meta-Analysis Study on the Effectiveness of Creativity Approaches in Technology and Engineering Education. Asian Social Science, 10(3), 242-252
Yildirim, B. (2016). An Analyses and Meta-Synthesis of Research on STEM Education. Journal of Education and Practice, 7(34), 23-33
Young, J., Ortiz, N., & Young, J. (2017). STEMulating Interest: A Meta-Analysis of the Effects of Out-of-School Time on Student STEM Interest. International Journal of Education in Mathematics, Science and Technology, 5(1), 62-74
Zeng, Z., Yao, J. Gu, H., & Przybylski, R. (2018). A Meta-Analysis on the Effects of STEM Education on Students’ Abilities. Science Insights Education Frontiers, 1(1), 3-16

[1]	Jianzhen Zhang, Xiaoyu Liang. Analysis of the Influence of Family Background on Students’ Subject Choices under the New College Entrance Examination System: Empirical Investigation Based on Survey Data in Zhejiang Province [J]. Journal of East China Normal University(Educational Sciences), 2023, 41(6): 47-58.
[2]	Hao Lei, Xue Li. Does the Digital Tools-supported Teaching Have an Impact on Students’ Learning Results:Meta-Analysis Evidence from 137 Experiments and Quasi Experiments [J]. Journal of East China Normal University(Educational Sciences), 2022, 40(11): 92-109.
[3]	Wanpeng Lei, Hongmei Ma, Huaming Huang. Evaluate Teaching Performance by Residuals of Student Achievement [J]. Journal of East China Normal University(Educational Sciences), 2021, 39(7): 84-91.
[4]	Slavin Robert, C.K. Cheung Alan, Tengteng Zhuang. How Evidence-Based Reform Could Advance Education [J]. Journal of East China Normal University(Educational Sciences), 2021, 39(3): 14-22.
[5]	Lihui Sun, Linlin Hu. Can Programming Really Promote the Individual Development of Children? A Meta-analysis of 28 Experimental and Quasi-experimental Studies [J]. Journal of East China Normal University(Educational Sciences), 2021, 39(11): 45-58.
[6]	Liu Jian, Zhao Liman, Du Xiaofeng, Xu Guanxing. A Study of the Warning Mechanism and the Ideal Matching Pattern between Sleep Time and Academic Achievement of High-school Students [J]. Journal of East China Normal University(Educational Sciences), 2020, 38(3): 71-79.
[7]	Zhao Chen, Chen Si, Cao Yan, Catherine Snow, Lu Mai. Targeted Poverty Alleviation through Education: Long-term Effect of China’s “One Village One Preschool” Project on Rural Children’s Academic Achievements [J]. Journal of East China Normal University(Educational Sciences), 2020, 38(2): 114-125.
[8]	David ANDERSON, JI Jiao. International Dialogue on the Role of Museum Education in STEM and STEAM Education [J]. Journal of East China Normal University(Educationa, 2017, 35(4): 122-129+139.
[9]	LIN Jing. The New Quality Assessment in America's Basic Education: Lessons from NAEP's TEL [J]. Journal of East China Normal University(Educationa, 2017, 35(1): 78-86+122.
[10]	SUN Yanjie,SUN Fangfang. A Meta-Analysis of the Moral Embodied Cognition [J]. Journal of East China Normal University(Educationa, 2014, 32(2): 88-96.