Starting again from three. Lessons from three counterfactual analysis experiences in education

Journal title RIV Rassegna Italiana di Valutazione
Author/s Giovanni Abbiati, Gianluca Argentin, Andrea Capurto, Aline Pennisi, Barbara Romano, Daniele Vidoni
Publishing Year 2014 Issue 2013/55
Language Italian Pages 26 P. 21-46 File size 702 KB
DOI 10.3280/RIV2013-055003
DOI is like a bar code for intellectual property: to have more infomation click here

Below, you can see the article first page

If you want to buy this article in PDF format, you can do it, following the instructions to buy download credits

Article preview

FrancoAngeli is member of Publishers International Linking Association, Inc (PILA), a not-for-profit association which run the CrossRef service enabling links to and from online scholarly content.

The article reviews three evaluations of interventions in education. All the programmes have been implemented recently in the Italian schools and were aimed to improve student maths achievement. The three programmes differ regarding their logic framework and contents, but have been evaluated adopting counterfactual methods. The authors focus on this point, showing that it is possible to implement counterfactual evaluations in the Italian education system. More precisely, it is argued that randomized control trials are viable. Nonetheless, it is also shown that rigorous estimates could also be gained using other methods, taking advantage of pre-existing administrative data. The three evaluation experiences are used here to stress pros and cons of counterfactual evaluation and to provide advice for its improvement in Italy.

Keywords: Math achievement, counterfactual evaluation, randomized control trial, chess, teacher professional development, educational policies

  1. Abbiati G. (2013), La formazione degli insegnanti in Italia. Valutazione di due politiche di formazione professionale per insegnanti della scuola media, Tesi di dottorato non pubblicata.
  2. Argentin G., Martini A., Pennisi A. e Vidoni D. (2010), “Valutare gli effetti del percorso di formazione M@tabel tramite un esperimento controllato”, Scienze dell’amministrazione scolastica, 3: 22-34.
  3. Argentin G., Pennisi A., Vidoni D., Abbiati, G., Caputo A. (2013), Preliminary evidence from the M@t.abel teacher professional development program in Italy (Working paper No. 19). Roma: INVALSI. Testo disponibile al sito: http://www.invalsi.it./download/wp/wp19.pdf [8
  4. Aprile 2013].
  5. Barbieri G., Cipollone P., Sestito P. (2008), Labour market for teachers: Demographic characteristics and allocative mechanisms (Temi di discussione n. 672), Banca d’Italia, Roma.
  6. Barbieri P., Raimondi E., Scherer S. (2013), “La terza forma a priori della rappresentazione. L’inferenza causale nelle scienze sociali e la congiunzione spazio-temporale come fine ultimo della scienza sociale”, Quaderni del Dipartimento di Sociologia e Ricerca Sociale dell’Univserità degli Studi di Trento, n. 62.
  7. Bellei C. (2009), “Does lengthening the school day increase students’ academic achievement? Results from a natural experiment in Chile”, Economics of Education Review, 28: 629–640, DOI: 10.1016/j.econedurev.2009.01.008
  8. Berkman R.M. (2004), “The Chess and Mathematics Connection: More than Just a Game”, Mathematics Teaching in the Middle School, 9(5): 246-250.
  9. Boruch R.F., Porter A.C., Merlino G.J. (2013), “Golfing in a Hurricane: Education System Instability, Randomized Controlled Trials, and Children’s Achievement”, Working Paper CRESP – Center for Research and Evaluation in Social Policy, University of Pennsylvania.
  10. Buki J., Ho F. (2008), The Effect of Math and Chess Integrated Instruction on Math Scores | PRLog, The Chess Academy. Retrieved fromTesto disponibile al sito: http://www.prlog.org/10086633-the-effect-of-math-and-chess-integrated-instructionon-math-scores.html [30 Marzo 2013], Chicago.
  11. Campitelli G., Gobet F. (2008), “The role of practice in chess: A longitudinal study”, Learning and Individual Differences, 18: 446-458, DOI: 10.1016/j.lindif.2007.11.006
  12. Caputo A., Pennisi A. (2011), “Insegnare la statistica a scuola: un confronto tra nuclei tematici nei diari di bordo del progetto M@t.abel a.s. 2009/2010”, Induzioni. Demografia, probabilità e statistica a scuola, 42: 33-50. Cohen J., Easterly W., a cura di (2009), What Works in Development? Thinking Big and Thinking Small, Brookings Institution PressDarling-Hammond L. (1999), Teacher
  13. Quality and Student Achievement: A Review of State Policy Evidence, University of Washington, Center for the Study of Teaching and Policy, Seattle, WA.
  14. Ferguson R. (1983), Developing Critical and Creative Thinking Through Chess, Study I. The ESEA Title IV-C Project. Testo disponibile al sito: http://www.main.nc.us/bcsc/Chess_Research_Study_I.htm [5 Febbraio 2013].
  15. Goldthorpe J.H. (2001), “Causation, Statistics, and Sociology”, European Sociological Review, 17(1): 1-20.
  16. Hong S., Bart W.M. (2007), “Cognitive Effects of Chess Instruction on Students at Risk for Academic Failure”, International Journal of Special Education, 22(3): 89-96.
  17. Kennedy M. (1998), Form and substance of in-service teacher education, National Institute for Science Education, University of Wisconsin-Madison, National Institute for Science Education, Madison, WI.
  18. Lavy V. (2012), Expanding School Resources and Increasing Time on Task: Effects of a Policy Experiment in Israel on Student Academic Achievement and Behavior (No. w18369), National Bureau of Economic Research, Cambridge, Mass. Liptrap J.M. (1998), “Chess and Standard Test Scores”, Chess Life (March): 41-43.
  19. Marcotte D., Hemelt S. (2008), “Unscheduled school closings and student performance”, Education Finance and Policy, 3(3): 316-338, DOI: 10.1162/edfp.2008.3.3.316
  20. Marguiles S. (1991), The Effect of Chess on Reading Scores: District Nine Chess Program Second Year Report, The American Chess Foundation dissertation, New York. Testo disponibile al sito: Retrieved from http://www.givewell.org/files/Analysis/margulies.pdf [21 Febbraio 2013].
  21. Martini A., Sisti M. (2009), Valutare il successo delle politiche pubbliche, il Mulino, Bologna.
  22. Martini A., Trivellato U. (2011), “Sono soldi ben spesi?” Perché e come valutare l’efficacia delle politiche pubbliche, Marsilio Editori, Venezia.
  23. Martini A. (2013), Treatment, offer or encouragement: the many faces of randomized controlled trials, in corso di pubblicazione, Polish Agency for Enterprise Development.
  24. Mori S., Abbiati G. e Vidoni D. (2012), Collaborazione nella scuola per l’efficacia della scuola: evidenze dal progetto PQM, Induzioni, 44: 13-30.
  25. Pennisi A., a cura di (2013), Valutazione PON M@t.abelM@t.abel+. Rapporto sui risultati preliminari sugli effetti del programma PON M@t.abel+ 2009/2010., INVALSI, Roma.
  26. Testo disponibile al sito: http://www.invalsi.it./invalsi/ri/matabel/documenti/rapporti/Matabel_0313_Vol1.pdf [10 Aprile 2013].
  27. Robin K.B., Frede E.C., Barnett W.S. (2006), Is More Better? The Effects of Full-Day vs Half-Day Preschool on Early School Achievement, National Institute for Early Education Research, , New Jersey.
  28. Sweller J., Clark R.E. e Kirschner P.A. (2006), “Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry based teaching”, Educational Psychologist, 41: 75-86, DOI: 10.1207/s15326985ep4102_1
  29. Weiss C. (1997), “La valutazione basata sulla teoria: passato, presente e futuro” in Stame, N., cur., Classici della Valutazione, FrancoAngeli, Milano.
  30. What Works Clearinghouse (2009), WWC Quick review protocol. Version 2.0. Testo disponibile al sito: http://ies.ed.gov/ncee/wwc/DownloadSRG.aspx [3 Aprile 2013].

Giovanni Abbiati, Gianluca Argentin, Andrea Capurto, Aline Pennisi, Barbara Romano, Daniele Vidoni, Ricomincio da tre. Lezioni da tre esperienze italiane di analisi controfattuale in ambito educativo in "RIV Rassegna Italiana di Valutazione" 55/2013, pp 21-46, DOI: 10.3280/RIV2013-055003