Zuzana Havránková: Smaller classes do not help pupils achieve better results in schools
What is it like to be a woman in economics? What interesting research projects has she worked on? And what is meta-analysis?
Read the interview with leading Czech economist Zuzana Havránková from the Institute of Economic Studies FSV UK. The interview was recorded as part of the De Facto podcast. This episode is hosted by Alice Němcová Tejkalová.
Alice Němcová Tejkalová (ANT). Did you know right from the beginning what you wanted to do and what you wanted to pursue during your PhD?
Zuzana Havránková (ZH): I knew it from the start. Although it crystallized during my Bachelor and Master studies. But I think the biggest crystallization took place in Helsinki, because there we got to see Professor Kunnas in a very common subject that is taught there for all foreign students – empirical analysis, econometric treatment of various things and so on. He introduced us to something called meta-analysis. From that came a seminar paper and my first publication, which was peer-reviewed. And from that I crystallized into what I do today, which is meta-analysis.
ANT: And could you explain to people who are not into economics what exactly you do?
ZH: Meta-analysis is an interdisciplinary concept. It’s basically a quantitative summary of the literature. Imagine we have a scientific question that we’re trying to answer. We have a lot of studies on it, and they often contradict each other – their results are different, bigger, smaller, positive, negative. When the media, politicians or legislators look at how to pull out the right answer to the question they want to address, it’s no wonder that it’s difficult and problematic to find arguments when it’s also very easy to find counter-arguments. And this is the problem with our science. Every year we see an exponential increase in what is produced, but at the same time the emphasis on so-called evidence-based policy is not that strong. And maybe it’s because we’re experiencing what we call a replication crisis. Indeed, it is often difficult to arrive at the same results on a topic that has been discussed before in the literature. And then the role of meta-analysis is that it can distill a result, a single conclusion from literature that is very diverse.
By the way, meta-analysis originally originated in experimental medicine research, in clinical trials. There the studies are very small because notoriously they are very expensive. So they needed to somehow put small clinical studies together and increase the statistical power of the results. That’s why meta-analysis naturally evolved in medicine.
Of course, meta-analysis brings various problems because we are looking at a large sample of studies that are very heterogeneous. First of all, their context, so what kind of data they use, whether they are men, women, different ethnicities, what kind of comorbidities they have and so on. That’s one of the challenges of meta-analysis – how you can control the context. The second challenge is publication selectivity. This arises when some results are more likely to be published. This is because we often have an internal a priori preference about what should come out in a study. Imagine I’m studying the effect of second-hand smoking on health. There you can find very large negative consequences, smaller negative consequences, and you can also find that it has zero effect on health. And you can even find positive effects. That is, passive smoking is beneficial to health.
ANT: Are there really studies like that?
ZH: Yes, there really are such studies. We have known internally for many years that smoking itself is very detrimental and harmful. If I were to do such a study now, I would go into it with a certain idea or preconceived notion of what the result should look like. I assume that the consequence will be negative or zero, but it will certainly not be positive. At the level of one study, these preconceptions are not such a problem, of course. As authors, we always try to pick a representative result so as not to overwhelm the reader with something that is not digestible. In short, we are selective about what we publish.
But the problem arises when this is done systematically in the literature. That is, most researchers do not report positive results. But instead they report the big, negative results, which in this case may be bad for exactly the same reason. That is, distorted by role of chance, bias, bad model.
ANT: I can imagine this is happening even with much less controversial topics than smoking. After all, the pressure for articles to be published in journals that have a particular focus is something we have also recorded at our Institute of Communication Studies and Journalism. We have discussed this with colleagues repeatedly, that we need to choose a journal that is, shall we say, more open to the fact that things can come out in different ways. I wonder how the researchers then feel about the results of that particular meta-analysis? When it turns out that there is a systematic bias?
ZH: This is not a problem for individual researchers. It’s not that we’re saying that someone is cheating or deliberately withholding. That’s not what this is about at all. Our job as meta-analysts is to figure out if there’s a problem and then fix it based on certain statistical assumptions.
ANT: It must be an unpopular position for you though, right?
ZH: That’s right. Meta-analysts are not exactly popular. When you look at someone’s life’s work and you criticize them, not consciously, but rather point out that there’s something wrong with the whole literature, that’s kind of an indirect criticism.
ANT: You are involved in a number of different topics. It’s definitely worth mentioning that you often collaborate with your husband Tomáš Havránek on research. Not only with him, but also with other colleagues you have quite stable research teams. As I have noticed in your publications, you have looked at, for example, the impact of class size on pupil achievement. What did you come up with?
ZH: Our intuition tells us that kids do better in smaller classrooms. The child gets more attention from the teacher, who in turn has more time for the child. The teacher has more peace to do things there. But we have found that the effect is zero and smaller classes do not help the results. Even during the covid, we saw that some covid funds were used to reduce classes, for example, although for a completely different reason than the effect of the classes themselves. But in general, we see that the arguments that various industry associations and legislators have are just tied to the fact that the literature supports this, and therefore it’s good to pour money into it. So intuitively it makes sense for parents and teachers because they have more peace and so on. But anyway, meta-analysis is only as good as the underlying literature. So we can say that if the underlying literature is good, our meta-analysis says that there is no such effect.
