Some hard facts about science, money and evaluations

2017-03-26

The most fun experiments are those that generate a lot of noise and heat. So why not ask a scientist about her latest grant application? Or about the reports she has been asked to write describing the results of her research? These simple questions will likely result in spectacular fulminations at the evil minions of government agencies, the incompetent idiots that call themselves university leaders, or the ill-will of society and the universe in general.

Scientists want to get on with their work. They hate the bureaucratic drudgery of writing grant applications and reports. Who can blame them? Every day spent on grant writing is a day less for real work. The worst part is that most grant applications fail. For example, the Swedish governmental science grants agency VR received a total of 222 applications in the field of natural sciences and technology in the round of 2016. The results were announced 23 Feb 2017. A total of 18 (eighteen) applications were successful. That's 8%. The rest, 92 %, were written in vain. From discussions in the media, it is clear that this is a general phenomenon in most of the world. Who can say that scientists should not hate this system?

Although similar hard numbers are difficult to find for the (non)effects of report writing, most scientists are convinced that an overwhelming part of the effort that goes into compiling reports is wasted. Who actually reads the report? Do the conclusions of the report actually influence any decisions? There has recently been a round of reporting at the outfit where I work (Science for Life Laboratory in Stockholm). The purpose was to look at the scientific impact of the service facilities of SciLifeLab. Anecdotal evidence suggests that the subsequent allocation of money did not bear any comprehensible relation to the data of the report. That is in any case what I hear from several persons involved. If that is what those affected believe, then the reporting exercise seems to have generated one tangible result: demoralization.

There is a problem, clearly. But it is worse than that. There is also a fundamental confusion in the discussion among scientists about this very problem.

Scientists typically argue as if it is the evaluation of grants and scientific work that is the root evil. If we got rid of grant applications and reporting, i.e. stopped doing evaluations, and just gave the scientists the money they need to get on with their work, everything would be fine. An example of this type of reasoning can be found in the academic paper Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition by Marc A. Edwards and Siddhartha Roy (Environmental Engineering Science. January 2017, 34(1): 51-61. doi:10.1089/ees.2016.0223). The paper makes many good points, for instance that any metric for evaluation is bound to become corrupted, reiterating the well-known Goodhart's Law which states: "When a measure becomes a target, it ceases to be a good measure." The paper also makes the valid point that the problem of irreproducible scientific publications may be caused, at least in part, by the incentives created by using bibliometric analysis as selection criteria. The conclusions are mostly about trying to find better incentive schemes, which is pretty lame given the reference to Goodhart's law. There is one single statement about funding levels being beyond the control of scientists (well, duh!). The paper does not say it explicitly, but as in so many other similar articles, there is a strong undercurrent that if just the politicians understood the problem more resources would be allocated to science.

At best, this is wishful thinking. It is not so easy to understand how intelligent persons, which most scientists are, can truly think that this is a serious suggestion. It is as if scientists are unable to dispassionately analyse their own situation and its inherent constraints in the same detached way as they view the subject of their own research. When studying, let's say, bacteria, it is a given that bacteria are constrained by the fundamental laws of thermodynamics, conservation of mass, evolution, and so on. So if a bacterium divides and grows, it requires energy, substrates and a certain environment. Anyone putting forth theories about the behavior of bacteria that violate these constraints will be rightfully laughed out of the seminar room.

So let me spell out some very simple, almost trivial, facts about science and resources.

First point: The amount of money allocated to science is finite. At any given moment, or in any budget put before parliament (national, regional, community), there is only so much money to be divided between all sectors, from the military, police, social services, medical care, culture, and so on. Science and higher education is one of these sectors. There will never be enough money to make everyone happy. Ever.

An interesting argument has been advanced that since money allocated to science can be viewed as an investment (assuming that economically beneficial results are obtained eventually), it must be viewed in a special way, such that more money should be given to science. This is rubbish.

At time t0, there is a certain amount of resources R to be allocated. We (the state) give a certain fraction f0 to science. Assume that we can then expect to obtain an amount of resources S at some later time t1, where S is hopefully larger than R. If the same fraction f0 is applied at that time t1, then science will of course obtain more in terms of absolute resources. But, please note, this is at time t1, not t0. I apologize for stating the obvious, but those increased resources S are not available at time t0.

The idea that science funding is an investment is not, in itself, a conclusive argument that more money should be spent on science. There are always other investment alternatives. Maybe building a bridge would be better. Or decreasing the number of pupils per teacher in primary school. Or something else. Science is in no way special in this regard. Uttering the word "investment" does not produce extra resources magically at time t0.

As an aside: the investment argument is often used in combination with the idea that we (the state) should borrow the money, since the results of an investment will pay the interest for the loan and more. To which one may give a very brief response: Greece.

Second point: The fraction of money allocated to science cannot reasonably be expected to increase significantly. This just will not happen unless scientists can convince others (voters, taxpayers, politicians) that this is a sensible policy. There are no hidden treasures. Without good arguments, no larger fraction will be allocated. Good arguments require some kind of reasoning about the impact of science, either in terms of its general place in our civilisation where truth and knowledge is valued highly, or that science produces more tangible economic, medical or other good consequences. For this, evaluations of some kind of are needed.

In this connection, I would like to point out that scientists need to consider if their arguments are effective. The above mentioned academic paper describes a gloomy state of affairs in science, and blames it on hypercompetition and lack of funding. What the taxpayer hears on being told this is: "The research system is broken. We produce lots of questionable papers. Give us more money." Can we expect the taxpayer to say: "But of course, more money is the obvious solution." No, I don't think so.

Third point: More people want to become scientists than it is possible to accommodate. There is competition for the available research positions. This is a good thing! Imagine if being a scientist was not considered attractive. That would really signal a crisis in our type of civilization. So, in order to decide who is going to become a scientist and who is not, some kind of selection is required.

Fourth point: The idea that a scientist should be given ample resources as part of her position is very attractive. In fact, this kind of arrangement does exist. There are research institutes that provide their scientists with fairly substantial amounts of resources. There are also special positions at some universities for young brilliant scientists who get grants as part of the position. These are almost always limited in time, the idea being that the scientist should be able to attract other grant money after this start-up period.

But the essential point with these arrangements is that they are invariably elitist. The selection of which scientists get such positions is very strictly meritocratic, i.e. the result of a very tough evaluation. It is unavoidable. There simply are not enough resources for every aspiring or current scientist to get this kind of deal.

Fifth point: Modifying or totally redesigning the allocation mechanism cannot change the fundamental issue of resource scarcity. If there is not enough money to award in grants, then there is not enough money, full stop. There is no allocation scheme that can change that.

Recently, Shahar Avin has made a very intriguing proposal, namely that we should allocate research funds by lottery. From the set of reasonable research proposals, simply select randomly which to fund. Then the evaluation will just amount to weeding out the worst proposals, keep the rest without bothering to pick the best, which is hopeless anyway, and let Lady Fortuna decide. I think this definitely is worth considering.

But please note that Avin's idea solves fewer problems than it seems. Avin begins his article by giving an example of a research programme which yielded the drug propranolol in the 1940s, but which would have been rejected using today's system because it looked far-fetched. Perhaps. But with Avin's system, it would probably have had the bad luck to lose the lottery, since most proposals would lose. So what is gained from the system perspective? Avin also points out that with the low success rate of grant proposals (8% in my example above), there is a lot of waste in writing grant proposals that eventually fail. Again, this is not changed by moving to a lottery system. The same 8% (or whatever) of grant proposals will succeed, just a different 8%. The waste in grant proposal writing will be exactly the same.

Things are, at the end of the day, very simple: Since resources are restricted, there has to be a selection at some point. Either the number of scientists is kept low, in which case each scientist can get more money, maybe without having to write grant applications. Or more people are allowed to become scientists, in which case they will have to share the same amount of resources by some allocation mechanism. The result is, unavoidably, that the average scientist will get less than in the former scenario, since more scientists compete for the same resources.

We need to discuss the problems of science such as bad incentives, shoddy publications, me-too research, and so on. This is very important. But it is futile to discuss these issues as if the solution was to just give researchers more money. That is not going to happen.

So, when discussing mechanisms for resource allocation, please take into account the fundamental fact that resources are finite, and the political fact that research as a sector in society is very unlikely to get a larger fraction of the available resources than today.

If your idea does not allow for these facts, then ask yourself: What is the point? A proposed solution that tries to avoid reality is not a serious proposal. It is a waste of time.