I find numerical evaluation systems bad for the community.
Evaluation criteria, notably h-index, are forced upon us by actors outside our control, e.g., funding agencies, promotions boards etc. Consequently, meeting such criteria becomes important. I do not understand why.
See this post for comments on acceptance ratio.
At ECRTS, we tried to move towards a conference complementing existing ones.
The economist Charles Goodhart stated in 1975 (“Goodhart’s law”)
Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes
(Goodhart, Charles (1975). “Problems of Monetary Management: The UK Experience”. Papers in Monetary Economics 1975)
Rephrasing: “Any qualitative measure, when used for evaluation, will loose being a qualitative measure.” Or here, using h-index for evaluation makes h-index pointless.
Instead of doing productive research, it favors just meeting the (now) useless criteria, e.g., h-index. Higher number of publications and citations beats productive research.
As a consequence, authors have to turn their research results into as many papers as possible, increasing their efforts, but also those of reviewers, TPC members etc, affecting junior researchers and TPC members disproportionately. In fact, even readers have to put in more effort to consolidate results spread over a number of papers.
Individual examples of h-index failing to represent research quality include Peter Higgs, Physics Nobel prize winner of 2013, h-index 10 (“I wouldn’t be productive enough for today’s academic system“) or Albert Einstein with h-index of 56.
Numerical evaluation systems provide ample opportunities to cheat or rig the system: Cathy O’Neil‘s award winning, bestselling book “Weapons of Math Destruction” (2016) has an eye opening example of how in 2014 U.S. News ranking of global universities the mathematics department at Saudi Arabia’s King Abdulaziz University landed in seventh place, right behind Harvard after being around for only 2 years, came ahead of several giants of mathematics, including Cambridge and MIT.
Lior Pachter, a computational biologist at Berkeley, looked into it. He found that the Saudi university had contacted a host of mathematicians whose work was highly cited and had offered them $72,000 to serve as adjunct faculty. The deal, according to a recruiting letter Pachter posted on his blog, stipulated that the mathematicians had to work three weeks a year in Saudi Arabia. The university would fly them there in business class and put them up at a five-star hotel. Conceivably, their work in Saudi Arabia added value locally. But the university also required them to change their affiliation on the Thomson Reuters academic citation website, a key reference for the U.S. News rankings. That meant the Saudi university could claim the publications of their new adjunct faculty as its own. And since citations were one of the algorithm’s primary inputs, King Abdulaziz University soared in the rankings.”
The numerical evaluation systems also force attitudes and procedures on the research community; I call such conferences “H-index Machines” in contrast to “Research conferences” fostering actual research without numerical regards.
When going open access with ECRTS, we had to take such aspects into considerations. Some comments regarded “I need an IEEE/ACM/… publication for promotion, evaluation, … to score well on h-index, CORE …”, which are valid concerns. I believe there are enough h-machine conferences for those needing to play the game.
Due to historic reasons and numerical evaluation pressures, many conference tend to get similar, in effect creating mono cultures of publication outlets – such mono cultures, as in nature, are prone to many dangers, level out what might be vital differences.
What we need are conferences:
- different and complementing the standard ones
- enabling exchange and collaboration
- value important research not reflected by numbers
- inclusive to the entire community