Implementing new procedures or goals can be challenging with the inertia of TPCs, here some hints.
So you are TPC chair (congrats) with ideas on how to run the evaluation process differently (good). Now, how do you ensure your new ideas get implemented properly?
I found TPC members in general:
- constructive, willing to put in quite some effort and follow guidance of the program chair
- typically overloaded, needing to be efficient to get through it all. Many have been in many TPCs, internalizing procedures “as always” (almost asking at TPC meeting with “which conference is it this time?”)
So, the guidance you give to TPC members will compete with the “as always inertia”.
Add, that due to being busy, many will not read your email in the detail you expect, but might just look at the subject, search for the link to the reviews, not read or internalize your text, even forget later which conference had which criteria, or even that you came up with specific ones.
So effective communication of your goals and guidance becomes essential (and not easy, see above re: overload).
In addition I found some intuitive misunderstanding of terms can be an unnecessary hindrance. E.g.,”Good paper” will mean different things to different people. A discussion during the process among the TPC what we mean by terms can be counterproductive.
So my recommendation is to be as specific as possible in giving guidance what you want. Try to ask quantitative questions to avoid qualitative discussions. In the context of theory vs practice papers we found that giving specific criteria helps TPC members in reviewing, and during discussions is quite effective in reigning in loose meandering review opinions.
With criteria defined (and noted) TPC chairs can ask specifically “does the paper fulfill X” or “how much X does the paper contribute” which can be answered more objectively than “is it good”.
Specific criteria help to select papers early for discussion sequence which can serve as examples, setting the tone. In the beginning, TPC members tend to pay more attention, which is a good chance to emphasize the criteria in examples you get to pick.
A good place to communicate and collect criteria responses is in the review forms. Even when emails etc fail, when entering review scores, TPC members will notice.
Enforcing criteria later in the process has its challenges as well. I remember, e.g., from a conference with dedicated application track: the application criteria were held high in the beginning of discussions, but after a while, began to be eroded when fear of number of accepted papers arose.
My recommendation is to prepare well, monitor the process. Should compromises become necessary, you can prepare, and take your educated decisions, and avoid a TPC criteria discussion on the fly.
So I reformulate my opening statement to “I found TPC members in general constructive, willing to follow guidance of the program chair, provided the guidance is specific and clear, and they are aware of it and the differences to “as usual.”