The evaluation process of conferences can be tricky and complex, quickly overwhelming if things go wrong (and some will). My advice to TPC chairs is to prepare as much as possible in all steps, including
- Communicate guidelines, evaluation criteria, procedures etc clearly to the TPC. Many TPC members are overloaded, have done it many times, might not pay attention, so will do “how they always do it”. I found most TPC to be members constructive, but forgetful.
- Monitor reviews as they come in, so you are aware of situations of papers, and can guide further reviewing, possibly asking for additional opinions.
- Once reviewing is complete, before TPC meeting:
- go through the papers and reviews (yes that is a lot, but see it as investment for the TPC discussions), ensure that they are done according to guidelines and evaluation criteria.
- weed out the reviews: check for issues such as reviewer identifying wording, inappropriate language, reviews not based on facts, arrogant opinions etc
- once you know the reviews, you can have your own opinions (hard to lead a discussion when you have to rely only on others’ opinions) and anticipate special situations. Which paper has very diverging opinions, where is a single review skewing the evaluation etc, so you can prepare yourself and possibly add extra help.
- make a prediction for yourself on which papers will go on, which you think would be good for the conference, etc.. This helps you counter TPC discussions first going extra strict, then not having enough papers, then trying to compensate on the spot.
- At the TPC meeting:
- make a discussion order of papers which supports you, e.g., a paper you find sets an example with good reviews to set the tone
- with your mental list of accepted papers monitor progress and possible counter steer too restrictive trends
- do a practice run with some non TPC volunteers of the TPC meeting, ensure you are on top of the tools (evaluation software, spreadsheets, etc) – the meeting requires your full attention, you cannot do it well if the tools take time, get help for the logistics
- ensure conflicts-of-interests (COIs) are enforced at all times, including excluding COI members, references to other papers in the discussion etc.