Evaluation criteria are mostly of a one-fits-all type. I believe they should be different, based on the type of contribution, e.g., theory or practice; and they should be made public.
Much of our understanding of what kind of papers are suitable for which conference is primarily based on historical knowledge, participation in TPCs, etc., and does not differentiate various types of contributions, e.g. theory vs. practice. As a consequence, inadequate criteria may be used for a particular paper (“there is no practical application”, “this contains no proofs”), leading to incorrect evaluations. Reviewers may assign varying weights to evaluation criteria implicitely, leading to inconsistent evaluations.
At ECRTS, we (notably Sophie Quinton) tried to be more explicit with criteria for contributions of different types, see for example here.
We welcome theoretical and practical contributions to the state of the art in the design, implementation, verification and validation of real-time embedded and cyber-physical systems. This includes case studies, implementations, tools, benchmarks, application scenarios, technology transfer success stories (or failures) and open problems.
We particularly encourage papers on industrial case studies, application of real-time technology to real systems, and system-level implementations. We believe that our community needs to validate the proposed assumptions and methodologies with respect to realistic applications. We welcome practical contributions even without novel theoretical insights, provided they are of interest to the research community and/or to industry. In such cases, authors should make an effort to provide proper motivation and justification of the relevance of their work and report on lessons learned.
The models, assumptions and application scenarios used in the paper must be properly motivated.
Authors (of both practical and theoretical works) must demonstrate applicability of their approach to real systems (examples of realistic task models can be found here). Deviations from such models or strong simplifications require justification. In particular, papers relying on simple models must have more fundamental theoretical insights than papers that include more details by construction, such as system-level implementations.
Whenever relevant, we strongly encourage authors to present experimental results. These can be obtained on real implementations, from simulations or via the use of data from case studies, benchmarks or models of real systems. The use of synthetic workloads or models is permitted, but authors are strongly encouraged to properly motivate and justify the choices made in designing their synthetic evaluation.
Such criteria help authors and reviewers. During discussions in the TPC, they can be used to align the discussion, ensure there are not implicit or explicit changes to the criteria as we discuss, and specific questions can be asked, e.g., by the TPC chair to steer the discussion.
In addition, we should say, in CfP, what we expect of papers w.r.t. evaluation etc.
Theory vs. practice were initial categories, I believe we should have more: for example, Sanjoy Baruah suggests conferences should be open to very novel ideas, which do not fit our standard criteria or may not be fully up to standard otherwise. That is an important point, there could be a category with properly defined criteria for it. (A well defined criterium can also avoid sloppy, similar sounding arguments , e.g. “can create discussion” to accept inferior papers etc)
The Technical Committee or other bodies responsible for the strategic development of a conference should define and maintain criteria for different types of contributions together with the TPC chairs.
Such criteria can also help define and publish the profile of conferences.
I believe it is important that criteria used in evaluations are made public (and adhered to during the evaluation processes.): the Call-for-Papers should be the agreed upon text communicated to authors and criteria used during evaluation by the TPC.