Sign-up is open until Mon, 10/06 12:00pm (noon). Send an email to Alex (pucher [at] cs.ucsb.edu) with “[cs270] signup” from the gmail account (or linked google account) you prefer for correspondence.
The alternative, more egalitarian approach addresses these issues, but requires the cooperation of a far greater number of people. By considering the opinion of many scientists who are peers instead of one authority, biases can be identified and removed, power is decentralized, and a more collective understanding of each discipline is possible. The drawback is that there is no single dogma that can be used to resolve conflicting opinions. The authoritarian approach would be more efficient.
Be that as it may, we generally rely on peer review of scientific work to determine its value. There are essentially three contexts which provides a necessary evaluation: conference publication, journal publication, and competitive funding.
For a conference, peer review ensures the prospective attendees that the work which will be presented has been pre-screened by a qualified panel of peers. Typically, a conference program chair will ask knowledgeable members of the relevant community to serve on a program committee which is responsible for reviewing all submitted papers. If the conference is large and the likely submissions complex, the program committee may press into service individual members of the community (read: make the graduate students do it). Note that not all disciplines use peer review to ensure conference quality, but computer science, generally speaking, does. The goal of most conferences is to present ``fresh'' ideas and results to the relevant community so that they may rapidly be incorporated (with due citation) into evolving work by others.
Journals, on the other hand, typically publish work for posterity. Journals typically appoint an editor or series of editors responsible for different topic areas. Editors contact knowledgeable researchers in each field and ask for reviews of prospective papers. Journal papers tend to be longer, more detailed, and more complete than their conference counterparts. A scientist reading a journal article many years after its publication should be able to understand what is written entirely.
Peer review of competitive funding helps the funding agency (which is typically inundated with requests for money) determine the research that most nearly meets its goals. Since proposed work and not completed work is what is being reviewed, we won't discuss grant proposal reviews more than to mention their existence, but they are an important part of the way science is conducted in the United States today.
There are two goals that all publication reviews must attempt to serve. The first is to provide feedback to the authors about the work they have completed, and the second is to indicate value to the publishing body. To accomplish both of these goals, a review must be credible. That is, both the authors and the publishing body must believe that the review, itself, has value. Reviewers, however, are generally anonymous to their authors for fairly obvious human-nature reasons, so credibility based on reputation is not an option. One common way to help establish credibility in a review is to provide a short but complete summary of the work. By summarizing, the reviewer demonstrates that he or she has read the paper completely and is qualified enough to digest its important points meaningfully. As an author, I read the summary carefully to determine whether the reviewer read the paper carefully and what points he or she thought were important. Often, when a point I thought was important is omitted from the summary, I assume I need to make that point more clear in the paper. Similarly, the conference program committee gets both a quick digest of the work and a reading on the likely quality of the review from the summary.
The rest of the review should cover both the paper's strengths and weaknesses. No paper is without flaw and no piece of research is entirely complete. Similarly, almost no work is entirely without merit. A good review points out the contributions that the paper and the research makes to the community and describes why those contributions are important. It also points out the weaknesses in the work and gives scientific reasons for why these points are considered weak. To make this differentiation, I usually ask myself questions about the paper such as
Often, a paper will make a mathematical claim, prove that the claim is true, and then argue that the mathematics describes something observable (the possible exception to this rule occurs in quantum theory, but for now leave that be). A review for such a paper should verify that the proof is, indeed, correct and then should evaluate whether the proved claim is relevant.
In this class, the standard will be the degree to which the paper presents an idea that has not been previously presented in the class. That is, the novelty is determined by the degree to which the paper differs in its contribution from all papers that come before it in the class reading list. The notable exception will be the first paper we will read which has no antecedent under this limited definition of novelty.
For completeness, novelty is usually determined by comparing the work to all work on the subject that has been previously published. A published paper that is submitted verbatim to another publication should almost always be rejected unless the editor or program committee explicitly says otherwise. Often, however, changes and extensions to previous work are submitted. Are the extensions sufficiently interesting to warrant publication? Here, the standards for conferences and journals come into play. Small extensions to a conference paper may be sufficient for a conference. It is common to extend a conference paper and submit it to a journal, but the nature of the extensions, new results, etc. should be fairly substantial. An extension of one journal paper to make another requires far more new work.
It is also important to evaluate whether the work improves what is known over what has previously been published by other authors. If the work is good, but reiterates something that is already accepted as true, it is generally not worthy of publication. For example, a series of experiments showing that, in everyday life, force equals the product of mass and acceleration is probably not publishable in the physics community, no matter how careful and well-written the paper is.
On the other hand, it is equally important to recognize when a contribution, however obvious it may seem after it has been explained, is shedding new light on a subject. Reviewers often say "The work in the paper is not novel because the authors of [Reference 1] demonstrated that it must be true" when they are the author of Reference 1 and the result can be derived from their work. If that is true, but they failed to make the derivation explicitly, then the work is new, even if it seems obvious in retrospect.
In all cases, however, work that is previously relevant must be cited. A paper that fails to credit adequately the work upon which it builds, either directly or indirectly, is in err. The reason is not ego or career based, but rather that a complete accounting of previous or competitive work allows the reader to judge the individual contribution.
Having answered these questions, I try to apply the answers to two more general questions: How can the paper be improved? and Why should the paper be or not be published? In both cases criticisms should be specific and constructive. For each flaw, a review should suggest a remedy. If the work is not novel, the review should cite explicitly the pre-empting work. If citations are missing, the review should give them. If the paper is to be rejected, are there specific changes that would make it acceptable. If the work is accepted, what future work would make a similarly useful impact? Keith Marzullo at UCSD also provides his classes with useful review guidelines under the "Reviews" button (he uses frames). His advice on the subject is excellent.
In this class, your reviews will take the form of brief slide presentations. Each presentation must consist of at least 5 slides, where each slide serves the following reviewing purpose.
It is fine to use bullet points, short sentences, or prose on each slide. There is no need to include figures or drawings (although you may do so if you choose).
Important:You must use PDF, HTML, or PPT for your reviews so that they can easily be displayed in class without the usual digressions for audio-visual misery.