As I have written previously, academic computer science differs from other scientific disciplines in its heavy use of peer-reviewed conference publications.
Since other disciplines’ conferences typically do not employ peer review, results published at highly selective computer science conferences may not be given the credit they deserve, i.e., the same credit they would receive if published in a similarly selective journal.
The main remedy has simply been to explain the situation to the possibly confused party, be it a dean or provost or a colleague from another department. But this remedy is sometimes ineffective: At some institutions, scientists risk a poor evaluation if they publish too few journal articles, but they risk muting the influence of their work in their own community if they publish too few articles at top conferences.
The ACM publications board has recently put forth a proposal that takes this problem head on by formally recognizing conference publications as equal in quality to journal publications. How? By collecting them in a special journal series called the Proceedings of the ACM (PACM).
In this post I briefly summarize the motivation and substance of the ACM proposal and provide some thoughts about it. In the end, I support it, but with some caveats. You have the opportunity to voice your own opinion via survey. You can also read other opinions for (by Kathryn McKinley) and against (by David S. Rosenblum) the proposal (if you can get past the ACM paywall, but that’s a topic for another day…).
Update: PACM has been approved, as has a new journal series called PACM PL that will collect papers accepted by major SIGPLAN Conferences. It will debut during late 2017.
Comparing Conferences and Journals
In disciplines like Physics, conference presentations are typically not peer reviewed — the conference is about reporting on previously published results (in a peer reviewed journal) and germinating new ideas among colleagues.
In computer science, conferences serve a similar purpose, but they are also the publication vehicle: To present at a major conference, you submit a paper that describes a substantial result, e.g., an innovation in compiler design along with an evaluation of it on a set of benchmarks. A program committee consisting of area experts chosen a the program chair reviews the paper for quality (including correctness), and only if there is sufficient consensus is the paper accepted. All of the accepted papers are published in the conference “proceedings.”
The process for top journals is similar. When a paper is submitted to a journal, an editor asks 2-3 (sometimes more) experts to review the paper for quality, and based on their reviews decides to accept the paper or not. There may be additional revisions and re-review required before finally accepting the paper.
We can see that the program chair of a conference acts like an editor at a journal, and the program committee serves as the expert reviewers.[ref]Conference committees often solicit outside experts to fill gaps in the committee’s expertise.[/ref] The conference review process sometimes allows reviewed revisions, too (e.g., OOPSLA’s two-phase process, or “shepherding” by PLDI), or such revisions may occur “between conferences” — i.e., a paper is rejected with the expectation it will be submitted to another conference in short order, and then re-reviewed.
It is easy to see, just by looking at the papers themselves, that the process for top conferences engenders high quality results. Quantitatively, simply reformatting a POPL or PLDI paper into ACM “Transactions” format (the format used by ACM journals) often turns a 10-12 page conference paper into 25-30 pages. Compare this to the relatively shorter journal format used in other disciplines; e.g., a paper in Nature is 5 pages long.
The Proceedings of the ACM
So, top conference papers in computer science are of the same ilk as journal papers, and should be viewed as such. How to convince the uninitiated of this? The ACM publication board’s proposal is to start collecting conference proceedings in a journal called PACM, for Proceedings of the ACM (or something else if we don’t like that name). Different communities would have their own “series” of PACM, e.g., PACM for Programming Languages. How a SIG would set up its series is somewhat up to them, but the conferences included have a sound review process, i.e., one of the sort I described above.
I should add that this proposal is particularly impactful for the PL community because of all the sub-areas of CS, programming languages is perhaps the most reliant on conference publications. Our flagship journal, TOPLAS, publishes no more than a couple of dozen papers per year (it has published 13 so far this year), while hundreds of papers appear at top conferences.
My opinion: Supportive, with caveats
In general I’m in favor of this proposal because it solves a problem for many people. But to really work it should be as streamlined as possible — conference papers already are of high quality, so why add extra hoops to fix what amounts to a marketing problem?
Here are a three elements of the current proposal that I think should be relaxed:
- This process would require written reviews by a minimum number of three qualified reviewers. While three is a fine and common number, I don’t think the bar needs to be that high. Highly regarded journals like Science and Nature require two, not three, reviewers (see Nature‘s process, and Science‘s process).
- The review process must permit at least one substantive revision by the authors and review of that revision by the reviewers to determine whether their concerns are adequately met. I also disagree with this requirement. While rounds of review might be useful for authors, they are not necessary for ensuring quality, which is the key motivator here. If the PC doesn’t think the paper is strong enough it can just reject it, and the paper can go to the next conference for re-review. That said, the (selective) use of shepherding ought to be sufficient to meet this requirement; as with journals, some papers have too much to fix to be shepherded, and can be rejected, while others can be changed (with vetting) in sufficient time.
- Papers may not have artificial limits on length that would prevent full disclosure of references, documentation of methods, and so on. Again: Too strong. Look at Nature: it has both page limits on material, and a limit on the total number of references. I’m not against relaxing page limits or allowing more references. I’m against the claim that journal papers must not have page limits.
Here’s another element that is important to get right. At the recent SIGB meeting, Kathryn McKinley pointed out that getting the right name is crucial. PLDI, POPL, OOPSLA, ICFP, etc. are all substantial brands. Over the years, many people have come to understand that a paper from one of these conferences is likely to be very good. Therefore, we want to keep that brand as part of the name of the PACM series, so that its effect is retained.
My last point, which is not really about PACM itself but what happens if we adopt it, is that the existence of PACM should not preclude other conference/journal combinations. One attractive model is “submit to a journal, present at a conference.” As examples: You can submit a paper to the ACM TACO journal, and once accepted it can be presented at the HiPEAC conference. You can do the same with PVLDB (journal) and VLDB (conference). I think this model has a lot to speak for it, and we should explore using it for our conferences. For example, one benefit is that you have the option to present your paper at the conference, it’s not a requirement; if you’d prefer not to travel, you don’t have to. You also get rolling submission deadlines (rather than having to rewrite your paper for the slightly, or vastly, different audience of the next conference). Agreeing to PACM should not be the end of the story (and I don’t think it the proposal says that it should be).
Conclusion
I support this new ACM proposal, with caveats that I think are easily addressed. I think the proposed journal series solves a real problem, and if done right should be little to no additional work for authors, conference PC members, etc., once the ball is rolling.
Please share your thoughts in the comments below, and in the survey that ACM is taking about the idea. As ACM SIGPLAN Chair, I will take your feedback up the chain. Things are moving fast, and changes could start happening by the end of the year.
The perception that journal papers should be long would hurt the field itself.
A good portion of the current 10-page conference papers are speculations, justification of limitation of the technique described in the paper or some tables of insignificant data.
The idea that scientific contributions must be at least 10 pages long and the longer the more scientific is absurd, but somehow accepted by CS journals and conferences. Lots of papers in other fields are way less than 10 pages long. Most papers in PLOS ONE have less than 10 pages and if you change it to ACM format, it is less than 6 pages.
Mike,
My feeling is that, in practice, the only issue is the poor choice of name of the proposal. All of the other points can be finessed.
The number of reviewers is consistent with the requirements of major SIGPLAN conferences. So, while arbitrary, it won’t change the way we operate.
The additional round of revision is what OOPSLA does and roughly corresponds to institutionalized shepherding. The main impact it that the review schedule must leave sufficient time for reviewers to have another look. While the proposal mandates it, the PC of a given conference may choose to only accept unconditionally. There is nothing in the proposal that mandates that reviewers must look at the papers again.
For the page limit, my personal preference is to relax our requirements (I never complained about OOPSLAs’ 20 page papers, I found them easier to read because authors had the space to give examples and explain details). But, if a conference does not want to go that way, one can limit the body of the paper to 10 pages and allow unlimited appendices — with the understanding that reviewers are not required to read them. This is standard in journals in other disciplines.
Great points. I agree with the goals of relaxing page limits and encouraging a second look. My feeling is that these should not be tied to the Conference/journal issue, and that we should pursue them separately. But, as you say, it’s not hard on our conferences to accept them, by and large.
This is a great idea and will help computer science faculty to gain recognition not only in our professional community but also in our own work place.
There are many ACM conferences that adopt review processes that are as rigorous as top journals. For those conferences, a second review may not be necessary. Perhaps, the editor-in-chief and the PC chair of a conference have better insight on this matter.
I am quite concerned with the proposal.
I think it goes into the direction of making it possible to produce more research faster (more defined as in “more papers on more topics”). Once something becomes possible, people will certainly do it, because of the highly competitive situation on the job market. As a result, we will have even more of shallow papers around, and I am not particularly happy about the depth of insight in an average CS paper already (including my own!). The journal system forces people to develop some lines of work further. This proposal will likely have the opposite effect.
Possibly, more people will get attracted to PACM conferences (which are likely already honeypots), more smaller domain specific venues will die out. Possibly, we will all work more and faster to meet new standard expectations for papers in a few years, given that you can now hop research topics even faster.
Finally, other conferences, pressed by competition are likely to follow with similar arrangements, but it is unclear whether with a similar quality level. The impact of this decision on CS research as a whole is hard to predict.
Incidentally, the proposal may also seriously undermine several existing journals. This might matter little for SIGPLAN; as you admit yourself, TOPLAS is marginal. There do exist successful and reasonably high volume journals in other research areas, though. I am more in favor of strengthening these, and creating new ones, than hacking the system with PACM. This is why, cautiously, I do support journal-first initiatives, where journal papers can be presented in conferences (on a voluntary basis).
I am convinced that our reliance on conferences has only historical reasons. There is no reason today, why we should be obsessed with conferences so much. Journal publication process is just as efficient as for top conferences (if you take out the time it takes the authors to get acceptance — but this is necessary for maintaining high quality; the publication time is otherwise comparable). There is also no reason to travel so insanely as we do. Someone here has invented the Internet in the meantime, and we learnt how to use it for research communication. Conferences are an extremely expensive and time consuming way of advertising research. We need more venues where the objective will be discussion not advertising. Exactly like the other research areas have. Wait, these are actually those that rely on journals for dissemination …
I agree with many of the things you say. In particular:
– I also favor a journal-first approach. I like the rolling deadlines, quick turnaround, and the option to present at the conference. I think this approach can be followed in parallel with PACM. That is, if a conference and journal want to adopt a journal-first approach then there is nothing requiring them to produce a PACM series instead.
– You may be right that some journals will be undermined, since now there will be less administrative pressure to submit to them. But my experience for SIGPLAN, at least, is that the end result will be positive because more people will be aware of work that instead goes to higher profile (conference) venues. There will also be less time-wasting on extending perfectly good conference results only for the purpose of sending them to a journal, due to an administrative expectation of a tenure review board. (Don’t get me wrong: I’m a fan of the extended journal article, and I have written many of them. But such articles should fill a need, not a administrative bucket list.)
– You may be right that PACM will increase the pace of research since more work will go to conference-style venues. However, I don’t think this will reduce quality. The point is that PACM conferences are essentially journals already, given their rigorous review processes; PACM is simply recognizing that fact.
– That said, we might be a little sad that conference papers are so often of journal quality. Someone emailed me with the lament that today’s conference papers need to be so solid and bullet-proof to get published that they rarely contain the exciting, but half-baked ideas they once did. Conferences like SNAPL are aiming to rectify this problem, and I hope that they will flourish. These would specifically not qualify for PACM status due to less rigorous review process expectations.
Mike,
thanks for offering a separate opinion and stating your preference. I feel that you left out important benefits of using journals over conferences. Jeremy Siek already summed up most of the problems, so I don’t need to restate what’s been said very well before: http://siek.blogspot.co.at/2014/01/the-publication-process-in-programming.html
Reviewer Load and Reviewer Expertise are the primary pain points of conferences today, and inevitably lead to Publication Delay. From the perspective of junior members of the SIGPLAN community, it is not clear why we don’t just follow other disciplines, like mathematics and biology. The only reason I can think of is that somebody *is* benefiting from the status quo, and they don’t want things to change. That aside, there is no rational reason to stick with conferences. I believe promoting conference proceedings to journals is the wrong direction to move forward, as it does not address the aforementioned pain points.
Thanks for sharing this. I don’t disagree that there are disadvantages with the conference system, and I agree with many of the points that Jeremy raises. But I think that these are beside the point of the PACM, which is to make official what many (but not all) already know, unofficially: that CS conference papers are as good as journal papers, in terms of quality, and therefore they should “count” just as much in tenure, promotion, hiring, etc. Then those people for which this particular recognition matters can get on with it, and we can focus on further improvements to the system, including those suggested by Jeremy, among others. As I said elsewhere, I have a good feeling about the journal-first model, and I would encourage more experimentation in this direction.
[I have not read the above replies. Apologies if I repeat points.]
Mike, I think you have a number of points wrong about journals vs conferences and about the ACM’s decision. Here is my enumeration from the perspective of an EIC for a journal (JFP. Cambridge Press).
1. Don’t confuse Nature and Science with proper journals. They have basically become magazines; perhaps they are not like CACM, but they seem to be on the way there.
2. A journal review process is quite different from a conference review process.
(a) A (managing) journal editor will pick people who he/she thinks are experts in the field. It is rare to get a full set of experts on a time-pressed conference committee.
(b) If he takes his task seriously, he will “referee” the process. A conference chairman does not have the time to referee 150 to 300 submissions. See http://www.ccs.neu.edu/home/matthias/Thoughts/_Non_refereed_conferences.html
(c) There is no artificial time limit on the review process. Yes, we all try to get submissions through the review process in a timely fashion, but if someone thinks that a results needs 62 pages to be presented, we will not force a reviewer to get thru this paper in 4 weeks. [Real story, all 62 pages were needed. It was a true journal paper.]
(d) A member of a conference committee reviews 10-20 papers in a short amount of time. The qualities of the resulting reviews are on average far below those that I see as EIC. I read almost all rJFP eviews that go through managing editors, so I can say this with confidence. I am not saying that all journal reviews are perfect and all conference reviews are bad. I am not saying that all journal submissions get better treatment than all conference submissions. I am talking about average quality.
(e) Conference committees decide about the fate of 120 to 350 submissions under tremendous time pressure (a day or two in real time, a week in virtual meetings). The taste of the committee, the member’s background, their articulation skills, their personalities and other factors play as much of a role as the quality of the reviews. In contrast, a journal decision is deliberate, one paper at a time, arrived at by a single managing editor-referee based on the advice of two, three, or four reviewers.
3. The ACM solution is mostly a sham. It basically acknowledges that computer scientists are not willing to live up to the journal process of other disciplines, so we will create a quasi-journal to make other disciplines think we do. Shame. Shame on us.
4. Are our conference papers as good as journal papers? As journal papers in other disciplines? How would you measure this? Citation rates? Silly! Download rates? Sillier. Actual re-use of the result??? We keep saying such things and we can’t back them up.
We should fix our publication system for real, not wth patches.
— Matthias
Thanks for your thoughts, Matthias.
I’m dubious of your claims of the improvements of the journal process given my experience a journal and conference author, journal AE, and multiple-time conference chair.
From my perspective, you are presenting journals in the best possible light, and conferences in the worst possible light. The truth is in the middle.
Just as one example, you say that conference decisions take place over 1-2 days. I have two problems with this statement. First of all, major SIGPLAN conferences mandate electronic discussion prior to the PC meeting. I would say that most decisions are really made during this period, and finally worked out for good at the PC meeting. That is, if you look simply at the state of the scores and reviews just prior to the PC meeting, and picked the top N papers, you will choose 80% of what ends up as the final program. And this electronic discuss is very healthy and important for making good decisions.
By contrast, in my experience with journals, the default approach is for the AE to get reviews from experts, and then to make a decision based on what comes back. That is, by default, reviewers do not get to see their fellows’ reviews prior to a decision being made. Nor can they initiate any sort of discussion. While an AE can make a discussion possible, I do not think this is the norm. (I do it for TOPLAS papers I manage, when there is contention.)
Another example is the task of getting expert reviewers. Having all experts sounds great, but in practice, this is hard to achieve even in a journal process. At least, I have a hard time getting people to agree to review journal papers when I ask, as a TOPLAS AE. Moreover, I’m not convinced we want all “experts” as opposed to informed outsiders. A paper’s impact should not be only among those who know the area, it should also be among outsiders who want to get into the area. As such, these informed outsiders present a valuable perspective. At the least they can speak about how to improve presentation. At best, they can put the contribution of the paper in a more objective light.
A final example I’ll address is time spent reviewing. The presentation of a journal review process in which everyone is conscientious and takes the time to read the whole paper and still gets it back in a reasonable amount of time, etc. doesn’t scale. it also leaves out some perspective, from having reviewed other papers at the same time. On the flip side, the deadline for the conference process does create time pressure and lead to hasty decisions, but it also ensures focused attention and creates a useful context and perspective for judgment.
I could go on. I think there’s nuance here. Maybe the best journal review process is better than the best conference review process. But in practice I don’t think it’s that much better, if it’s better at all. And I think the point of PACM is to acknowledge that the conference review process as carried out by many top conferences is on par with journal processes as they are carried out today.
I am not opposed to further improving the process. I like journal-first. See my comments above. But I think that the PACM approach is reasonable, and low cost, and rectifies an issue that many in our community face. Hence it’s worth doing.
Mike, I accept your criticisms but I anticipated them and already wrote “not all journal reviews are perfect.” I missed nuance in this whole discussion so I thought I’d present a counter-point. [I will say things must have improved a lot since I last chaired a conference in 2012 :-)]
None of your response goes to the heart of the problem in the end, however. Indeed, I think your sentence that equates the journal reviewing of TOPAS with that of our conferences is the key problem, namely, as I said our (CS) publication culture/process is broken. Instead of tackling the depth of it, we tackle the symptom of counting beans.
I want to conclude with wholehearted agreement with your “journals first” proposal. Two of my most widely cited papers appeared in journals only, and it makes me happy.
Of course, what we need is a change of incentives. It is not quantity that matters. It’s quality. Or with my usual hat on: I just don’t understand how John Reynolds could have gotten tenure at CMU — just look at his sparse publication list and cry about a waste of a tenure slot.
Let’s publish less. Let’s base tenure cases on a candidate’s self-selected best five papers from the last 8 years. Let’s not allow PhD graduates to apply for jobs with a CV; ask them for descriptions of their best three ideas instead with a max of three citations. Raise the bar for what we consider an accomplishment.
Points well taken. Now we just have to figure out what “quality” is … 🙂
I know it when I see it and .. this … is not that. — Counterpoint accepted but it doesn’t absolve us of solving this problem.
Pingback: Carbon Footprint of Conference Travel - The PL Enthusiast