I am on the NeurIPS advisory board and on the ICLR board, and I have been involved in the organization of these conferences at all levels, for many years. I have been part of discussions with program committees about how to improve these conferences, but usually the discussions are about incremental changes. I wonder if it may be time for rethinking the overall publication process in the field of machine learning (ML).
The landscape has changed in the last couple of decades. In great part this is thanks to the popularity of arXiv, which has greatly accelerated the cycle of information discovery and dissemination. We now have many conferences publishing ML papers (for example my group publishes mostly at NeurIPS, ICML and ICLR, but also in major conferences in computer vision and natural language processing), which means we go from one deadline to the next every two months, roughly.
The research culture has also changed in the last few decades. It is more competitive, everything is happening fast and putting a lot of pressure on everyone, the field has grown exponentially in size, students are more protective of their ideas and in a hurry to put them out, by fear that someone else would be working on the same thing elsewhere, and in general a PhD ends up with at least 50% more papers than what I gather it was 20 or 30 years ago.
The field has almost completely switched to a conference publication model (in fact a large part of computer science has done so too), and each conference paper does not get the chance to be cleaned up as well as a typical journal paper, rarely benefitting from the repeated iterations to improve the paper which is typical of journals. So we are more productive, on the surface, but this stress, productivity fast pace have a price on the depth and quality of the papers we produce. Many papers end up being submitted that in the past would not have been submitted. They may contain errors, lack in rigour or simply be rather incremental.
In the rush preceding a conference deadline, many papers are produced, but there is not enough time to check things properly and the race to put out more papers (especially as first or equal-first author) is humanly crushing. On the other hand, I am convinced that some of the most important advances have come through a slower process, with the time to think deeply, to step back, and to verify things carefully. Pressure has a negative effect on the quality of the science we generate. I would like us to think about Slow Science (check their manifesto!).
Motivated by this feeling, I have been thinking of a potentially different publication model for ML, which has some similarity to what has been experimented elsewhere (e.g., VLDB). I shared these thoughts with the NeurIPS board, and I share them to you here. Here is the content of my message to the board:
I would like to see more discussion about ideas to improve the publication process as a whole in ML, with reviewing being a crucial element. I’d certainly like to hear your thoughts.
My feeling is that besides the poor incentives for reviewing, our current system incentivizes incremental work and creates a lot of pressure and stress on grad students (and researchers in general) to submit as many papers as possible at each deadline. Students sometimes come to me two months before a deadline asking if I have ideas of something which could be achieved in two months.
In addition, we now have a bunch of ML conferences (in particular NeurIPS, ICML and ICLR) with a strong overlap in their content and community, so people just resubmit their rejected work to the next conference and draw a new sample of reviewers (and because of the noise, it means a paper eventually gets accepted, after using a lot of resources from the community). Plus the fact that all good reviewers are in demand at very specific weeks of the year makes it harder for area chairs to find the appropriate reviewer for their paper (unlike what happens with journals).
This brings to mind a different model, one where papers are first submitted to a fast turnaround journal (which could be JMLR in this case) and program committees of each conference then pick the papers they like the most from the list of already accepted and reviewed (and scored) papers (assuming the authors are interested in having their work presented at a conference).
In the old days, conferences were important to speed up the research cycle and have a fast turnaround of ideas. But now we have arXiv which plays that role much better, so the main role of conferences, besides the socializing, should be to select work to be highlighted and presented orally, to create a diversified offer of the best and most important ideas arising in our community, to synchronize researchers around this progress. It doesn’t even have to be super-recent work, it could be work which got done 1 or 2 years ago and is only recently picking up steam in terms of impact. The deadline system of conferences creates an incentive to submit half-baked work (and often not even fix things properly later if the paper is accepted, rather move on to another publication). If there is an implicit soft deadline (because if I submit my paper now to JMLR there is uncertainty as to when it will be accepted and available for selection by a conference) then there is an incentive to continue working on the paper until it is better polished, rather than submit it too early. In addition, the richer iterative feedback of the journal process should lead to higher-quality results at the end of the day. And having our work in journal form would make it easier for ML researchers to collaborate with researchers in other disciplines who value journals and not conferences.
I guess this is the beginning of a discussion, and the many researchers who have participated in the conference and journal process in ML would certainly have interesting things to say towards improving this process. And everyone in this community who submits or reviews papers has information about what works and what doesn’t. Let’s put our heads together and explore how we can at the same time improve the quality of our science and improve our lives as human beings.
P.S. see follow-up discussion there.
Photo by Brian Erickson on Unsplash