Two things are needed to make it successful - First, comments made on articles can't be made by anonymous and good system to verify the commenters need to be established. Second, for the system to work, then scientists have to comment - On the social web - less than 1% of viewers comment or contribute content. This could end up being a serious problem - Perhaps the authors of the papers and the F1000 faculty can start the conversations and encourage others to join in. For example, this article has been viewed 580 times so far and I am first one to comment!! There are a lot of lurkers out (which is fine) but you need to get comments for this works

Post publication review needs to prove itself and gain credibility before positioning as an alternative or better mechanism. Initially it may be best as a complimentary form of review of material. Areas where it could work best include calling out of flaws, additional relevant information & positive reinforcement (draws attention). However, I think it will not replace conventional review as this acts as a limiting filter and QC step. However, there are many forms of data that are squeezed into papers that are inappropriate in that medium such as theses, posters, etc. Post pub review would be a good way of highlighting these contributions and might eventually relieve the overburden of papers published due to being only currency recognised.

A corollary of post pub review is a means of recognising the reviewers in order to encourage input and participation This could/should be subtle and unobtrusive - as in Wikipedia.

Thanks Abdallah, for the invitation to discuss this here. For me, the most useful thing about PPPR is that it can connect the actual "users" (the experimenters). I think it is a critical part of the OA movement and will help to move prestige to open-journals. Exciting times ahead!

that is a good point by Bala. Connecting with experimenters and especially if they engage in conversations could be beneficial to all parties. The one part that I think current principal investigators can do is to encourage their experimenters whether they are students or postdocs to engage in these conversations. Too many times, the head of lab is paranoid about getting scooped and that translates down across the entire lab

Thanks to Abdallah for inviting me to chat. I would personally like to see us move over to PPPR entirely in the long run. I'm not personally convinced that pre-publication peer review does a particularly good job of filtering and in the long run I think that rigorous PPPR will force authors to get friendly peer reviews in advance and make sure that the quality of their papers is high. There's nothing worse that having a public paper that that is broadly criticized in this system, whereas now if it's in Nature that is still a net plus for someones career. As others have pointed out, this may take some time. I think that we basically need to shift the social contract from pre-pub peer review to PPPR, but that will be a slow process.

Ethan, you are assuming papers will attract significant #s of comments/reviews. Given that only a fraction of papers are ever cited, will there be a significant % of papers that attract no comments? Perhaps this is good, but it will only work if there's broad buy-in and willingness of a significant number of people to not only participate in review, but also people wiling to put out their material in the first place.

There are definitely advantages to transparency of the reviews & reviewers. EMBO J has instituted some interesting policies that make reviews available - but only for accepted papers. It may be a challenge to ensure quality of PPPR since it will make it relatively easy for a "drive-by" comment that submarines the paper.

Actually, I just don't really care about those papers. If no one is paying any attention to them then I'm not personally worried about how right they are. That said, you are right that buy in on both sides is critical and while I think we will end up there I do think it will take some time.

I do like the transparent review processes that some journals have implemented. I just think that making it fully open and real time is better.

The "drive-by" is certainly a concern and it is why Abdallah is correct that PPPR either has to use real names or a good reputation system like that being proposed by hypothes.is. If Identity is maintained then drive-by's hurt the commenter.

agree that the system needs to encourage participation which is a challenge for all those on the social web. However, by getting the author of the paper to invite others to comment on the paper then the system might start to work and get people to comment. The key part is for Principal investigators to encourage their own post-docs and students to comment and get over the paranoia of getting scooped if you reveal that you are interested in an article. This, in my opinion, is a serious problem in the scientific community and the change might need to come from top-down

Many people are generally unwilling to "schlepp" their work to others in the hope of attracting comments. Moreover, if you are invited to review by someone you know, and decide to comment, the likelihood is that you will be gentle/positive. The last thing we need is a bunch of five star type reviews that say the same thing. Others might be less likely to stick their necks out and point out the faults. There also needs to be some sort of vetting to prevent generation of false/fabricated reviewers (e.g. pseudonyms for the author). It's amazing the lengths some people already go. retractionwatch.wordpress.com

agree with all the points you make Jim and that is a great link you provide. I am just thinking of ways to initiative the process but in the end the growth of comment has to be organic in nature. Perhaps, having verified accounts such as those you see on Twitter could be one way to ensure that commenters are legit people. Anyways, this is a big topic but is one that needs to be discussed more openly within the scientific circles

A core issue separate from transparency and accessibility is advertising dollars. This drives a lot of management-level decisions. The better a given journal's impact factor, the more advertisers are attracted and the more dollars are generated. That impact factor increases through publication of high-quality research, the submissions of which are often driven by a journal's impact factor (seen in the public as a generalized prestige for some journals). It's out of the reach of most, if not all, journals to publish non-peer-reviewed articles.

There's also a certain amount of liability involved. If a researcher has a typo in a paper that creates a dosing error, then that has huge consequences for a potentially large group.

Only a relatively few journals attract enough income from advertising to make a significant difference to their business model. Many journals, like PLoS ONE, etc., are entirely on-line and so their costs are reduced. Both open & closed access journals charge authors - but only if a paper is accepted. PPPR accepts every paper received & so would have greater support from author contributions. It's possible/likely that the author charges could relate to the relative impact of the particular PPPR journal. What is to stop a PPPR (can we call it P3R?) journal working to attract commentors itself? Serves to increase profile of the papers & the income of the P3R journal. Is there any need to retract a P3R paper (I guess to avoid embarrassment).

Print advertising dollars are huge revenue generators. The journal I work for as an editor is fourth in our specialty by impact factor, but ad revenues are nevertheless high. This is even more so for more prestigious journals. Online revenues don't compare. Too, journals are often published by societies that depend on those revenues and the exposure the journal offers.

Ah, but your journal is one of the cream that does benefit from advertising - likely 5% of all (3000) biomedical journals? I can understand (and previously argued) that quality control will be challenging for P3R and that there is no way to gate-keep submissions (except perhaps via author charges but that would favour the "Mitt Romney's" of the scientific world). There is also an absolute requirement for professional editing. Otherwise, what is to stop any scientist self-publishing (aside from not having a PMID)? Talking of which, has anyone self-published through iTunes Author?

Not just "the cream." Given the amount of ad revenues spent each year in medical journals alone (it was in the hundreds of millions as far back as 2003), the issue affects scientific publishers of all sizes and ranges. (Subscription revenues are a separate issue.) And liability is yet another issue. P3R is going to have to exist in a low-risk and thus low-return area because journals are going to be reluctant to publish research that hasn't been vetted for errors before release. Dosing errors, for instance, have critical consequences in people's lives.

But P3R doesn't obviate editors. Dosing errors should also be identified post publication. Not sure that anyone is going to leap to dosing patients based on an immediate publication. Personally, I think there is a case for a combination approach of pre and post review (with pre reviews being published alongside paper). This actually might be the best way to encourage and legitimize P3R while gaining some traction among "reputable" journals. The concern (reality) is that initially P3R will be associated with lower quality and hence condemned to the periphery - reducing its opportunity to attract high quality work.

No one single publication will have the power to change clinical treatments. This is especially true for research from basic science work which is considered to be preliminary work that needs many more years before it ever reaches the clinic. The P3R path will be an open and transparent process and the community will judge it. It will be great to see major research institutes support it and big name labs back it up. I view anything that increase productive scientific communication as a good thing because I don't think we have enough of that.

Jim: Never said it did; I addressed the expenses involved in producing a journal. Agreed that pre- and post-reviewed articles would contribute more to the dialogue.

Abdallah: You'd be surprised about changing treatments. If you're talking about P3R for only basic science, then fine, but what are you going to do about pharma-sponsored, proprietary research? That's got legal protections and won't go public, and there's a ton of money involved in it.

Here is me jumping off this branch to catch something!
goo.gl

Bala, you suggest anonymous reviews/commentors would be a good thing. I respectfully disagree. There is a huge difference between Reddit (or the AVC blog) and research publications. Credibility cannot be assumed, only earned and the "product" of publication is directly tied to future funding potential. Hence, the system cannot be freeform in such an idealized manner - certainly not if it is to gain any traction. I hope the P3R initiatives gain momentum, but they first need to gain credibility (and a sustainable business model).

Jim: I'd like to run the anonymous reviews/commentors experiment just to see what happens. Perhaps it can be semi-anonymous, similar to how the authors do not know the identity of 3-4 reviewers (but the editor does) in the current monoculture model. I wonder whether some research topics are more accommodating to this option. Anyway, I think anon comments will add a whole new dynamics to the discussion process.

If the comments are anonymous, how will they be rewarded and encouraged? I know that current anonymous reviewers are not rewarded directly and I think that is a problem, but they are rewarded in other indirect ways such as access to pre-published data, possible pick-up for commentry articles, etc. as well as out of scientific obligation (we depend on each other). But if people are to put effort into reading and then writing reviews, then there needs to be some means to value that effort. It could be done by a registration system, I guess, but isn't full transparency the aim? You are right about adding new dynamics - that may not be so positive.

One way to reward reviewers is through CMEs, which is currently done by some journals. Those don't need to be publicly announced, but they are important to reviewers.

Jim: Absolutely agree with the reward aspects. It will help convince many commenters not to be anon. I'm just making a case for flexibility to make the ecosystem more vibrant and open. I think it will be worthwhile to try a 3PR system where the option is left to the commenter.

I am generally not in favour of annonymous comments on Reddit or any other online sites. I have to agree with Jim on that the reward potential of this system should be a major driving factor. Still, this is going to be a bit of an experimental system and different journals can try different methods. I mean even the main stream publishing media companies are still trying to figure commenting on their sites. Instead of trying to figure a perfect system, lets just start testing some and adjust accordingly!!

Heather, sorry to sound naive but what are CMEs?

Abd: I still don't understand the core logic behind why an interested individual has to reveal his/her identity just to make a comment. How is this information relevant to the scientific discussion, especially in the absence of any evidence that this interferes with the process detrimentally somehow. It's like saying everyone should identify themselves before asking a speaker a question in public sphere. It doesn't matter who the person is, the individual is most likely a tax-payer who paid for the research! Anyway, I think such distractions may impede the free flow of conversations. Oh, yes, I'm glad to hear that you support experimentations. Considering that we have such a dismal level of commenting one has to try several methods.

Abdallah: Good morning! CME = continuing medical education. Professionals in a wide variety of disciplines are required to earn a certain number of CME units per year to keep up their accreditation by one body or another. CMEs come in the form of webinars, self-study units, lectures, etc.

Bala, the identity of the authors is known, why not the identity of the commenters? It would help keep things civil, considered and allow reputations to be established. I can see that commenters may feel intimidated in posting critically but civil discourse with valid points are their own defence. There are two extremes: drive-by comments that are defamatory and sycophantic comments - each distracting and adding no value. Methods to guard against either abuse (such as moderator approval) would reduce a need for identification (esp. if the journal required registration for commenting).

The main reason I'm generally in favor of experimenting with anon-option for commenters is to see if it increases the participation. I too think there is much for the commenter to lose by being anon.. but again that's the commenters choice. I'm quite confident that such an option won't create much friction, this is based on witnessing numerous instances of decent online discussions amongst perfect strangers in other platforms, for eg. Twitter and Reddit. So, if I were to start a 3PR platform I'd allow anon-commenting option. Having said that, as a safeguard, I'd also have a registration system and a comment-upvote/downvote counters in place to make the self-correcting/moderation process more effective.

Ooooh, up/down votes..... No chance there of manipulation :) The biggest concern has to be with a lack of P3R participation and QC. If the approach results in a large number of generally weak papers with few comments, it will simply become a dumping ground for mediocrity. There is, of course, plenty of that already (I'd argue the majority of published material - when considering the whole gamut, is mediocre). Efforts such as F1000 do provide a service in shining light on interesting papers that people may have missed and makes an effort to not focus on the high profile journals. Nonetheless, it is very rare to see an F1000 review on a paper in a truly remote journal.

Jim: I think the 3PR ecosystem can evolve differently as well. On the bright side, we may see a more efficient "idea selection process" and faster turn around time through rapid critiquing/prototyping. There is also this promise of new collaborations that could move projects towards a formal publication. I have seen some signs of these trends with my own data sets that I have submitted to Figshare. I must say, there is something very organic (for the lack of a better word) about the 3PR idea and I like it a lot. I suppose in the end it all comes down how does an individual experimenter benefit from it and will he/she come back to release more data into the platform? I'm very optimistic about this new landscape.

Jim: Another thought. 3PR may work better in smaller niche research topics than say super-populated areas like cancer biol.

Unrestricted open access for submission paired with anonymous reviewers could (and probably would) lead to a free-for-all of internet science trolls and/or fanatics of every flavor. On the other hand, some research would go unnoticed and receive little benefit of being published 3PR. There would need to be a form of signaling that permits reviewers and authors to signal to the community that they provide informed criticism and/or quality original research, while also incentivizing new authors/reviewers to participate in meaningful publishing and reviewing. A signal could also solve some of the anonymity problem (weight carried by signal, not name). A hybrid of the traditional model and the 3PR model may support something similar.