question for fellow #academics out there: How do you feel about signed peer review? My senior colleagues all recoil in horror at the mere suggestion, but I've seen enough unprofessional comments in peer reviews that I'm inclined to think it might be a good idea. Retaliation is a serious concern, though.
@bgcarlisle Absolutely! Do you think it would help if peer review was still managed (e.g. editors pick reviewers, manage reviews), but would be made public at the end of the reviewing process?
@tiana_athriel I wonder if this is the sort of thing that could be settled with a randomised trial
It's such a complex system, I'm not sure that it's possible to foresee from the armchair what the trade-offs would possibly be, let alone their relative magnitudes
@bgcarlisle That would be an amazing experiment! But it might be hard to convince scientists/journals to participate.
@santiago Oh, personal attacks on student authors are probably the most egregious.
@tiana_athriel I think signing peer review would 1. decrease snideness, 2. make it easier to follow up on unclear comments, 3. make bullshitters identifiable so we can remove them from the review pool
@DialMforMara That would be the hope. The question is what happens when there's a large power differential between a (senior) author and a (junior) reviewer ...
@tiana_athriel can there be codes of conduct for responses to reviews, that prevent retaliation (or reduce fear of retaliation)? Or does that just not generally work?
@DialMforMara I think some journals already have codes of conduct, but it's not clear that they can prevent retaliation. What should the journal do if the senior person starts badmouthing the junior person at conferences? Codes of Conduct are definitely a step in the right direction, though!
@tiana_athriel Yeah. How far does the journal's responsibility extend, and when does it become the responsibility of the conference organizers or a university to make sure senior professors aren't being assholes?
@tiana_athriel i'm not an academic, by any stretch, but i spent a little time as a peon-grade editorial assistant at a university press, which meant that i coordinated the machinery of peer review on a bunch of manuscripts.
i guess i have mixed feelings. the process i saw had its fuzziness and subjectivities, and definitely wasn't isolated from political considerations, but i also think it was fairly honest and sincere in its operation...
@tiana_athriel ...and i can see that it gave cover for some critical input that might have had a hard time happening in public, out in full view with reputations and relationships at stake.
@tiana_athriel (mostly though the more i think about it the less i feel like i'm qualified to comment at all.)
@brennen Yes, I think many reviewers honestly try to do a good job. But we're all human, and sometimes I wonder if even with good intentions, sometimes our anonymous reviews are more critical (in unhelpful ways) and condescending than they should be due to a lack of accountability.
@tiana_athriel
this is the peer review process at the Journal of Peer Production, a young journal: http://peerproduction.net/peer-review/process/
note that upon acceptance and publication, reviewers can say if they want to be anonymous or not - their reviews are published, along with the manuscript drafts.
I think this makes for both careful first submissions and thoughtful + considerate reviews.
during the review process the reviewers are anonymous.
(by signed peer review, does it mean not blind or single blind?)
@ckohtala I meant not blind. That's super interesting, though. Thanks for that link! I think just the act of publishing (even without a name attached to it) could perhaps help!
@tiana_athriel F1000 uses "open peer review," which in practice seems to be that the authors friends go on and gush
This also meant that a reviewer attacked me personally when I went public about a paper published there that lifted sentences and ideas from my blog without attribution
I assume the reaction came because they were "on the hook" for endorsing this paper publicly
There are tradeoffs