There have been several instances where commentators at this blog have criticised others for publishing their information on websites rather in peer reviewed journals. The inference being that if its not in a peer reviewed journal, the idea has little merit.
Interestingly, there’s a new medical journal to be published by Elsevier called ‘Medical Hypotheses’ and it plans to take a deliberately different approach to peer review:
“Most contemporary practice tends to discriminate against radical ideas that conflict with current theory and practice. Medical Hypotheses will publish radical ideas, so long as they are coherent and clearly expressed. Furthermore, traditional peer review can oblige authors to distort their true views to satisfy referees, and so diminish authorial responsibility and accountability. In Medical Hypotheses, the authors’ responsibility for the integrity, precision and accuracy of their work is paramount. The editor sees his role as a ‘chooser’, not a ‘changer’: choosing to publish what are judged to be the best papers from those submitted.
Papers in Medical Hypotheses take a standard scientific form in terms of style, structure and referencing. The journal therefore constitutes a bridge between cutting-edge theory and the mainstream of medical and scientific communication, which ideas must eventually enter if they are to be critiqued and tested against observations.”
What a great idea! And doesn’t the new journal neatly articulate the problems with peer review for those working outside of established paradigms.
The quote was sent to me with a link to a blog piece by Andrew Leigh in which he suggests the concept has application to economics.
Jennifer Marohasy BSc PhD is a critical thinker with expertise in the scientific method.
Interesting but welcome concept!
It seems that there may be a forum for ideas that can be discussed without the usual shutdown cries of;
* ” non-peer reviewed ”
* ” outside of the concensus view ”
* ” the debate is over ”
funny thing though – if the majority position is always to be pre-eminent then why the push from the usual suspects on gay marriage for example?
Because the concensus is only to be respected when it lines up with our position?
Jim, rather a bizarre segue, from peer review to gay marriage! But it does very nicely focus attention on the problem that agenda driven people have with the scientific method and that is that in science, unlike in politics or moral issues, all opinions are not equal.
This is the key concept that bloviators on all sides of the politically charged scientific issues refuse to grasp. This is why it is quite legitimate to give greater weight to the findings of peer reviewed journals rather than venues where absolutely anyone can publish absolutely anything. Of course one of the foundations of this point of view is a belief that there is a reality out there that is idependent of our understanding of it. If you don’t believe that, well then I suppose any opinion is just as good as any other.
As for the topic of the post (ignoring the strawman nature of its setup) I find it a very interesting idea, and one that would be great to extend into the theoretical sciences too.
I would not, however, recommend policy decisions based on findings that can only be published in such a venue.
Sorry Coby – it was the best example I could have come up with ( and I know how touchy some here are about ” off topic” stuff).
My point was that if concensus is the final determinant then it’s not consistently applied – it wasn’t meant to imply a position on gay marriage!
Jen,
I think it is an excellent idea.
What I have learned from a variety of blogs and topics is that peer review has a lot wrong with it. It did not, for example, catch any of the recent examples of blatant scientific fraud.
Clearly the process of peer review is for the benefit of the publisher, not the author,or the subject. But has the publishers image/profits diminished?. Probably not.
Was there damage done to the credibilty of the science?. Almost certainly.
Are the reviewers,who seemingly signed off on the science been penalised?. No. They are not known, so how can they be.
It would seem to me that when reviews are undertaken by secret/anonymous group, who are not accountable to anyone,then how can peer review be anything but a sham.
“Interestingly, there’s a new medical journal to be published by Elsevier called ‘Medical Hypotheses’ and it plans to take a deliberately different approach to peer review:”
So who volunteers to have a medical treatment from the above journal?????
Hmmm, medicine is very much evidence based so I doubt if economic theorists would get a leg in – a peer review by economists would be like the weekend literary supplement, initially interesting but ultimately composted!
Don’t get too carried away. So peer review has been shown to be flawed like juries. But it’s the best we’ve got.
However it doesn’t stop there. Research has to be repeated by others and confirmed. Ideas get overturned. The literature changes and history progresses. Courts indeed have processes of appeal.
All this business about “shock horror” – peer review didn’t work is overly dramatic and really being too precious. Penalising reviewers – come on – who would do the job – most reviewers do a very good job (the bastards!) but to err is human. Do we penalise jurors? Peer review is not the last word.
If peer review is a sham – how has most of technological advances in the last 100 years actually occurred. For example – how come we have genetic engineering working. Pretty hard to make it up !
But posting reams of untested b/s and rampant speculation on the web isn’t science by a long shot. Good fun maybe – but caveat emptor to the max.
P.S. Peer review does catch science fraud – the issue is that if it’s a crap paper it probably won’t get published. So you don’t hear about it. Of course some areas of science are not 100% certain and so differences in opinion or different schools of thought can be held simultaneously.
Rog,
I’ve actually read some stuff in medical journals about the extent to which western medicine is increasingly lacking an evidential basis. The medical researchers went on about the importance of ‘evidence based’ medicine. I’ve no links … read hard copies in the UQ biological science library some years ago.
Luke,
In one of the recent cases, the papers that were peer reviewed contained material and diagrams that had been shown in a previous paper, with different headings.So much for peer review being even half competent.
It is a poor argument to try and say that look at what has been achieved (in spite of peer reveiw), and not think, about how one could do it better.
Cant see what the problem is with reviewers being known.It might improve the standard somewhat if they were.
The peer review is definately not designed to detect fraud or plagarism and it is not really clear to me that it should be. The frauds are caught soon enough by the competitive nature of scientific research. It becomes too big a task if you ask reviewers to be auditors as well, the system relies alot on trust. And as I said, when this trust is violated it does not escape notice for long.
Anonymity of reviewers is important for the same reason one votes privately.
Rubbish,
In the two most recent cases the incompetence of th peer review process cost millions to both private investors and public monies,and conned people for years.Why? because people relied upon the so called “integrity” of the review process.
They may have eventually been caught but not until others had made substantial losses,because they had relied upon the work being truthful.
The peer reviewers/publishers should be sued for abject incompetence.
It is also a trite nonsense and a little wierd in logic to equate anonymity of PR, to that of voting for a government.
Heavens – some poor little investor bet the bank on a single paper – so the entire the entire peer review process is corrupt – don’t flick the light switch – it’s built on a lie – don’t drive – it’s bullshit – there is only one answer to stem this insidious tide of wrong doing – yes it’s .. it’s the Spanish inquisition.
Nobody expects the Spanish Inquisition! Our chief weapon is surprise..surprise and fear..fear and surprise.. Our two weapons are fear and surprise..and ruthless efficiency.. Our three weapons are fear, surprise, and ruthless efficiency
What driveling nonesense.
Where as everyone else has to be accountable for the advice they give, the poor academic darlings mustnt be held up to public scrutiny for fear of being found out when they have been quite incompetent.
Malcolm, you remind me of some whiny socialist complaining that someone should protect him from stubbing his own toe. Try taking a little responsibility for a change. Market forces are there precisely to eliminate fools who bet the bank on poorly vetted information. The scientist lied and betrayed an important trust. And you want to blame society, or maybe the scientist’s parents who spanked him too hard. Grow up! Blame the criminal, not the peer reviewers who are not cops nor are meant to be.
Coby,
It is not only the scientist that lied, but the reviewers did as well. Either that or they were incompetent.
What is wrong with the reviewers being known.
The system is at fault as much as the fraudulent scientist. Most intelligent people would say correct the system as well, as occurs in most other fronts of societal activity.Thats how we make progress.
Before “fixing” a system you need to demonstrate a. it is broken and b. you have a better way. A few anecdotes about a bad outcome or two is not evidence of a systemic problem.
As for a better way, I would just remind you that everything in life is a trade-off and if you want to put a greater burden of verification on the review process you will lose something, it is inevitable. I will leave it to you to think of what the cost of your proposed changes are, if you haven’t thought about them then you should have no confidence in your own opinion.
If you suspect collusion on the part of the reviewers, I don’t think you understand what the review process is. The idea is to have a peer in the field assess if the paper is logical, interesting and based on solid references. The reviewer is not there to check the arithmetic, validate the data or rerun the experiment to see if their results are the same, that is the job of subsequent reasearch. It is not required to be criminal or incompetent to let something through that is a plausible but fabricated result.
The downside to not having anonimity is fear of professional reprisals, fear of being associated with something you may respect but disagree with, fear of being seen as harming a friend’s career, fear of pressure or intimidation. There are undoubtably upsides, I did claim above everything is a trade-off and mean it, but these are pretty significant negatives.
Boo hoo – Malcolm – what a generic whinge – get over it.
If the system is so riddled with corruption how pray tell has science progressed at all. None of our technology we currently enjoy would work.
The Korean scientist who made up the genetic engineering stuff is in disgrace. Same as Foundation 41.
If you want to get so ultra severe on any errors (see mistakes) you won’t find anyone wanting to do the job of reviewer. Certainly our children are getting the message that you don’t do science – too hard, no money and for nerds. Now you want to jail the practitioners as well for any error. Go fly a kite.
And it’s bloody obvious why reviewers (sometimes) are not disclosed. Some reviews can be quite harsh – there are plenty of opportunities for bias/payback/lack of cooperation on other issues down the track.
Imagine if I was a reviewer on your paper that you’d spent 3 years working up. I red ink it and give it a severe ragging because it deserved it. Actually I suggest your entire hypothesis is crap and your methodology fundamentally flawed.
I’m sure you’ll be very happy to be totally cooperative to me next time we meet. You’ll rush up and shake my hand. You’ll only see my comments as pure logic and non-emotional. You won’t be upset at all. You’ll be grateful (pigs bum you will – you’ll be calling me a pig-ignorant philistine that does not appreciate your genius. You’d just love a chance to get back at me).
In any case – what normally happens is that papers will be reviewed by your closer colleagues and maybe internal panels before they even get to go out to journals. It all depends on the agency and its quality controls.
Coby,
I think your list of fears is pathetic. Anyone who is professional and competent and on top of their subject should hold no fears about providing an opinion on a colleagues work, and its suitabilty for publication.The legal profession has been doing this since Adam was a boy.
I notice that the UK Committee on Publication Ethics found in 2001 that there were 106 cases os misconduct within the system, out of 137 cases examined.
That is addition to claims of bias, a propensity to only publish positive material,settling of old scores,plagiarism and disrupting rival research.
I also understand that a German funding agency found a suspicion of fraud in 65 out 347 papers including 27 instances of falsified material
Hardly a trivial problem, but thats what happens when things like this are done in secret.
Even the UK Govt acknowlegdes that a more open system would reduce the level of abuse.
Malcolm
I’m sorry we’re going to reject publication of your last comment. You have provided no references, not examined the counter-position and have made selective use of statistics. Further you have made an insufficient introduction to demonstrate cause, a vague wandering case, and no conclusion. You also need to remove the emotional aspects to your argument. Use of unsubstantiated biblical figures such as Adam are unacceptable, unless you can supply references as to Adam being a real person. We find the use of “Adam” also sexist, but any use of “and/or Eve” would also not be suitable.
You also need to define the terms “fraud” and “falsified”. The other issue is that the work bears striking resemblance to other such material presented in public fora i.e. “blogs” and so the panel are also concerned as to the authenticity of your material. In fact it seems to appeal to a contrarian consensus and not as such individual analytical thought.
The panel therefore rejects your submission to have your treatise on “Failure of peer review” published. On minor points we also noted several typographic errors and a general lack of punctuation. You also need to present in the third person.
The panel hopes you will accept these comments in a positive and constructive light and although it will take some signficant work – a total re-evaluation of the data and rewrite of your position is recommended.
Nice try Luke. To bad it is irrelevant. But then that doesnt surprise me one bit.
I am not making any other case than the need for secrecy. The references I have used are readily available to anyone. Just do a Google or Wiki.
Go on, you might surprise your self as to the extent of the problem and what can be done to improve it.
BTW if someone wrote that about a paper I was trying to have published, and he had put his name to it I would take on the chin,and do something about it. But in the context of blog it is just point scoring crap.
Judging by the intellectual content and logic of the repartee being peddled herein, there arnt too many scientists involved, or are there?
And how many journal papers have you written Malcolm? Are you speaking from experience perchance.
I suspect if you had 3 years invested in a piece of research you would be miffed by a red ink review or your blood is made of silicon.
Look out Malcolm, Jennifer has been known to gently chide those on her blog who seem incapable of a sense of humour.
You do realise Luke was being funny, and funny is always relevant. Gee down old son!
Back to the serious: one potential problem that I can see with having the peer-reviewers’ identities always made public is the increased opportunity for the reviewer to be influenced, either for or against the findings of the paper. A fairly obvious problem don’t you think?
For example, a hypothetical peer-reviewer who is very smart and knowledgable but timid and nerdy could be hassled or intimidated inappropriately by a third party who approached them with an abrasive and emotional arguing style – such as your own style in this thread – if their identity were public.
A poor and unscrupulous peer reviewer might be persuaded to accept money to find a particular result etc.
Having anonymous reviewers goes a long way to avoiding this, and having multiple reviewers (as is generally the case) makes the possibility of shenanigans even more tricky.
Malcolm, I like that you brought up lawyers as setting an example for an alternative approach. No politics or corruption there.
As for your “the studies are out there just google it” you may not have heard of the recent analysis of alledged citations made by people on the internet with no link or specifics provided. It turns out that in 96% of all cases the reference is fabricated and the professed unwillingness to provide a link is in fact disguising an inability.
You can google for it.
Hi Steve,
Yes I did realise that Lukes epistle was an attempt to be funny/clever, and to also make a point about what can be said about PR, as applied to my blog post.
As for anonymous PR’ers being more ethical than the openly declared I fail to see how that could be that case. I would of thought that an open system would make it very difficult for shenanigans of that type.
After all, the reviewers name should be identified with the paper, and made known. That stops plagiarism and theft and if the pp is a dud then he wears it (to a limited extent) as well.
Luke, I would be mightily pissed off if some anonymous person red inked a piece of my work and I had no opportunity to discuss or negotiate, with the reviewers and the publisher.
That would be a denial of natural justice.
But I would be even more pissed off if some anonymous reviewer made use of my work to advance his own,and to do this he also scuttled my paper. Dont tell me this doesnt happen because according to the evidence it does, and is more common than anyone will admit.
Why would anyone be a scientist when the system is so corruptible and unfair. Not me.
I have enough trouble with IP protection in commerce and industry.
Yes, I have had papers published but they were not subjected to the classical PR process.
I never expected that a simple observation would create so much angst and opposition when all the evidence is that an open system would be fairer and more efficient and less corruptible.
There are oodles of parallels in other domains where openess works, and given that the publisher has the final say just what is the issue here.
According to the UK Parliamentary Office of Science and Technology, an open system would:
1.Reduce the abuses of the system.
2.Render referees more accountable for their comments.
3.Increases the credit given to referees.
They are pretty powerful reasons for change.
Some referees do make their identities known.
Some referees decline to review certain material for conflict of interest or lack of competence in the specific area.
Most referees do make signficant comments on areas of rejection.
And indeed the submitting authors can also cite reviewers who they think may have a conflict of interest or bias. Of course this also be an attempt to avoid scrutiny.
Malcolm – do you have an idea of the overall degree of corruption. And do you have a proposal as to what standards Science and Nature for example should employ. There is also the issue of data and computer program availability as well. Do we expect reviewers to actually repeat methods. In areas were supercomputer size calculations are used it would be nigh impossible.
(I apologise for taking the piss before – couldn’t help myself.)
few people know that the works of Einstein and Planck in the “Annalen der Physik” were not peer reviewed.
In medical peer review there is no anonymity and conflicts of interest are always noted otherwise you would have no credibility.
Or less I should say, the really significant papers are conducted by double or triple blind study. Otherwise you are just expressing an opinion.
Hans Erren,
..and also the famous Watson Crick paper of 1953 was not peer reviwed either.
So the odd genius is very very smart!
According to the book “Science: A History” by John Gribbin, Watson and Crick owe much of their success to their being able to view the excellent x-ray crystallography photographic results of one Rosalind Franklin. They were able to view these results without the knowledge of Franklin, without her permission, and downplayed the significance of this in their published work.
Gribbin goes as far to suggest that this was ‘a serious breach of etiquette’, that Franklin had been working on a similar paper to the Watson Crick one, and that Watson and Crick to some degree snatched the credit for the discovery of the structure of DNA from under Franklin’s nose.
In any case, Gribbin is quick to point out in this example, and in many other examples in the book that science proceeds by evolution not revolution, that new theories are generally the result of a build up of knowledge from contributions from many people despite the attachment of names and people to famous ideas. That is, scientific consensus is the approval process by which models and hypotheses become theories, rather than human knowledge abruptly changing overnight because an individual paper is published, regardless of whether it was peer reviewed or not.
I wasn’t aware that the Watson & Crick paper was not peer reviewed, Perhaps the lack of peer review meant that there was no reviewer to question closely what resources were used to find the result. Maybe appropriate recognition may have then been given to Frankin, Pauling and a bunch of others that all contributed greatly to the discovery of the structure of DNA.
Well if the system was more open from the beginning then Rosalind Franklin et al, may not have felt as though the were dudded.
Or alternatively if Franklin and Pauling had been the secret reviewers then …. fill in your own scenario.
Single author papers are becomming rarer these days so this should reduce the risk of serious corruption as does multiple reviewers. The question these days is do scientists have enough time to devote to quality reviewing? Good and bad papers are usually fairly straight forward decisions, but in between papers can require quite a many days of research and reading, a luxury few can afford these days. It seems now that if you want to publish in a certain journal the editors have you in their sights as a reviewer.