As a family physician, I did not submit papers and seldom read primary research papers, being overwhelmed more by application type studies, but the issue still trickled down to the level of practice. When a new drug came to market, some doctors were early adopters, some gave it a little while before prescribing, and some were late adopters, waiting until results were well established before using. I tended to be in the mid to late adopter group, as there was a little distrust of the process, and I thought it good to wait to see if adverse side effects showed up that were not found in the initial studies. That in turn might be due to bias and pressure on the part of the companies funding the research.
That too has changed a bit over the years, particularly because of direct marketing by drug companies, leading consumers to ask for and seek out drugs that are new to the market. We are really seeing that with such things as the weight loss drugs originally developed for diabetes, where clinics pop up and pharmacies are compounding and selling it for weight loss, with little track record as to long term effects or efficacy. Economics seem to rule in our society, with little regard to reason and ethics.
I like that approach. It does require more time from the reviewers which may be a sticking point.
Agreed. The results are mixed (to put it mildly) when peer reviewed papers are filtered through the media. There are some good science journalists out there (e.g. Carl Zimmer), but they are sadly few and far between.
“Tell people there’s an invisible man in the sky who created the universe, and the vast majority will believe you. Tell them the paint is wet, and they have to touch it to be sure.”
― George Carlin
I agree that the review system has weaknesses because of the humans doing it, and the interesting publications (or the popular versions of the story) often try to claim too much importance for their finding because they want to promote their research. Although this is common, there are great differences depending on your branch of science and the journals where you send your manuscript.
Some journals only publish <15% of the manuscripts they receive. In these journals, there is an attempt to speed the editorial process by suggesting that editors reject any manuscript that does not seem to have good chances to become accepted (editorial rejection) and reviewers are asked to rate the content of the manuscript relative to other papers. If the reviewers do not rate the manuscript within top 10%, that is enough to warrant rejection, unless the editor has another opinion. To avoid rejection, the authors try to market their manuscript by claiming that the results are very significant and show spectacular novelties, even when that is not true. If the authors are skillful in how they use words, some busy reviewers may buy their claims and think that the manuscript is better than it is. Expert reviewers are busy and do not have much time to evaluate one manuscript, so they may easily miss hidden weaknesses in the manuscript.
If the manuscript includes interesting results that challenge previous consensus, the manuscript has good changes to become accepted. Even if the significant result would be just a statistical anomaly, the results attract attention and citations in papers that want to refute the claims. Good for the author and the journal, so who cares if it is true or not. The following papers then show whether there was some truth in the paper or not. Almost 5% of statistical analyses give a statistical confidence of p<0,05 just by chance, so a lucky researcher may claim to have found significant results, even if that would be just a chance result.
So yes, a single test or paper is not enough to tell what is true or not. There is a need to get cumulative evidence for a model to be trustworthy.
Edit: I used the word ‘model’ to avoid the pitfall of using ‘theory’ or a similar kind of word. Thanks, @Christy for the suggestion to use the word ‘model’, in another thread.
Oh, wow, the urgency is very true! Our system forbids all drug rep visits (thank goodness; though I don’t think they meant ill), so we have not had any since our practice joined the large group, about 7 years ago. We get requests for off label use of Ozempic, etc, all the time; and that is really a struggle, as we want to help people who want to lose weight. The Vioxx problem was a good lesson for me, when they withdrew the drug from the market after an increasing risk of heart attacks was attributed to it. That also helped me realize the heart attack risk of other meds, like ibuprofen, Celebrex, etc–something I should keep in mind more. As my dad, a surgeon, used to say, “every medicine is a poison with a desired side effect.” I could have responded, “Well, Dad, isn’t every scalpel a weapon with a desired side effect?”
The potential for thyroid cancer, pancreatitis, and the fact that the appetite comes right back when stopping the med are good arguments against these.
PS in “Evidence Based Medicine,” which is basically the attempt to put medicine through the scientific method instead of relying on hearsay, the review papers are considered better. There are sites, like Cochrane.org, which assess the level of evidence better. Some of the best conclusions include that “we need more studies and data.”
In my experience, journals suffer from lack of reviewers with expert knowledge. Sometimes, you need to explain facts that are cristal clear for the experts. Of course, the writing quality is essential and non-native speakers have a disadvantage. It’s almost impossible to write as balanced in a foreign language as in your own language. It’s not to complain but simply the case.
I can’t say I’ve ever been asked to rate a paper in terms of its percentile rank. I would just refuse – I don’t think that’s going to yield a meaningful answer.
Unless the study has a pre-registered detailed analysis plan, the probability of reporting a chance result as significant (at a 5% threshold) is usually a good deal higher than 5%; there are just too many choices one can make in analyzing data.
From the viewpoint of an editor, getting experienced reviewers became more and more difficult during the last 20+ years because experts are busy and reviewing a manuscript is not a merit that would be worth the time - many do it just as a tit-for-tat strategy or because someone they know asks and they are too polite to say no. A merited expert may get many requests to review within a week, so it is natural that they must learn to say no.
From the viewpoint of an author, non-native speakers have a disadvantage. Saying something so that it sounds convincing when in fact you do not have sufficient data to prove it is an art. You really have to be a skillful user of words in those cases. I have admired the skills of some professionals that came from Cambridge or Oxford. The presentations sounded really convincing but when you started to dig deeper, there were many logical gaps and weaknesses in the story. Those telling the story just had the skills to tell matters like they were proven facts, without lying at any point.
It also helps if others know you, for example many British scientists have different attitudes towards the work of those they know and those they do not know, especially if the person works in some other part of the world. A non-native English speaker working far has to have really strong evidence to convince the reviewers, a well known person that is skillful in the use of words may get a manuscript published in a top journal like Science without any statistical tests and the reader does not even notice that the claims have not been tested at all.
Academic world is not fully equal, some persons are more equal than others.
Hi Paul,
Your comments above make me think of listening to the “News” or almost anyone with a hidden agenda. For instance, there is often more “spin” than substance in a lot we are presented with too much “smoke and mirrors.” We all need to be careful to find truth.
It is true that the Scientific Method should be a strong tool to scientifically evaluate discoveries, methods, and principles through predictable and reproducible processes which are always open-ended toward being invalidated.
We all should be leaning cautious in matters of significant bearing, and gracious all the time we can.
It’s important to realise that “the News” isn’t science, and even popular science journalism often gets the science it’s reporting misleading if not wildly inaccurate.
That is one thing that, fortunately, I don’t have to worry about, given that the popular press (and, for that matter, broader-interest science publications) ignore invertebrate paleo.
