Alas, scale ruins everything. Nevertheless, I strongly believe “science is friendship”
Although it could raise the standard of the process itself (methodology, writing) to very high levels, it restricted innovative ideas or unpopular outcomes.
If it works better, it works better. The problem with the society method is it didn't work better than a decentralised scientific system.
I’m having trouble finding a downside besides vote buying or voting rings, now that which way one has voted is now attributable. Can you think of any risks under the new system?
I hope that making things transparent will help reduce the situation where big labs have an easier time getting their work into high impact journals through relationships with the editor.
This is desperately needed as AI could further deteriorate the quality of science if the publishing process is not made more strict. This represents a significant step forward in rigorous science. I hope other publications follow suit.
I highly doubt it's a meaningful factor in public trust.
I suspect the number is low. If that's the case, they're unlikely to be more convinced by the presence of published peer review, either.
Your posting doesn't give me the impression that you're very familiar with "science".
Maybe structural engineers feel safer after their Master's when they traverse a bridge, but I bet that's more the exception than the rule.
[1] https://www.theregister.com/2020/04/02/boeing_787_power_cycl...
That is very unhelpful, to say the least. The amount of noise has increased, but it does not mean that the scientists who know their subject disappeared. They are still around and not any less bright than their predecessors were 30 years ago.
Why should anyone trust science? Skepticism should always be the default position. Putting it on a pedestal to be worshipped is what led us all into this mess. If science needs trust to work, then whatever it is doing is something I'd like to see fail.
When people talk about those who distrust science, they aren't referring to the carefully sceptical. They're talking about people who come to a conclusion and then reject any evidence against it.
I'd actually be curious what fraction of his supporters share his views on vaccines. It strikes me as more likely that they like one of his random, idiosyncratic views and are willing to excuse the anti-vaccine nonsense to get those policy outcomes.
That’s a red herring. No scientist actually says that. What we say is that on some subjects the evidence is overwhelming and to overcome the current understanding you need compelling evidence and theories, not screeches about bias and liberal elites.
No. We are not going to take seriously someone’s pet theory about a new perpetual motion machine, or cold fusion, or lack of global warming, or the ineffectiveness of vaccines, or anything that is contrary to massive amounts of accumulated evidence.
There's that word again. No works are trustworthy. You shouldn't trust, instead verify. Be skeptical. I'm more concerned with which of their works can be verified.
>An ironic stance considering that it's the inverse of the blind worship
I'm not sure there's any difference worth mentioning between blind worship and cuddly warm trusting that you want to do. One may seem a little less zealous, but it's the same pig just with extra lipstick.
By "trustworthy" I mean nothing more than the same presumption of innocence absent evidence to the contrary that you extend at the grocery store when you assume that the food you're purchasing isn't poisoned or rotten or etc. That doesn't mean the published results are reproducible, the measurements without (unintentional) error, or the conclusions logically sound. Merely that there isn't an active attempt being made to deceive or otherwise knowingly mislead the reader to the advantage of the author.
What I'm describing is an incredibly low bar with broad applicability that society generally requires in order to function.
Skepticism is not anti-scientific. Hell, distrusting results and theories is not anti-scientific. Distrusting the scientific method is. There is a difference.
> Putting it on a pedestal to be worshipped is what led us all into this mess.
Scientists did not ask for this. Amongst all high profile politicians, those who whine about science becoming political are those who made it so, by taking contrarian positions to rile up their voter base. Most people who make this point are not being honest. If you are, you should make specific arguments rather than rehash propaganda.
One might think that to be the case, but science is in a hurry to save our planet, and it has no patience for your skepticism. It may already be too late.
>Distrusting the scientific method is.
Maybe I don't trust it either. Maybe because I'm distrustful, the scientific method should prove, over and over again, that it's worth a damn. And maybe when it's finally proven that, maybe I go on distrusting it... and make it prove it some more. If one truly though the scientific method worthy of any respect, such a person might say "that's ok, because it will go on re-proving what has already been proven". But so little do I ever hear of that sentiment. Like in this very thread.
>Scientists did not ask for this.
Probably not. I'd be skeptical of the theory that they asked for it. But it doesn't seem altogether implausible that they might have enjoyed it once it happened even though they didn't ask. And liking it, they started behaving in ways that encouraged it just the same. And those scientists who enjoyed it the most elbowed their way to the front to egg it on even more. And so that's where we are now. I mean, I'm a little skeptical that it happened that way, and I'd welcome evidence that disproves it.
>Most people who make this point are not being honest.
It's really sort of hilarious how it's only my side that can ever be honest. Not the other guys'. And this is true no matter who "me" is. None of your opponents, for instance, can ever be genuine... they're always trying to cheat. And if your side tries to cheat, well, it's with the best of intentions because the stakes are so high. Not that you would cheat, only the other side ever does that.
>f you are, you should make specific arguments rather than rehash propaganda.
If we can't start here, then there can be no real conversation. Basically you're just asking me to surrender and admit that I'm the bad guy trying to cheat and swindle, everything I say is a lie, and you're the innocent victims. Haha.
As a matter of fact, it is already too late for some of the consequences. The permafrost is not going to re-freeze, for example, and wildfires are already significantly on the rise.
The thing is, I don’t believe this because science, I know this for a fact because it was proven. It’s like nihilistic edgelords giving cynicism a bad name. Skepticism is not the rejection of facts because they are inconvenient. It is not stubbornness in the face of evidence, either.
> Maybe I don't trust it either. Maybe because I'm distrustful, the scientific method should prove, over and over again, that it's worth a damn.
Right, but then to prove itself it needs to be evaluated and assessed. You need a kind of method to do so, including a standard of proof and the basics of reasoning and logic. This framework is itself the scientific method. If you start from the axiom that it does not work, then the only conclusion is that it does not work, because how would you prove otherwise?
> If one truly though the scientific method worthy of any respect, such a person might say "that's ok, because it will go on re-proving what has already been proven".
Well, yeah. That’s what we do. Though the standards shift over time in the face of overwhelming evidence. We can not start from first principles every time, it just does not work and there are theories we can rely on. But when holes are found in these theories we try to figure out what the problem is and how to build better ones. The scientific method itself has been refined over centuries and the standards now are far from what Newton used. A common example is the evolution of what is considered to be a proof in Physics.
> But it doesn't seem altogether implausible that they might have enjoyed it once it happened even though they didn't ask.
I am sure some of them do, and not necessarily the best ones. But most of us are not attracted by public engagement and just want to do our science in our lab. We did not ask to become pawns in sectarian struggles. We rang the alarm because, honestly, when the house is on fire you have to do something and not doing it would mean some degree of responsibility in the outcome.
> It's really sort of hilarious how it's only my side that can ever be honest.
I am not following. I am not talking about sides. Regrettable attitudes towards science are not confined in one political or social tribe and there are examples in history of both right- and left-leaning ideologies becoming anti-science. But yeah, making a topic political in order to then claim that it is political is blatantly dishonest. It’s a cheap rhetorical trick with lasting consequences. What is political is how you address the situation, not whether reality is real.
> If we can't start here, then there can be no real conversation. Basically you're just asking me to surrender and admit that I'm the bad guy trying to cheat and swindle, everything I say is a lie, and you're the innocent victims. Haha.
You are reading way too much. First, I am not attacking you. Then, a good starting point is facts we can observe and things we can infer from them instead of the a priori position that reality is not real and the other side is just making stuff up.
Wildfires at at relatively low levels, historically. US burn acreage is about 10M acres per year at the moment, same as it was in 1955. During the hot dustbowl years burn acreage was as high as 50M/year.
https://web.archive.org/web/20210129125036/https://www.nifc....
You're unaware of this because "scientists" like to truncate wildfire graphs at around 1983, knowing that insufficiently skeptical people won't check if there's data available from earlier years. The choice of 1983 isn't arbitrary, it's when the downward trend in the graph goes into mild reverse as people started to realize that suppressing wildfires isn't always the best policy. There follows a small, gentle rise to a level still far below what it was historically (and arguably, what it should be).
Now, obviously, they don't say that. They say oh dear, in 1983 we changed our process and - incredibly - that they have no idea where data came from before that, so please ignore all previously collected data. The numbers just, like, magically appeared in their spreadsheets. This sort of trick is common, and it correctly leads to distrust as obviously, if they genuinely believe data before 1983 is useless, they can't make statements about the effect of climate change on wildfires one way or another. But that would also be an enormous scandal given all the money spent collecting that data.
> I don’t believe this because science, I know this for a fact because it was proven.
You haven't actually gone out and monitored wildfires with your own eyes, that's impossible. When you say it's a "fact" and "proven", what you mean is, people you trust from institutions you aren't skeptical of told you that they did go out and count wildfire acreage, and that it's a "fact" that wildfires are getting worse. Then you assumed they're completely trustworthy and would never do things like drop data points to force a trend, so it became a fact in your mind and you became unable to understand why anyone might disagree.
Now repeat this problem 1000x. That's where the fighting comes from, where the distrust comes from.
The world is bigger than the US. FFS...
https://ourworldindata.org/grapher/annual-burned-area-by-lan...
Wildfires have been decreasing globally for over 20 years.
Those that distrust authority as a whole and lean into conspiracies cannot be saved with this kind of thing.
But i think news and science are having similar perception issues recently.
Distrust for news growing among the average population (for some good reasons). People are loosing faith in the objectivity of established media organisations. Most people are exposed to science through these traditional news.
So adding back some sense of confidence and authority to scientific institutions is very valuable to non-academics. Even if they themselves would not read the papers or revews.
Ignoring the problem is not going to fix it, and in fact continuing to regard these people as beneath you is only going to accelerate the downfall of this system.
Of course it does, billion-dollar interests who have a vested interest in attacking it have a well-funded propaganda arm, and as we've been discovering over the past century - angry, loud, and frequently repeated bullshit with a profit motive floods any signal out of the room.
What's really sad is people who have legitimate concerns or desires for improvement hitching their horse-cart to the former.
No amount of peer review or replication is going to convince someone whose fortune is built on peddling snake oil.
> Ignoring the problem is not going to fix it,
Here's a solution. Hold the people making these attacks to the same level of rigor as what they are attacking. Stop giving proven liars a loudspeaker. The day the press will do that is the day some meaningful progress to fix that damage may be made.
Until you do that, we're going to continue getting shit outcomes.
Source: I've published such skepticism in the past and met people who read my articles, including politicians.
Nonetheless, you're right that simply publishing peer reviews won't help improve trust. The situation is bad enough that there's no One Weird Trick that can dig academia out of its hole. Some of the most intellectually fraudulent papers I've written about in the past did have transparent peer review, and all it showed was that peer reviewers were often aware of the critical problems found inside and waved it through anyway. Or that their feedback was ignored. Or that they agreed with obviously bogus practices. Or that the parts of the paper that revealed the problems weren't reviewed (eg. appendices, github repositories). After all, nobody cares about papers that got rejected by the system, the distrust is driven by the papers that weren't, so almost by definition such papers either were badly reviewed or the review wasn't used.
The thing is, peer review is not a stamp of quality, and never was. It is just the basic level of due diligence. The referees cannot reproduce the results most of the time for a lot of very good reasons. They are here as a sanity check, to ensure that the work avoided common pitfalls and actually makes sense.
What most people do not understand is that articles are not good because they are peer-reviewed; it’s the lack of peer review that is a red flag. Amongst reviewed articles, a lot of them will turn out to be wrong or flawed in ways that are impossible for the reviewers to find out.
Imagine saying "no" to a researcher with a big social media profile. Imagine 4chan coming at you with style-detection and deanonymization tools simply because their favorite racist or antivaxer got their nonsense rejected and sent their followers after you. And this is not just me feeling this way - quoting myself from a previous comment, and according to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents" and "support for public review inversely correlated with reviewing experience".
A measure that women ~~and inexperienced researchers~~[2] do not support is a measure that favors only those who are already part of the club.
[1] Original here (currently offline): http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019..., summary here: https://www.aclweb.org/adminwiki/images/f/f5/ACL_Reviewing_S...
[2] This part has been correctly pointed out as being wrong.
"The identity of the reviewers will remain anonymous, unless they choose otherwise — as happens now."
(Also "support for public review [being] inversely correlated with reviewing experience" means inexperienced reviewers are more likely to support it. Not less.)
As for the anonymous part, that's why I wrote "with style-detection and deanonymization tools". If the Internet could find Shia Labeouf's flag in a day [1], could they find a reviewer based on their writing?
[1] https://www.dailydot.com/unclick/4chan-shia-labeouf-secret-l...
You don’t have to actively demonetize your reviews, you just have to prime your audience ahead of time that your ~~election~~ publication was stolen and they’ll do the rest.
Though, we do eventually need to have a conversation about deanonymization of online accounts. That's not a thing we want to be done so easily
Or are you just for creating classes of people that just can't be critiqued in any circumstance?
This kind of sounds like, 'Wont anyone think of the grifters?!'
Not all academia is Elsevier.
[1] This policy has been altered recently, though, and now submitting a paper comes with reviewing duties.
The ACL leadership and senior members of the field are very much aware of this and are trying their best (ARR being an attempt to improve the situation, but I am unsure how much better it really is compared to the old system of conference reviewing now that we are a few years in). But there appear to be no easy fixes for a complicated, distributed system such as peer review. Every discussion I have with said leadership and other senior members always ends with us agreeing on the problems and likewise agreeing that despite considerable mental effort we are failing to come up with solutions.
Returning to the main topic. Nature is worthy of praise for making their peer review transparent and I say this as a massive Nature critic. It is a move I loved seeing from NeurIPS (then NIPS) and ICLR over a decade ago, as it helps younger researchers see what good (and bad) communication looks like and that even papers they know now are greatly appreciated received a fair amount of criticism (sometimes unwarranted). I have argued for ACL to introduce the same thing for nearly a decade at this point, but we still do not and I have never heard a solid argument as to why not (best argument was the technological effort, but OpenReview, with all its flaws, makes this even easier than with Softconf; not that it would have been that hard with Softconf either).
I would suggest to Nature that this phrase hints at much larger a problem than showing authors arguing with Reviewer 2.
To little, to late? Maybe.
Surely very little, surprisingly late.
There are so many great reasons to distrust "The Science"... perhaps the greatest is that without powerful and persistent skepticism, science simply isn't science. Questioning and skepticism is profoundly fundamental to the scientific process. "Trust the science" or worse yet "Believe the science" are statements that are about as destructive to the foundation of science as one can get.
The whole point of science was that everyone has access to the true nature of the universe which we can discover through theorizing and experimentation. Empirical experience is the great equalizer that puts the Arch Duke of Where-and-What or the Highest of High Priests the on the same footing as, say, Joseph Preistly or Benjamin Franklyn. Questioning and skepticism must be accepted from all quarters. To say that only the most select, distinguished and credentialed can be admitted to the discussion smacks of priesthood and aristocracy.
The argument that we need to re-persuade the people to trust in science is missing the point by such a wide margin as to be a symptom of the problem.
It is more or less the same as fretting that people aren't going to church so much any more.
The science should not be trusted. The science should be tested. Those who say "Trust the science" have completely lost the thread.
jostmey•7mo ago
jxjnskkzxxhx•7mo ago
yupitsme123•7mo ago
Research gets cited constantly in public debates and is used for policy decisions, so the public should be able to quickly separate the good from the bad, the "maybe this is true" from the "this is empirically proven."
The public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus.
thomasfedb•7mo ago
Replication and meta-analysis are an important part of this.
Most scientists are in fact very conservative with how they claim their results - less so university PR departments and “study shows” clickbaiters.
jxjnskkzxxhx•7mo ago
blix•7mo ago
Instead I got PIs happy to say that weak evidence "proved" their theory and to try suppress evidence that negatively impacted "fundablity". The most successful scientists I worked with were the ones who always talked like a PR puff piece.
jxjnskkzxxhx•7mo ago
blix•7mo ago
jxjnskkzxxhx•7mo ago
I did theoretical physics (no money) and my experience totally matches what the other person described.
JumpCrisscross•7mo ago
The public has lost trust in science because 10 to 30% of it is scientifically illiterate [1]. (Tens of millions of American adults are literally illiterate [2].)
That's what lets activists and politicians cherrypick bad science that supports their position or cast a scientific consensus as unquestionable.
[1] https://carsey.unh.edu/publication/conspiracy-vs-science-sur...
[2] https://nces.ed.gov/pubs2019/2019179/index.asp
fc417fc802•7mo ago
Obviously reproducing results as part of peer review is not a workable (or even coherent) solution. I don't pretend to have any idea what a solution might be. The obvious issue is that academic publications were never intended as political tools and should not be made into that.
On several occasions I've had interactions with laymen where I found myself thinking "if only you hadn't had access to pubmed and way too much motivation we'd both be better off right now" yet I firmly believe that free and open access to information is a huge net benefit to society on the whole.
kergonath•7mo ago
It is not, and it cannot be. It is unrealistic to expect a referee working in their free time to confirm studies that often cost millions of dollars. This is a fundamental misunderstanding of what peer review is and why it is useful in popular or heavy vulgarised science.
Politicians, journalists, and university press offices are guilty of this, and they are those abusing peer review to give some studies more weight than they deserve.
aDyslecticCrow•7mo ago
There is a massive incentive to publish. Inflate the value, inflate the results, and stretch out projects to multiple smaller papers, fake results to make it seem important. This is lazy and fast, and can be caught by a stricter review and scrutiny.
Papers that are properly done all the way through, but with faked data meant to push an agenda, can be disproven by counter research.
fc417fc802•7mo ago
Genuine mistakes, logical errors, and other oversights are even more common than that. For all the issues it has, peer review is quite good at catching the things that it's intended to catch.
yummypaint•7mo ago
Maybe people could learn about what peer review is before posting their strong feelings about it? The purpose certainly isn't to replicate people's experiments, that happens after publication and not by referees. One of a reviewer's duties is to look at whether the study could be replicated given the included information. That is a very different thing.
Also, just because something has made it past peer review also doesn't mean it isn't controversial in the field.
jostmey•7mo ago
This is my academic profile https://scholar.google.com/citations?user=bxn78bkAAAAJ&hl=en
mike_hearn•7mo ago
What he is saying is that peer review is treated by academia as a close to gold standard, when it's really more like a bronze standard. It's a bit like finding a software company that exclusively uses volunteer post-commit code review with no unit tests or static typing, and in which the only testing process is to push to production and see if anyone inside the company complains.
It's not useless as a concept, and it's generally better to have a paper that's reviewed than one that isn't unless the field has been captured by ideologues. But there are so many common problems it can't fix, not even in principle.
kergonath•7mo ago
This is a misunderstanding of the role of peer review. The point is not to prove that a paper is correct, the point is to ensure a minimum level of quality. You are entirely right that most reviewers cannot hope to reproduce the results presented, and very often for very good reasons. If I write 3 proposals over the course of a year to get some beam line on a neutron source, it is completely unrealistic to expect a referee to have the same level of commitment.
I think this hints at a more profound problem, which is that a lot of studies are not replicated. This is where the robustness of a scientific result comes from: anybody can make the same observation and reach the same conclusions under the same conditions. This is the real test, not whether you convinced 3 or 5 referees.
The real value of an article is not in whether it was peer reviewed (though the absence of peer review is a red flag). Instead, it is in whether different people confirmed its main results over the years that follow publication.
mike_hearn•7mo ago
https://blog.plan99.net/replication-studies-cant-fix-science...
Merely replicating a study doesn't tell you anything about whether the claims are correct, although failure to replicate can be a signal that the claims probably aren't correct.