This article pointed to a few cases where people tried to do the thing, i.e. the pledge taken by individual researchers, and the requirements placed by certain funding channels, and those sound like a solid attempt to do the thing. This shows that people care and are somewhat willing to organise about it.
But the thing I don't understand is why this can't happen at the department level? If you're an influential figure at a top-5 department in your field, you're friends with your counterparts at the other 4. You see them in person every year. You all hate $journal. Why don't you club together and say "why don't we all have moratorium on publishing in $journal for our departments?"
No temptation for individual research groups to violate the pledge. No dependence on individual funding channels to influence the policy. Just, suddenly, $journal isn't the top publication in that field any more?
I'm sure there are lots of varied reasons why this is difficult but fundamentally it seems like the obvious approach?
> So the solution here is straightforward: every government grant should stipulate that the research it supports can’t be published in a for-profit journal. That’s it! If the public paid for it, it shouldn’t be paywalled.
The article then acknowledges this isn't a magic solution to all the problems discussed, but it's so simple and makes so much sense as a first step.
I'm no expert here and there are probably unintended consequences or other ways to game that system for profit, but even if so wouldn't that still be a better starting point?
Only difference is that the author is writing for a wide audience and his best angle to change the world is probably to influence the thinking of future policymakers. While I am just an annoying "why don't you just" guy, my "audience" is just the friends I happen to have in prestigious research groups.
Adam M also probably has lots of friends in prestigious research groups (IIUC although he complains a lot about academia he was quite successful within it, at least on its own terms). And the fact that he instead chooses to advocate government policy changes instead of what I'm proposing, is probably a good indication that he knows something I don't about the motivatioms of influential academics.
(Maybe you'd need an exception for fields where the centre of mass for funding is well outside of the US, though).
That's the problem, they don't hate these journals, they love them. Generally speaking they're old people who became influential by publishing in these journals. Their reputation and influence was built on a pile of Science and Nature papers. Their presentations all include prominent text indicating which figures came from luxury journals. If Science and Nature lose their prestige so do they (or at least that's what they think)
This was very apparent when eLife changed their publishing model. Their was a big outpouring of rage from older scientists who had published in eLife when it was a more standard "high impact" journal. Lots of "you're ruining your reputation and therefore mine".
I see: my friend has 10-15 years of experience in their field, they have enjoyed success and basically got the equivalent of a steady stream of promotions.
I map this onto my big tech/startup experience. I mentally model them as: they are "on top of the pile" of people that still do technical work. Everyone who still has the ability to boss them around, is a manager/institutional politician type figure who wouldn't interfere in such decisions as which journal to publish in.
But probably this mapping is wrong.
Also, I probably have a poor model of what agency and independence looks like in academia. In my big tech world, I have a pretty detailed model in my head of what things I can and can't influence. I don't have this model for academia which is gonna inevitably lead to a lot of "why don't you just".
Same thing happens to me when I moan about work to my friends. They say "I thought you were the tech lead, can't you just decree a change?" and I kinda mumble "er yeah but it doesn't really work like that". So here I'm probably doing that in reverse.
I think it’d be a big ask from someone whose role doesn’t typically cover that sort of decision.
It doesn't. Not even close.
Peer review doesn't even mean that it's free from errors, free from fraud, free from methodological mischief; it doesn't mean anything at this point. Yet we continue to act like it does.
Darwin's work wasn't peer reviewed. Nor Einstein's. It's something we cooked up in the mid 1900's to deal with the fallout from another mistake ("publish or perish") that meant people had to try to publish even if they had nothing to say.
Except it was…? This is absurdly ahistorical and the fact that you cross disciplines in trying to make an incorrect argument questions whether you are in science at all.
The structure of peer review in Darwin’s time was different, where experts wrote monographs and gave lectures at symposia that then led to letters among their peers. Which is what happens now, if you take a step back.
The volume of new work these days is incompatible with the older informal system, and is in some ways our new paradigm is superior as there is a formal period in which new works are reviewed.
What you're calling "peer review" is what I would call "discussed" or "debated" which it certainly was.
I dispute your claim that the new paradigm is superior.
Accepted. But now there is Arxiv and Biorxiv and even Medrxiv—so we're back to where things were, it seems.
No actual working scientist thinks this.
“Glitchc” has it right elsewhere in this thread: the motivating force behind journals is prominence and reputation, not truth.
Would scientists feel the same if the public was more educated about how bad journals and peer review are? Not so easy to disentangle IMO.
People who do need to work professionally with peer review, do understand what it actually does and its limitations.
You seem stuck somewhere in the middle, caring deeply about a system you don’t seem to fully understand.
You'd need to provide evidence or an argument for this. The media reports on things in part based on journal prestige, and likely when questioned, people will say they can trust such things because good scientists have looked at the work and say it is good. This would be an implicit belief that peer review is generally working well, even if they don't use the term "peer review".
> You seem stuck somewhere in the middle, caring deeply about a system you don’t seem to fully understand.
Extremely presumptuous, as I work in this system, and have provided plenty of evidence for my claims. You've provided only sneers.
You've provided evidence that peer-reviewed science often turns out to be incomplete, inaccurate, wrong, fraudulent etc. All true. But it is not the job of peer reviewers to assure completeness, accuracy, or freedom from fraud.
A peer reviewer reads a paper and make comments on it. That's it! They don't check primary data, they don't investigate methods, they don't interrogate scientists, they don't re-run experiments just to double check. They assist a journal's editors in editing--that's it.
The check on published scientific results is the scientific process itself, not the publishing process. Prominent results attract further investigation, which confirms or disproves the reality of the underlying phenomena. Again: that's not the job of peer review.
Do some people ascribe too much authority to peer review? Yes, for sure. IMO your comments in this thread are exacerbating that problem, not addressing it.
Um, what? I have done all these things in reviews, and know other academics that have done these things as well. More confusingly though, if you are saying most reviewers don't do these things (which I agree with), this would only strengthen my point?
I'll let readers decide if it is my comments that exacerbate the problem, or if, perhaps, it is apologism for journalistic peer review that might be causing bigger issues in the present day.
EDIT: I still want review from a community of scientific peers. I just don't want this review to be in the hands of a tiny number of gatekeepers entangled with journals that largely just slow things down.
I mean, right, yes, of course. Much of the downvotes are cognitive dissonance, obviously. I suppose I meant the question rhetorically.
The problem isn't access, it's citations. arXiV is not considered a credible citation source since anyone can publish anything. TPCs don't use it in their list of citations, neither do grant funding agencies or government institutions.
The current academic enterprise relies heavily on third-party gatekeeping. We rely on others to do the vetting for us. The first thing an academic does is check where a paper is published, before even reading it. It's a crutch.
Any gatekeeper will naturally tend towards charging for access over time: It's a captive market, the economics demands it. Unless we eliminate that dependency, we cannot change the system.
I’ve never seen the slightest relationship between the charge to read a paper and the quality of review.
Yelp, TripAdvisor, wire cutter, hell even Google results themselves.
Once you start poisoning that well, it's difficult if not impossible to claw it back.
But who watches the watchers? I guess review fraud will need to be considered as well.
IMO, academics that do this are not very competent, because we have plenty of research suggesting that higher-profile journals are in fact less trustworthy in many ways, or that there is no correlation at all between reputation and quality (see my other post here in this thread).
Yes, some trash journals publish all trash, but, beyond that, competent researchers scan the abstract, look at sample sizes and basic stats, and if those check out, you skip to the methods and look for red flags there. Also, most early publications will be on an arXiv-like place anyway so you can't look to reputation yet.
Likewise, serious analytic reviews like meta-analyses don't factor in e.g. impact factor or paper citations, since that would be nonsense. They focus on methodology and stats.
I really think we ought to shame academics that are filtering papers based on journal alone, it is almost always the wrong way to make a quick judgement.
Where's the cry-laugh emoji when I need it.
Of course academics check where stuff is published. Please...
There are still real journals put there, although you might not know which is which.
I know what journals are better / not. But reputation only is helpful in letting you ignore trash journals, once you are out of trash land, rep is just not a very meaningful factor, and you have to focus on methodology and substance.
Also I didn't say incompetent, I said "not very". More competent researchers make journal rep only a very small factor, and it is not via the "high rep = more trustworthy" direction (which is the bad heuristic), it is "pay-to-publish journals = not trustworthy" (better heuristic).
Once you have ruled out a publication being in a trash journal, reputation is only a very minor factor in consideration, and methodological and substantive issues are what matter.
This is actually what ruined my respect for Academia.
My Science PhD buddy looked at the journal title and the claim, then said: "Its true!"
I look at him with horror. Who cares about the journal, I want to know data and methodology.
I've basically never forgiven Academia since this. I see even Ivys put out bad research and journals will publish bad research (Replication crisis and the ivy fake psychology studies)
For outsiders, there is a prestige to being a PhD or working as a professor. Now that I'm mid career and lived through the previous events I mentioned + seeing who stuck with academia... These are your C grade performers. They didnt get hired by industry, so they stayed in school. They are so protective of their artificial rank because they cannot compete in Industry. Its like being the cool person on the tennis team. They are locally cool, but not globally cool.
Spoken like someone who never went through grad school at a competitive R1 program
It was already a grueling 60-80 hour grind every week with frequent all nighters, high-pressure deadlines, absolute minimal pay, thankless duties, and plenty of politics. It's about the same for professors too.
We already paid our dues by helping peer review (for free) a half dozen papers for each one we submitted. Why should we be expected to review random papers on arxiv too...?
You know what else works really hard? A washing machine. Hard work alone doesnt create value. I could give you a spoon and tell you to dig a hole, or I can teach you how to use a Digger.
Unless you are a Claude Shannon type, adding fundamental new knowledge to humanity's corpus is generally actually hard - at least in science & engineering. If you feel differently, I look forward to reading your groundbreaking papers!
Academia will refer to my stuff. Various levels of the US government use my data.
To be honest, I think I got lucky + I was a (hardcore) Stoic for a decade + my hobby was scientific.
My washing machine creates a lot of value for me. The time it saves me is incredibly valuable.
Most machines that work really hard are valuable because they free up time.
This wasn’t the clever burn you thought it was.
And chances are: Those overbearing advisors are very invested in the current system.
You don’t need a degree to understand how much utter junk science is being published by those who think they are superior to you. Just read a few actual papers end to end and look at the data vs conclusions and it becomes totally obvious very rapidly that you cannot “trust the science” since it’s rarely actual science being done any longer.
The academic community has utterly failed at understanding they needed to cull this behavior early and mercilessly. They did not, and it will be generations at best to rebuild the trust they once had. If they ever figure out they need to.
Things are going to get much worse before they get better. You can’t take any published paper at face value any longer without going direct to primary sources and bouncing it off an expert in the space you still trust to give you the actual truth.
The GP is not saying to review each paper you read or cite. They're complaining that a colleague accepted a claim after just reading the title and where the paper was published. Between that and doing a full review there's surely a world of options.
I do some due diligence work from time to time. Uploading to arXiV is becoming a favorite tactic from companies trying to look impressive for investors. I’ve read a lot of “papers” submitted by startup founders that are obviously ChatGPT written slop uploaded to arXiV. They then go to investor and show their record of “published research”. Smart investors are catching on but there are a lot of investors who associate journals with quality and filtering and assume having a paper on there means something.
The filtering and curation problem is real. It seems like academic pettiness or laziness from the outside, until you see the volume of bad “papers” that everyone is trying to publish to chase the incentives.
Piggy back this system so that the funding source publishes the papers itself, and researchers can only publish their papers that are directly funded.
This system requires the cooperation of an organization to build the publishing infrastructure, but this could be a lowest capable bidder, and less drag on the system overall.
Because someone is surely going to try to defend journals via peer review in this thread, I want to provide a counter to the arguments that journal peer review does much good. Also, since everyone knows that if you just go to a poor enough journal, you can be published, I am going to focus on the (IMO mostly false) claim that higher-profile journals are still doing a good thing here.
There are numerous studies showing that higher-profile journals in general have more retractions and research misconduct [1-2], lower research quality [3], in fact weaker statistical power and reliability [4], and that statistical reliability even in high prestige journals is still extremely poor overall [5]. Also, making it through peer review is highly random and dependent on who you get as a reviewer [6], or is just basically a coin toss even when looking at reviewer groups:
In 2014, 49.5% of the papers accepted by the first committee were rejected by the second (with a fairly wide confidence interval as the experiment included only 116 papers). This year, this number was 50.6%. We can also look at the probability that a randomly chosen rejected paper would have been accepted if it were re-reviewed. This number was 14.9% this year, compared to 17.5% in 2014. [7]
[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC3187237/
[2] https://www.pnas.org/doi/10.1073/pnas.1212247109
[3] https://pmc.ncbi.nlm.nih.gov/articles/PMC9382220/
[4] https://journals.plos.org/plosbiology/article?id=10.1371%2Fj...
[5] https://www.frontiersin.org/journals/human-neuroscience/arti...
[6] https://journals.plos.org/plosone/article?id=10.1371%2Fjourn...
[7] https://blog.neurips.cc/2021/12/08/the-neurips-2021-consiste...
You are trying to say that high profile journals have more retractions, which is well known as you share.
How does that have anything to do with peer review? Are you saying that there is more review or less review in some cases and that influences retraction rate? In what evidence? In what world does the arxiv system moderate this discrepancy?
I already addressed this. People know peer review can be bad, but some think "good journals" still do good peer review. This is not so clear.
> In what world does the arxiv system moderate this discrepancy?
Open systems allow the scientific community to figure out ways to properly assess research quality and value more cheaply, and without passing through (often arbitrary and random) small numbers of gatekeepers that don't even do a reliable or good job gatekeeping in the first place.
I understand that we want arxiv to exist, and it does, and it’s growing. That doesn’t mean we don’t want Nature or Science to triage the most compelling stories.
Importantly, we can already begin the search for these ‘cheaper’ review strategies while not losing the helpful information filter we get by seeing where things are presented/published
No, it doesn't. The argument is that peer review is incompetent gatekeeping in general, and so slows things down and makes thing expensive. Also, I am countering the argument "we need journals because journals do peer review" by arguing "peer review by journals isn't clearly actually good", I am not saying "peer review in general is unneeded", as I support review by the entire scientific community, rather than journal gatekeepers.
> you fail to show how doing any peer review is strictly worse than doing no peer review
I wasn't trying to show that. I have provided plenty of arguments to show why killing journal-based peer review could definitely speed things up and so potentially make things better. I want actual organic review by the community, not by tiny groups of gatekeepers.
Given that reviews are not a mechanism to check for truth but soundness, the higher profile the thing I would imagine there would be more misconduct. I mean would one risk prison to steal 10$ or to steal 1 million $?
> lower research quality [3]
To cite exactly from your link "the evidence is mixed about whether they are strongly correlated with indicators of research quality.". I find saying "lower" a bit too strong given the original quote.
> in fact weaker statistical power and reliability [4]
For a specific field "cognitive neuroscience and psychology papers published recently"!
> statistical reliability even in high prestige journals is still extremely poor overall [5]
According to https://www.frontiersin.org/journals/human-neuroscience/arti... they kind of targeted bio/medical/psychology field for the analysis. Which seems to me very focused to be able to draw general conclusions.
> Also, making it through peer review is highly random and dependent on who you get as a reviewer [6], or is just basically a coin toss even when looking at reviewer groups:
It's a coin toss if paper could get accepted at all, and that's less than ideal but what the system should do (at least) is reject obvious crap, not ensure that something gets clearly accepted. The danger is False Positive (accepted even if it's crap) rather than False Negative (rejected even if it might be something useful).
Overall note: the review system is not ideal and should be improved. But it's a hard, complex and delicate problem.
I just wanted to make a strong rhetorical case by highlighting some things that might be surprising to people making more naive defenses of journals via peer-review-based arguments.
If you're going to write an article titled "The one science reform we can all agree on, but we're too cowardly to do" and that one thing isn't explicitly stated in the first paragraph, I'm out.
Stop with the meandering nonsense and make your argument.
People who write such sentences have no idea what they are talking about or are being intentionally naive for whatever reason.
Just because your one-sentence solution reads simple doesn't make the actual solution simple. Because such a solution involves changes to laws, changes to entrenched interests, changes to distribution of money involved in the whole system, and changes to balance of powers between stakeholders. Unless the push for such changes is significant enough to overcome the current state of affairs (due to public opinion, redistribution of power or money, etc.), nothing will happen.
Everything you mention will also have to happen, which means that doing this will definitely not be _easy_. That said, it is still a very _straightforward_ solution.
Fun fact, all of those things happened and this is already government policy for any NSF grant: https://www.nsf.gov/policies/document/faq-public-access
So maybe consider that when you give up on obvious things that are good based on some conspiracy theory that the "man" is trying to keep you down, what you're actually doing is being part of the system and endorsing it. Changes like this do happen, they just happen despite you.
There are plenty of simple solutions to real problems whose only blocker is upsetting the status quo. "We have no housing...let's build more housing" is, in fact, a very simple solution. That it doesn't happen has nothing to do with it the solution itself.
Journals are an academic-career-advancement service. It therefore makes sense that they do not pay academics. You don't pay your customers.
That means they need to generate a secondary customer base elsewhere, who will pay. Those secondary customers happen to be the employers of the academics who are the primary customers. That socializes the cost of providing the service, since academics individually wouldn't be willing and able to pay.
Once journals have established a reputation, their policies and paywalls and fees are the result of trying to signal exclusivity and set an optimum market price.
Until the supply side of the research market largely agrees on a way to use open-access repositories like arXiv as a primary career-advancement signal, complaining about closed-access journals is tilting at windmills.
Changing the law to prevent journals from being able to copyright anything could potentially force the research industry to rapidly develop a new solution, but at the cost of short-term chaos and career instability for new academics.
Wow! Surprised that hasn't been mentioned here already. Jumped out to me immediately as a morbidly curious bit of trivia.
And he apparently stole £763m...
I feel like this is one of those classic local minima where a community starving for resources fights vociferously amongst itself because they have internalized that they can’t win externally. From where I sit outside academia the problem with science seems obvious: there is not nearly enough money going into it.
I doubt bringing the heads of for-profit journals would change that under current national conditions in the U.S.
while the author is correct that the for-profit publishing is definitely a negative externality, i can't help but feel they are missing the forest for the trees when it comes to all the other worse issues in academia.
a full explanation of which would be much too onerous for a hn comment, but in no particular order: rampant scientific fraud, waste of tax payer dollars, wage suppression via "students" and visa-dependent laborers (J1 visa abuse), publish or perish evaluation criteria, lack of management training, blatant and rampant racism, etc. etc. etc.
the whole system needs to burn down and be rebuilt from the ground up.
Though at least in my field part of that is budgets are so tight it seems like most of the effort is needed to just keep the lights on. I don't see anyone who has bandwidth to help burn things down or rebuild in my department as much of the staff are already working unpaid overtime (and good luck getting funding for hiring many more).
The second problem, however, is a modern one: the pure, naked, and raw commercialization of science through "publish or perish", whereby the researcher is a Ford-style assembly line worker to be managed and who must be replaceable.
Without a MENTAL paradigm shift, even before a material one, we will only be able to plug small leaks on a ship with a torn hull.
Computer science as a discipline has always been relatively open and has had its own norms on publication that are different from most other fields (the top venues are almost always conferences rather than journals, and turn-around times on publications are relatively short), so it isn't a surprise that CS is one of the first areas to embrace open access.
Still, having a single example of how this approach works and how grass-roots efforts by CS researchers led to change in the community is useful to demonstrate that this idea is viable, and to motivate other research communities to follow suit.
[1]: https://authors.acm.org/open-access/acm-open-for-authors-hom...
[2]: https://creativecommons.org/licenses/by/4.0/
[3]: https://creativecommons.org/licenses/by-nc-nd/4.0/deed.en
And what, pray tell, is this advantage? If there is no utility to anyone in publishing in Science or Nature then how can it be an advantage.
I suspect it’s simply that these guys are a curation service. They separate the cranks from the science. They can be imperfect at this so long as important people separate the cranks from the science.
This kind of winnowing is pretty useful in general. Many universities are pretty much that and people pay to attend them.
It makes sense that a credentialing service would charge for the credential. It doesn’t make that much sense to say “no credentials allowed; you and timecube guy must be considered the same”. I want you to show your credentials. We all do because science is an empirical field and empiricism depends on facts. I cannot process your paper with pure reason.
If you looked under the microscope and saw light I don’t know that you didn’t. At scale I need someone to figure out “this wasn’t a Photoshop situation; that’s totally fluorescence”. Arguing that we should remove these organizations is similar to saying you should remove diplomas and so on. Anyone can continue to attend universities. They just can’t hand out diplomas. So no credentials. Only learning. Simple thing. Or perhaps not so.
The credential is the useful thing.
See also from the same author:
* https://www.experimental-history.com/p/the-rise-and-fall-of-...
* https://www.experimental-history.com/p/the-dance-of-the-nake...
* https://www.experimental-history.com/p/lets-build-a-fleet-an...
1) pay reviewers. 2) you can't publish unless a reviewer replicates your work.
yes. It can be done.
There are 3 realistic scenarios for your proposed solution:
- it will not pass
- it will be reformed later, or, if successful
- it will just make the capitalists appear at another point in the supply chain
What you hate is capitalism and capitalism will do this to any industry wherever it can attain steady profits.
SciHub has shown us a new way to spread knowledge to all that are interested. I don't have the rigor for publishing but other individual experimenters might. It would be great if they could contribute to building human knowledge.
I think the only real solution is a distributed federated publishing and review platform. A node would be a library of papers for the host's interests. Just like physical journal collections, bigger institution would host more topics. Anybody can participate in the publication and review process. SciHub nailed storage and retrieval. Review is the hard part. Any rating system can be gamed. It would be very hard to convince people it is trustworthy.
There shouldn't be any prestige in publishing a paper. The prestige comes from being proven correct, from building our knowledge.
sito42•2h ago
saghm•1h ago