1. The data in most of the plots (see the appendix) look fake. Real life data does not look that clean.
2. In May of 2022, 6 months before chatGPT put genAI in the spotlight, how does a second-year PhD student manage to convince a large materials lab firm to conduct an experiment with over 1,000 of its employees? What was the model used? It only says GANs+diffusion. Most of the technical details are just high-level general explanations of what these concepts are, nothing specific.
"Following a short pilot program, the lab began a large-scale rollout of the model in May of 2022." Anyone who has worked at a large company knows -- this just does not happen.
Could a Benford's Law analysis apply here to detect that?
https://en.wikipedia.org/wiki/When_Contact_Changes_Minds
https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&d...
> As we examined the study’s data in planning our own studies, two features surprised us: voters’ survey responses exhibit much higher test-retest reliabilities than we have observed in any other panel survey data, and the response and reinterview rates of the panel survey were significantly higher than we expected.
> The firm also denied having the capabilities to perform many aspects of the recruitment procedures described in LaCour and Green (2014).
The data quality for that would need to be unimaginably high.
% gunzip -c arXiv-2412.17866v1.tar.gz | tar xOf - main.tex | grep '\bI have\b'
To summarize, I have established three facts. First, AI substantially increases the average rate of materials discovery. Second, it disproportionately benefits researchers with high initial productivity. Third, this heterogeneity is driven almost entirely by differences in judgment. To understand the mechanisms behind these results, I investigate the dynamics of human-AI collaboration in science.
\item Compared to other methods I have used, the AI tool generates potential materials that are more likely to possess desirable properties.
\item The AI tool generates potential materials with physical structures that are more distinct than those produced by other methods I have used.
% gunzip -c arXiv-2412.17866v1.tar.gz | tar xOf - main.tex | grep '\b I \b' | wc
25 1858 12791
%Nice to include the giants we stand on as implied coauthors.
(In this q&a, the audience does not really question the validity of the research.)
I miss google search's Cache. As with the seminar, several other hits on MIT pages have been removed. I'm reminded of a PBS News Hour story, on free fusion energy from water in your basement (yes, really), which was memory holed shortly after. The next-ish night they seemed rather put out, protesting they had verified the story... with "a scientist".
That cassyni talk link... I've seen a lot of MIT talks (a favorite mind candy), and though Sloan was underrepresented, that looked... more than a little odd. MIT Q&A norms are diverse, from the subtle question you won't appreciate if you haven't already spotted the fatal flaw, to leaving the speaker in tears. I wonder if there's a seminar tape.
Better title:
MIT disavows heavily-discussed economics preprint paper about Artificial Intelligence and Scientific Discovery.
(Since press release titles about negative news tend to studiously avoid saying anything, we tend to classify them in the "misleading" bucket of https://news.ycombinator.com/newsguidelines.html, which justifies rewriting them.)
[0]: https://info.arxiv.org/help/withdraw.html: "Articles that have been announced and made public cannot be completely removed. A withdrawal creates a new version of the paper marked as withdrawn."
> The paper was championed by MIT economists Daron Acemoglu, who won the 2024 economics Nobel, and David Autor. The two said they were approached in January by a computer scientist with experience in materials science who questioned how the technology worked, and how a lab that he wasn’t aware of had experienced gains in innovation. Unable to resolve those concerns, they brought it to the attention of MIT, which began conducting a review.
Seems pretty serious if they kicked him out.
Lay low for a year, work on some start-up-ish looking project, then use his middle name to get hired at one of the many AI startups? (only half joking)...
Also, there are companies who will see that win at any cost mentality as a positive trait.
I'm betting whoever it is, is okay now.
They may have thought they could jump into an industry job, including the paper and all of its good press coverage on their resume. Only the author can retract an arXiv paper, not their academic institution. It wouldn't be hard to come up with a story that they decided to leave the academic world and go into industry early.
MIT coming out and calling for the paper's retraction certainly hampers that plan. They could leave it up and hope that some future employer is so enamored with their resume that nobody does a Google search about it, but eventually one of their coworkers is going to notice.
He probably will never be someone of significance, but he also will probably be able to have a standard middle class life.
That high-level Apple employee was probably a manager and oversaw hiring people.
I would tell myself every day, "I wouldn't hire me."
It's not self-defeating.
It's not being a victim.
I wouldn't let it stop me from trying.
It's being accurate about what kind of company you'd want to build yourself, and the internal state of a lot of hiring managers. And with a true model of the world you can make better decisions.
Between this and the subtle reference to “former second-year PhD student” it makes sense that they’d have to make a public statement.
They do a good job of toeing the required line of privacy while also giving enough information to see what’s going on.
I wonder if the author thought they could leave the paper up and ride it into a new position while telling a story about voluntarily choosing to leave MIT. They probably didn’t expect MIT to make a public statement about the paper and turn it into a far bigger news story than it would have been if the author quietly retracted it.
Edit: this comment was only partially serious, not meant as legal advice to MIT.
Furthermore, if the author could demonstrate to arXiv that the request was fraudulent, the paper would be reinstated. The narrative would also switch to people being angry at MIT for impersonating a student to do something.
Emails are not people. You can impersonate a person, but you can't impersonate an email. If I own a company and I issue the email dick.less@privateequity.com but then have to fire him... using this email address to transfer company assets back to someone who can be responsible for them isn't fraud (for that purpose, at least). How is this not the same issue?
Although not explicitly stated, i read previous comments as using dick.less@privateequity.com to cancel his personal Netflix account. (Let's say that privateequity.com allowed personal usage of company email.)
I see a difference between accessing an email account and impersonating the previous account holder.
[0] E.g. arxiv/0812.0848: "This paper has been withdrawn by the author due to a crucial definition error of Triebel space".
It sounds like "we don't like it and won't tell you why, we're hiding behind MIT policy and vague notions of privacy".
MIT should just demonstrate in a paper what the shortcomings are and print it, adding it to the citation tree of the original.
Looking very briefly at the paper and speculating wildly, I could imagine that the company who were subject of it - or their staff - might not appreciate it and have put pressure on MIT??
Solid amount of Streisand Effect going on here -- lots of attention has been bought to the paper (and that is everything after all!).
The apparent issue is that the data appears to have been entirely fabricated and is a lie. The author appears to simply be a fraud
FERPA is federal law. It is quite likely that MIT is legally bound to not release some pieces of evidence which are crucial in this case (hypothetically, for example: that the student's educational record is inconsistent with claims made in the paper).
(Incidentally, I don't think misplaced trust in preprints is much of an academic issue, people that are experts in their field can easily judge quality for themselves. It's laypeople taking them at face value that's the problem.)
*Note that this creates a new version lacking any download which also becomes the default but any previous ones are still available.
non-scientific studies can't be replicated
Whether MIT is right or wrong, the arrogance displayed is staggering. The only thing more shocking is that obviously this behavior works for them and they are used to people obeying them without question because they are MIT.
I think MIT is trying to protect its reputation as a would-be place of fraud-free research, unlike Harvard.
Edit: Since the paper has been cited, others may still need to reference the paper to determine if it materially affects a paper citing it. If the paper is removed it’s just a void.
So the PhD student might have been kicked out. But what about the people who "championed it". If they worked with the student, surely they might have figured out the mythical lab full of 1000s material scientists might not exist, it might exist but they never actually used any AI tool.
Sadly, it has devolved into a pompous bastion of elitist snobs that relishes in bullying and stifling access to information.
Never forget that MIT killed Aaron Swartz for setting information free.
pvg•7h ago
https://news.ycombinator.com/item?id=42115310
tsurba•5h ago
mmooss•2h ago
dang•4h ago
Artificial Intelligence, Scientific Discovery, and Product Innovation [pdf] - https://news.ycombinator.com/item?id=42115310 - Nov 2024 (47 comments)