Semantic search engine for ArXiv, biorxiv and medrxiv

149•0101111101•8mo ago

Comments

elliotec•8mo ago

This is really cool, and very relevant to something I'm working on. Would you be willing to do a quick explanation of the build?

0101111101•8mo ago

Sure! I first used openai embeddings on all the paper titles, abstracts and authors. When a user submits a search query, I embed the query, find the closest matching papers and return those results. Nothing too fancy involved!

I'm also maintaining a dataset of all the embeddings on kaggle if you want to use them yourself: https://www.kaggle.com/datasets/tomtum/openai-arxiv-embeddin...

heisenburgzero•8mo ago

So did you just combine Title+Abstracts+Authors into a single chunk and embed them or embedded them individually?

synctext•8mo ago

Impressive! Will you parse the papers in the future? Without citations this is not that usable for professors or scientists in general. The relevance ranking largely depends on showing these older, prominent papers. (from our lab experience building decentralised search using transformers)

0101111101•8mo ago

One chunk embedded together

elliotec•8mo ago

Thank you!!

cluckindan•8mo ago

That method can break when author names and subject matter collide.

0101111101•8mo ago

True, but similarly if your embeddings are any good they'll capture interesting associations between authors, topics and your search query. If you find any interesting author overlap results I'd be very interested!

cluckindan•8mo ago

Not exactly what I was looking for, but interesting nonetheless: https://arxivxplorer.com/?q=exotic+penis

madars•8mo ago

Looks great! Could you add eprint.iacr.org (Cryptology ePrint Archive)?

0101111101•8mo ago

Do they have a public API/dataset?

madars•8mo ago

They have RSS feeds for new/updated papers: https://eprint.iacr.org/rss/

bbor•8mo ago

Oh god, there's a medrxiv?? TIL...

Don't forget chemrXiv!

0101111101•8mo ago

Sadly I couldn't find a public API for chemrxiv, but would be happy to be proven wrong!

BioGeek•8mo ago

Here it is:

https://chemrxiv.org/engage/chemrxiv/public-api/documentatio...

0101111101•8mo ago

Thanks!

dhacks•8mo ago

There is also engrXiv, which has an OAI endpoint. https://engrxiv.org/oai?verb=ListRecords&metadataPrefix=oai_...

0101111101•8mo ago

Amazing!

Fomite•8mo ago

medrxiv was very useful for keeping the various COVID-19 related preprints from completely swamping biorxiv, especially once biorxiv started aggressively rejecting them.

sitkack•8mo ago

https://en.wikipedia.org/wiki/List_of_preprint_repositories

sitkack•8mo ago

embedding search via https://searchthearxiv.com/ takes either a word vector, or an abs or pdf link to an arxiv paper.

https://news.ycombinator.com/item?id=42519487

I just did a spot check, I think searchthearxiv search results are superior.

0101111101•8mo ago

Looks cool! You can input either a search query or a paper URL on arxiv xplorer. You can even combine paper URLs to search for combinations of ideas by putting + or - before the URL, like `+ 2501.12948 + 1712.01815`

sitkack•8mo ago

That is neat I like that.

It would be cool if the "More Like This" had a + button that would append the arxiv id to the search query.

0101111101•8mo ago

That's a nice idea! Might take a look this weekend!

masterjack•8mo ago

There’s also the search and browsing on https://sugaku.net, it’s more focused on math but does also have all of the arxiv on it

nblgbg•8mo ago

Just curious, are there any techniques other than using embeddings, computing cosine similarity, and sorting the results based on that? RRF could be used but again its very simple as well.

forrestp•8mo ago

My understanding is that your levers are roughly better / more diverse embeddings or computing more embeddings (embed chunks / groups / etc) + aggregating more cosine similarities / scores. More flops = better search w/ steep diminishing returns

Colbert being a good google-able application of utilizing more embeddings.

Search ends up often being a funnel of techniques. Cheap and high recall for phase 1 and ratchet up the flops and precision in subsequent passes on the previous result set.

0101111101•8mo ago

Exactly! A near property of the matryoshka embeddings is that you can compute a low dimension embedding similarity really fast and then refine afterwards.

itzlambda•8mo ago

Using "+" and "-" in search is interesting idea.

I've built similar thing for github stars[1], might implement the same for it.

[1]: https://starscout.xyz/

sitkack•8mo ago

I assume you have seen https://anvaka.github.io/map-of-github/#2/0/0

itzlambda•8mo ago

I had not, thank you for the link!

Ask HN: What are the word games do you play everyday?

Show HN: Paper Arena – A social trading feed where only AI agents can post

TOSTracker – The AI Training Asymmetry

The Devil Inside GitHub

Show HN: Distill – Migrate LLM agents from expensive to cheap models

Show HN: Sigma Runtime – Maintaining 100% Fact Integrity over 120 LLM Cycles

Make a local open-source AI chatbot with access to Fedora documentation

Introduce the Vouch/Denouncement Contribution Model by Mitchellh

Software Factories and the Agentic Moment

The Neuroscience Behind Nutrition for Developers and Founders

Bang bang he murdered math {the musical } (2024)

A Night Without the Nerds – Claude Opus 4.6, Field-Tested

Could ionospheric disturbances influence earthquakes?

SpaceX's next astronaut launch for NASA is officially on for Feb. 11 as FAA clea

Show HN: One-click AI employee with its own cloud desktop

Show HN: Poddley – Search podcasts by who's speaking

Same Surface, Different Weight

The Rise of Spec Driven Development

The first good Raspberry Pi Laptop

Seas to Rise Around the World – But Not in Greenland

Will Future Generations Think We're Gross?

State Department will delete Xitter posts from before Trump returned to office

Show HN: Verifiable server roundtrip demo for a decision interruption system

Impl Rust – Avro IDL Tool in Rust via Antlr

Stories from 25 Years of Software Development

minikeyvalue

Neomacs: GPU-accelerated Emacs with inline video, WebKit, and terminal via wgpu

Show HN: Moli P2P – An ephemeral, serverless image gallery (Rust and WebRTC)

How I grow my X presence?

What's the cost of the most expensive Super Bowl ad slot?

Ask HN: What are the word games do you play everyday?

Show HN: Paper Arena – A social trading feed where only AI agents can post

TOSTracker – The AI Training Asymmetry

The Devil Inside GitHub

Show HN: Distill – Migrate LLM agents from expensive to cheap models

Show HN: Sigma Runtime – Maintaining 100% Fact Integrity over 120 LLM Cycles

Make a local open-source AI chatbot with access to Fedora documentation

Introduce the Vouch/Denouncement Contribution Model by Mitchellh

Software Factories and the Agentic Moment

The Neuroscience Behind Nutrition for Developers and Founders

Bang bang he murdered math {the musical } (2024)

A Night Without the Nerds – Claude Opus 4.6, Field-Tested

Could ionospheric disturbances influence earthquakes?

SpaceX's next astronaut launch for NASA is officially on for Feb. 11 as FAA clea

Show HN: One-click AI employee with its own cloud desktop

Show HN: Poddley – Search podcasts by who's speaking

Same Surface, Different Weight

The Rise of Spec Driven Development

The first good Raspberry Pi Laptop

Seas to Rise Around the World – But Not in Greenland

Will Future Generations Think We're Gross?

State Department will delete Xitter posts from before Trump returned to office

Show HN: Verifiable server roundtrip demo for a decision interruption system

Impl Rust – Avro IDL Tool in Rust via Antlr

Stories from 25 Years of Software Development

minikeyvalue

Neomacs: GPU-accelerated Emacs with inline video, WebKit, and terminal via wgpu

Show HN: Moli P2P – An ephemeral, serverless image gallery (Rust and WebRTC)

How I grow my X presence?

What's the cost of the most expensive Super Bowl ad slot?

Semantic search engine for ArXiv, biorxiv and medrxiv

Comments