The Bluesky Dictionary

https://www.avibagla.com/blueskydictionary/

212•gaws•6mo ago

Comments

neaden•6mo ago

Is this not working or am I missing something, it just shows as seeing 0 words for me. Firefox on a PC.

SirFatty•6mo ago

Same... maybe you need a Bluesky account, which I don't have.

gpm•6mo ago

It doesn't... I can open it in a private browsing window.

GalaxyNova•6mo ago

It's working fine for me on Firefox

accrual•6mo ago

You may need to allow scripts from the domain avibagla.com, it shows 0 when the scripts are blocked.

zem•6mo ago

ugh, it ought to be building the results on the server and serving up static pages.

rafram•6mo ago

But it updates live...

forgotmypw17•6mo ago

It could do both...

dymk•6mo ago

then go build it…

Noumenon72•6mo ago

All the scripts are ERR_SSL_PROTOCOL_ERROR for me in Chrome, I'm assuming because of a corporate firewall.

71bw•6mo ago

Indeed, all I get in Firefox are CORS issues

AgentME•6mo ago

For me it took a minute to start loading data and switch from just showing 0.

GalaxyNova•6mo ago

fascinating! I think it's really cool that this is possible, and at the same time kine of sad that the norm is slowly moving towards more locked-down APIs.

timeon•6mo ago

> slowly moving towards

Depends what we accept as norm.

75345d4c•6mo ago

I just saw it indexed "eluvium," but the post was referring to a band with that same name

Kye•6mo ago

GeologySky will get to it soon enough.

k7sune•6mo ago

Thanks to this I just learned about alluvium, eluvium, illuvium, and colluvium.

atlgator•6mo ago

I checked out the author's other projects and this is common issue. For example, he has a "lean checker" for bluesky that claims it is right-leaning simply because of all the people saying "That's right," "He was right," etc. None of the supposed right-leaning posts were actually conservative in nature. They just used to word right to mean correct.

avibagla1•6mo ago

one, thank you for checking my website. two, that is the joke, 100% - at the time people kept talking about how "left leaning" bsky was and that idea came to mind

OneDeuxTriSeiGo•6mo ago

lmao that's fantastic

wantlotsofcurry•6mo ago

I'm very curious as to how this works in the backend. I realize it uses Bluesky's firehose to get the posts, but I'm more curious on how it's checking whether a post contains any of the available words. Any guesses?

bangaladore•6mo ago

Maybe I'm being naive, but with only ~275k words to check against, this doesn't seem like a particularly hard problem. Ingest post, split by words, check each word via some db, hashmap, etc... and update metadata.

somebehemoth•6mo ago

I think the cool part is watching words go brrr.

gpm•6mo ago

Probably just a big hashtable mapping word -> the number of times it's been seen, and another hashset of all the words it hasn't seen. When a post comes in you hash all the words in it and look them up in the hashtable, increment it, and if the old value was 0 remove it from the hash set.

250k words at a generous 100 bytes per word is only 25MB of memory...

f311a•6mo ago

You can probably fit all words under 10-15MB of memory, but memory optimisations are not even needed for 250k words...

Trie data structures are memory-efficient for storing such dictionaries (2-4x better than hashmaps). Although not as fast as hashmaps for retrieving items. You can hash the top 1k of the most common words and check the rest using a trie.

The most CPU-intensive task here is text tokenizing, but there are a ton of optimized options developed by orgs that work on LLMs.

stwrzn•6mo ago

I very much hope that the backend uses one of the bluesky jetstream endpoints. When you only subscribe to new posts, it provides a stream of around 20mbit/s last time I checked, while the firehose was ~200mbit/s.

avibagla1•6mo ago

yes it does!

avibagla1•6mo ago

Hey! this is my site - it's not all that complex, i'm just using a sqlite db with two tables - one for stats, the other for all the words that's just word | count | first use | last use | post.

I... did not expect this to be so popular

gumboshoes•6mo ago

What is your source dictionary to compare to? Seems kind of small. Also, how are you handling inflected forms?

avibagla1•6mo ago

https://github.com/words/an-array-of-english-words

using this, a combo of "covered enough" for the bit and easy to use

also, since i'm tracking every word (technically a better name for this project would be The Bluesky Corpus) all inflected forms are different words, which aligns with my thinking

blendo•6mo ago

What are the table sizes?

And what ingress bandwidth do you have?

avibagla1•6mo ago

DB is currently 58mb (damn lol)

Ingress is actually pretty manageable, ~900kbps

spullara•6mo ago

I did this against a pretty large tweet archive and got hits on about 125k of the words in the unix dictionary.

pona-a•6mo ago

For a moment I thought it would be an AT-Proto based Urban Dictionary clone.

evbogue•6mo ago

This

tough•6mo ago

Words We Haven't Seen

- Search unseen words

made me chuckle

crm9125•6mo ago

I've found content for all of my future skeets.

refreeze654•6mo ago

I've wondered how blueksy affords the bandwidth to let anyone stream the full firehose.

dgacmu•6mo ago

Not an answer to your question, but I suspect most people don't -- my bot (a pi searcher bot, of course) just runs on Jetstream, which is pretty lightweight and heavily compressed.

(The website in question uses jetstream also.)

psionides•6mo ago

From what they say, it is a lot, but it's generally on the order of a few hundreds of connections total at the moment

blintz•6mo ago

I'm surprised at how normal some of the unseen words are. I expected them to all be archaic or niche, but many are pretty reasonable: 'congregant', 'definer', 'stereoscope'.

gkoberger•6mo ago

For what it's worth, there's 1.7bn posts on Bluesky according to this: https://bsky.jazco.dev/stats

The dictionary site has only checked 4,920,000 posts, which is 0.28% of all messages.

SomewhatLikely•6mo ago

It's likely that the commenter has read less than 5 million posts worth of text though. So perhaps this still points to a lack of diversity in content.

wolfram74•6mo ago

You got me wondering. Supposing the average post is 10 words, and a typical page of text is 250 words, that would only be ~50 pages of text a day over the last 10 years. Which I don't think I manage, but over 20 years I am probably in that window.

IshKebab•6mo ago

It now claims to have checked 11 million posts but only seen "the" 16 thousand times. I'm not sure its numbers are entirely reliable.

awinter-py•6mo ago

dentel, exclaustrations, gryding, datolite, frabbing?

Kye•6mo ago

I can't keep up with all these new Pokemon.

fudged71•6mo ago

I'm just surprised that there's revolt when Bluesky posts are used for LLMs, but regular NLP is fine for some reason.

max_•6mo ago

This website is so pretty!

runflyswim•6mo ago

thank you!! design support and advice from my good friend vedantswarup.com

anywhichway•6mo ago

I noticed one of the cited bluesky posts was all in French, so one might argue that technically it didn't find the English word "mouch", but rather a different French word that happens to be spelled the same. But trying to sort that out seems unrealistically challenging. "Mouch" is only in the dictionary as an alternative spelling to mooch, so probably a pretty rare word to see in English.

steinuil•6mo ago

Bluesky lets you select the language your post is written in before posting it and it is attached as metadata to the skeet. I guess the backend for this only searches posts in English, but it's possible the dataset is not 100% accurate due to some users forgetting to switch language before posting.

croes•6mo ago

So now someone is simply posting a dictionary

xnorswap•6mo ago

Someone just got a double-combo:

> We just visited wheal Martyn museum in Cornwall, nice scones and a waterwheel, they also have a lot of gutters, sluices and pipes and a bit of a fixation about China Clay. More importantly they appear to be unattached at the moment

Both "wheal" (kind of cheating, that should be Wheal and is a place name) and "sluices" were new to the dictionary.

OpenCiv3: Open-source, cross-platform reimagining of Civilization III

The Waymo World Model

What Is Ruliology?

How we made geo joins 400× faster with H3 indexes

Unseen Footage of Atari Battlezone Arcade Cabinet Production

Show HN: Look Ma, No Linux: Shell, App Installer, Vi, Cc on ESP32-S3 / BreezyBox

Jeffrey Snover: "Welcome to the Room"

Monty: A minimal, secure Python interpreter written in Rust for use by AI

Show HN: I spent 4 years building a UI design tool with only the features I use

Microsoft open-sources LiteBox, a security-focused library OS

Sheldon Brown's Bicycle Technical Info

Hackers (1995) Animated Experience

Show HN: If you lose your memory, how to regain access to your computer?

An Update on Heroku

Delimited Continuations vs. Lwt for Threads

Dark Alley Mathematics

PC Floppy Copy Protection: Vault Prolok

Was Benoit Mandelbrot a hedgehog or a fox?

How to effectively write quality code with AI

Introducing the Developer Knowledge API and MCP Server

I spent 5 years in DevOps – Solutions engineering gave me what I was missing

Understanding Neural Network, Visually

Show HN: R3forth, a ColorForth-inspired language with a tiny VM

I now assume that all ads on Apple news are scams

Why I Joined OpenAI

Female Asian Elephant Calf Born at the Smithsonian National Zoo

Learning from context is harder than we thought

Vocal Guide – belt sing without killing yourself

FORTH? Really!?

Zlob.h 100% POSIX and glibc compatible globbing lib that is faste and better

OpenCiv3: Open-source, cross-platform reimagining of Civilization III

The Waymo World Model

What Is Ruliology?

How we made geo joins 400× faster with H3 indexes

Unseen Footage of Atari Battlezone Arcade Cabinet Production

Show HN: Look Ma, No Linux: Shell, App Installer, Vi, Cc on ESP32-S3 / BreezyBox

Jeffrey Snover: "Welcome to the Room"

Monty: A minimal, secure Python interpreter written in Rust for use by AI

Show HN: I spent 4 years building a UI design tool with only the features I use

Microsoft open-sources LiteBox, a security-focused library OS

Sheldon Brown's Bicycle Technical Info

Hackers (1995) Animated Experience

Show HN: If you lose your memory, how to regain access to your computer?

An Update on Heroku

Delimited Continuations vs. Lwt for Threads

Dark Alley Mathematics

PC Floppy Copy Protection: Vault Prolok

Was Benoit Mandelbrot a hedgehog or a fox?

How to effectively write quality code with AI

Introducing the Developer Knowledge API and MCP Server

I spent 5 years in DevOps – Solutions engineering gave me what I was missing

Understanding Neural Network, Visually

Show HN: R3forth, a ColorForth-inspired language with a tiny VM

I now assume that all ads on Apple news are scams

Why I Joined OpenAI

Female Asian Elephant Calf Born at the Smithsonian National Zoo

Learning from context is harder than we thought

Vocal Guide – belt sing without killing yourself

FORTH? Really!?

Zlob.h 100% POSIX and glibc compatible globbing lib that is faste and better

The Bluesky Dictionary

Comments