For now, I only have nouns (and some proper nouns) in the dataset, and pick the most common interpretation among the homographs. Also, it's case sensitive.
For now, I only have nouns (and some proper nouns) in the dataset, and pick the most common interpretation among the homographs. Also, it's case sensitive.
Navratilova - woman + man = Lendl
man - brain = woman
woman - brain = businesswomanman+vagina=woman (ok that is boring)
this is pretty fun
I think you need to disable auto-capitalisation because on mobile the first word becomes uppercase and triggers a validation error.
Edit: these must be capitalized to be recognized.
female + age = male
six (84%)
Close enough I suppose
life + death = mortality
life - death = lifestyle
drug + time = occasion
drug - time = narcotic
art + artist + money = creativity
art + artist - money = muse
happiness + politics = contentment
happiness + art = gladness
happiness + money = joy
happiness + love = joy Life + death = mortality
drug + time = occasion
drug - time = narcotic
> a drug (such as opium or morphine) that in moderate doses dulls the senses, relieves pain, and induces profound sleep but in excessive doses causes stupor, coma, or convulsions
https://www.merriam-webster.com/dictionary/narcotic
So we can see some element of losing time in that type of drug. I guess? Maybe I’m anthropomorphizing a bit.
Are you using word2vec for these, or embeddings from another model?
I also wanted to add some flavor since it looks like many folks in this thread haven't seen something like this - it's been known since 2013 that we can do this (but it's great to remind folks especially with all the "modern" interest in NLP).
It's also known (in some circles!) that a lot of these vector arithmetic things need some tricks to really shine. For example, excluding the words already present in the query[1]. Others in this thread seem surprised at some of the biases present - there's also a long history of work on that [2,3].
[1] https://blog.esciencecenter.nl/king-man-woman-king-9a7fd2935...
The dictionary is based on https://wordnet.princeton.edu/, no word2vec. It's just a plain lookup among precomputed embeddings (with mxbai-embed-large). And yes, I'm excluding words that are present in the query because.
It would be interesting to see how other models perform. I tried one (forgot the name) that was focused on coding, and it didn't perform nearly as well (in terms of human joy from the results).
(Goshawks are very intense, gyrs tend to be leisurely in flight.)
hmm...
queen - woman + man = drone
rice + fish + raw = meat
hahaha... I JUST WANT SUSHI!
data + plural = number
data - plural = research
king - crown = (didn't work... crown gets circled in red)
king - princess = emperor
king - queen = kingdom
queen - king = worker
king + queen = queen + king = kingdom
boy + age = (didn't work... boy gets circled in red)
man - age = woman
woman - age = newswoman
woman + age = adult female body (tied with man)
girl + age = female child
girl + old = female child
Curious tool but not what I would call accurate.
data + plural = datasets
data - plural = datum
king - crown = ruler
king - princess = man
king - queen = prince
queen - king = woman
king + queen = royalty
boy + age = man
man - age = boy
woman - age = girl
woman + age = elderly woman
girl + age = woman
girl + old = grandmother
The prompt I used:
> Remember those "semantic calculators" with AI embeddings? Like "king - man + woman = queen"? Pretend you're a semantic calculator, and give me the results for the following:
(what I meant to say is that it doesn't do embedding math "LIKE" the OP — not that it doesn't do embedding math at all.)
The more I think about it the less surprised I am, but my initial thoughts were quite simply "now way" - surely an approximation of an NLP model made by another NLP model can't beat the original, but the LLM training process (and data volume) is just so much more powerful I guess...
Your embedding model is literally the translation layer converting the text to numbers. The transformers are the main processing unit of the embeddings. You can even see some self-reflection in the model as the transformer is composed of attention and a MLP sub-network. The attention mechanism generates the interrelational dependence of the data and the MLP projects up into a higher dimension before coming down so that this can untangle these relationships. But the idea is that you just repeat this process over and over. The attention mechanism has the benefit over CNN models because it has a larger receptive field, so can better process long range relationships (long range being across the input data) where CNNs bias for local relationships.
(some might say all an LLM does is embeddings :)
> The results are surprisingly good, I don't think I could've done better as a human
To clarify some of the issues:
data is both singular and plural, being a mass noun[0,1]. Datum is something you'll find in the dictionary, but not common in use[2]. The dictionary lags actual definitions. I mean words only mean what we collectively agree they mean (dictionary definitely helps with that but we also invent words all the time -- i.e. slang). I see how this one could trick up a human, feeling the need to change the output and would likely consult a dictionary but I don't think that's a fair comparison here as LLMs don't have these same biases.
King - crown really seems like it should be something like "man" or "person". The crown is the manifestation of the ruling power. We still use phrases like "heavy is the head that wears the crown" in reference to general leaders, not just monarchs.
king - princess I honestly don't know what to expect. Man is technically gender neutral so I'll take this one.
king - queen I would expect similar outputs to the previous one. Don't quite agree here.
queen - king I get why is removing royalty but given the previous (two) results I think is showing a weird gender bias. Remember that queen is something like (woman + crown) and king is akin to (man + crown). So subtracting should be woman - man.
The others I agree with. These were actually done because I was quite surprised at the results and was thinking about the aforementioned gender bias.
> But keep in mind that this doesn't do embedding math like OP!
After the embedding sub-network is your main transformer network. The purpose of this network is to perform embedding math! It is just that the goal is to do significantly more complicated math. Remember, these are learnable mappings (see Optimal Transport). We're just breaking it down into their two main intermediate mappings. But the embeddings still end up being a bottleneck. It is your literal gateway from words to numbers.
[0] https://en.wikipedia.org/wiki/Mass_noun
[1] https://www.merriam-webster.com/dictionary/data
[2] https://www.sciotoanalysis.com/news/2023/1/18/this-data-or-t...
[3] https://arxiv.org/abs/2005.14165
[4] https://arxiv.org/abs/2303.08774
[4] https://www.lesswrong.com/posts/3duR8CrvcHywrnhLo/how-does-g...
The rant about network architecture misses my point, which is that an LLM does not just do a linear transformation and a similarity search. Sure, in the most abstract sense it still just computes an output embedding from two input embeddings, but only in a very distant, pedantic way. (Actually, to be VERY pedantic, that would not even be true, because ChatGPT's tokenizer embeds tokens, not words. The in- and output of the model is more than just the semantic embedding of words; using two different but semantically equivalent words may result in different outputs with a transformer LLM, but not in a word semantics model.)
I just thought it was cool that ChatGPT is so good at it.
You're right that there's subjectivity but not infinitely so. There is a bound to this and that's both required for language to work and for us to build these models. I did agree that the data one was tricky so not really going to argue, I was just pointing out a critical detail given that the models learn through pattern matching rather than a dictionary. It's why I made the comment about humans. As for ruler minus crown, I gave my explication, would you care to share yours? I'd like to understand your point of view so I can better my interpretation of the results, because frankly I don't understand. What is the semantic relationship being changed if not the attribute of ruler?
The architecture part was a miscommunication. I hope you understand how I misunderstood you when you said "this doesn't do embedding math like OP!". It is clear I'm not alone either.
> Actually, to be VERY pedantic, that would not even be true, because ChatGPT's tokenizer embeds tokens, not words.
I was just trying to help you understand a different viewpoint.
Can not personally find the connection here, was expecting father or something.
High dimension vector is always hard to explain. This is an example.
Also, if it gets buried in comments, proper nouns need to be capitalized (Paris-France+Germany).
I am planning on patching up the UI based on your feedback.
Accurate.
I’ve been unable to find it since. Does anyone know which site I’m thinking of?
Is the famous example everyone uses when talking about word vectors, but is it actually just very cherry picked?
I.e. are there a great number of other "meaningful" examples like this, or actually the majority of the time you end up with some kind of vaguely tangentially related word when adding and subtracting word vectors.
(Which seems to be what this tool is helping to illustrate, having briefly played with it, and looked at the other comments here.)
(Btw, not saying wordvecs / embeddings aren't extremely useful, just talking about this simplistic arithmetic)
100%
actor - man + woman = actress
garden + person = gardener
rat - sewer + tree = squirrel
toe - leg + arm = digit
And, worse, most latent spaces are decidedly non-linear. And so arithmetic loses a lot of its meaning. (IIRC word2vec mostly avoided nonlinearity except for the loss function). Yes, the distance metric sort-of survives, but addition/multiplication are meaningless.
(This is also the reason choosing your embedding model is a hard-to-reverse technical decision - you can't just transform existing embeddings into a different latent space. A change means "reembed all")
India - Asia + Europe = Italy
Japan - Asia + Europe = Netherlands
China - Asia + Europe = Soviet-Union
Russia - Asia + Europe = European Russia
calculation + machine = computer
I'll have to mediate on that.
cherry - picker = blackwood
I built a game[0] along similar lines, inspired by infinite craft[1].
The idea is that you combine (or subtract) “elements” until you find the goal element.
I’ve had a lot of fun with it, but it often hits the same generated element. Maybe I should update it to use the second (third, etc.) choice, similar to your tool.
great idea, but I find the results unamusing
map - legend = Mercator projection
noodle - wheat = egg noodle
noodle - gluten = tagliatelle
architecture - calculus = architectural style
answer - question = comment
shop - income = bookshop
curry - curry powder = cuisine
rice - grain = chicken and rice
rice + chicken = poultry
milk + cereal = grain
blue - yellow = Fiji
blue - Fiji = orange
blue - Arkansas + Bahamas + Florida - Pluto = GrenadaOther stuff that works: key, door, lock, smooth
Some words that result in "flintlock": violence, anger, swing, hit, impact
Getting to cornbread elegantly has been challenging.
Or maybe they would all be completely inscrutable and man-woman would be like the 50th strongest result.
paleolith + cat = Paleolithic Age
paleolith + dog = Paleolithic Age
paleolith - cat = neolith
paleolith - dog = hand ax
cat - dog = meow
Wonder if some of the math is off or I am not using this properly
hacker - code = professional golf
Very few papers that actually say something meaningful are left unnoticed, but as soon as you say something generic like "language models can do this", it gets featured in "AI influencer" posts.
That could be seen as trying to find the true "meaning" of a word.
woman + intelligence = man (77%)
Oof.
blue + red = yellow (87%) -- rgb, neat
black + {red,blue,yellow,green} = white 83% -- weird
Good to understand this bias before blindly applying these models (Yes- doctor is gender neutral - even women can be doctors!!)
huh
antidnan•8h ago
https://neal.fun/infinite-craft/
thaumasiotes•6h ago
It provides a panel filled with slowly moving dots. Right of the panel, there are objects labeled "water", "fire", "wind", and "earth" that you can instantiate on the panel and drag around. As you drag them, the background dots, if nearby, will grow lines connecting to them. These lines are not persistent.
And that's it. Nothing ever happens, there are no interactions except for the lines that appear while you're holding the mouse down, and while there is notionally a help window listing the controls, the only controls are "select item", "delete item", and "duplicate item". There is also an "about" panel, which contains no information.
n2d4•6h ago
thaumasiotes•6h ago
n2d4•6h ago
[0] https://youtu.be/8-ytx84lUK8