What you said makes sense. The previous learning methods of language models are no longer feasible. My friends and I have recently been looking for new training methods. We believe that topology will be the next breakthrough point in the structure of language models. Anyone who is interested can discuss with me!
o11ywhisperer•6mo ago
Tell me more about this idea of topology.
From the observability realm (check username!), the relationship of data is a challenging problem. Standards like OpenTelemetry try to solve this by focusing on the relationship between technology elements with attributes and resource.attributes, along with context propagation using span and trace ids.
OTel is effectively a relational database schema. The larger questions like “If the Detroit Tigers make it to the playoffs, how much will a head of lettuce be in Berlin?” require context that machines (and humans!) lack. And, since the question is entirely made up, there might not be any relevant context.
Context powered by topology feels like the next step. Extrapolating that topology to search queries still feels like science fiction today.
HsuWL•6mo ago
I really liked your Detroit Tigers + lettuce in Berlin example.
It nails one of the core problems: language models are still dealing with “relatedness” in a super linear and flat way.
They can’t really hold a jump like that.
When I brought up topology, I wasn’t talking about anything spatial.
I meant more like a model’s thinking path needs to form its own system, a kind of closed semantic topology map.
Each node is a meaning unit, all linked by invisible threads.
The input sentence is like a little pacman moving through the map⸜( ´͈ Ⱉ `͈ )⸝
pulled along by those threads until it reaches the node that resonates the most.
That’s where the answer comes from.
So it’s not calculating, it’s being guided.
Kinda like gravity, but made out of meaning.
What you described feels super close to this.
Maybe that’s what context modeling is really heading toward…
We just haven’t found the right way to talk about it yet. Σ(๑Ⱉ⸝⸝Ⱉ๑;)੭⁾⁾
ricardobeat•6mo ago
Why do you talk/write like ChatGPT?
HsuWL•6mo ago
Because I’m not a native English speaker, I translate my comments by GPT.Haha
HsuWL•6mo ago
o11ywhisperer•6mo ago
From the observability realm (check username!), the relationship of data is a challenging problem. Standards like OpenTelemetry try to solve this by focusing on the relationship between technology elements with attributes and resource.attributes, along with context propagation using span and trace ids.
OTel is effectively a relational database schema. The larger questions like “If the Detroit Tigers make it to the playoffs, how much will a head of lettuce be in Berlin?” require context that machines (and humans!) lack. And, since the question is entirely made up, there might not be any relevant context.
Context powered by topology feels like the next step. Extrapolating that topology to search queries still feels like science fiction today.
HsuWL•6mo ago
When I brought up topology, I wasn’t talking about anything spatial. I meant more like a model’s thinking path needs to form its own system, a kind of closed semantic topology map.
Each node is a meaning unit, all linked by invisible threads. The input sentence is like a little pacman moving through the map⸜( ´͈ Ⱉ `͈ )⸝ pulled along by those threads until it reaches the node that resonates the most. That’s where the answer comes from.
So it’s not calculating, it’s being guided. Kinda like gravity, but made out of meaning.
What you described feels super close to this. Maybe that’s what context modeling is really heading toward… We just haven’t found the right way to talk about it yet. Σ(๑Ⱉ⸝⸝Ⱉ๑;)੭⁾⁾
ricardobeat•6mo ago
HsuWL•6mo ago
因為我用GPT幫我翻譯的啦！Σ(๑Ⱉ⸝⸝Ⱉ๑;)੭⁾⁾我的母語是中文 所以有點GPT味 我已經努力在訓練他講話流暢一點了啦哭哭