Ask HN: (Why) Was the LLM breakthrough useful for images, audio, etc.?

3•rogerrogerr•14h ago

This has been bothering me for a while. I feel like I have a decent conceptual grasp of what LLMs are doing with written text. But it seems like they also unlocked a bunch of progress in understanding and generating images, audio, and video. I can’t twist my brain into understanding the connection.

Is the boom in generated non-text content also built on LLMs, or is it just correlated with it because a bunch of excitement drove investment into the industry? I’m hoping for an ELI-non-ai-but-cs-major, this has been bothering me for a while.

Comments

jerlendds•13h ago

VLM != LLM. Vision language models basically treat text tokens and image tokens the same. Post-training an LLM on images+text can improve its capabilities. Id recommend searching around the keyword VLM to find more resources on how multi-modal AI works.

- https://huggingface.co/blog/vlms

- https://en.wikipedia.org/wiki/Multimodal_learning

verdverm•12h ago

transformers were first an image understanding technique, the text processing came later, it's all about the training data and gradient descent, and allegedly attention

Ask HN: How do you manage skills files?

Ask HN: Fable hacked my piano, can I release the results?

Ask HN: Would you read a statistics textbook?

Ask HN: Show your micro-SaaS

Tell HN: Anthropic should make Claude the Author and me the Co-Author

Ask HN: UK Rescue Rocket Sheds/Houses Information

Ask HN: Looking for an old submission about sneering culture

Ask HN: Connecting Kubernetes dependencies to application telemetry

Ask HN: What are other "uncrackable benchmarks"?

Ask HN: Any Software Engineers here who enjoy their AI-native dev workflow?

Ask HN: (Why) Was the LLM breakthrough useful for images, audio, etc.?

Ask HN: Perplexity Pro vs. Claude – what are you paying for?

Ask HN: Who is using MCP in production?

Ask HN: Resources to get good at soldering?

Ask HN: Why were OpenAI, Claude, and Grok simultaneously down?

Ask HN: How safe are our password managers in face of LLM cyber attacks?

Ask HN: What is your biggest struggle while learning?

Ask HN: Why don't we bring back old school OkCupid?

Ask HN: What are you working on?

Tell HN: I want to see the same moon as you

Ask HN: Model sycophancy benchmarks?

Tell HN: Both recent GCP outages caused by fiber optic maintenance

Ask HN: Are we normalizing surveillance in the name of safety?

Astra still fails at creating acceptable Rotational Ambigram

Ask HN: Are others seeing Google's reCAPTCHA rejecting Firefox users?

Ask HN: 30, burnt out, with 6-10 months of runway. What would you do?

Ask HN: Do you route only certain websites through a VPN?

APAC Alpha – Zero-lag Intel terminal for Asian markets

Ask HN: What's the best place to learn about skills and AI for coding in general

Ask HN: Any Alternatives to Archive.ph & co?

Ask HN: How do you manage skills files?

Ask HN: Fable hacked my piano, can I release the results?

Ask HN: Would you read a statistics textbook?

Ask HN: Show your micro-SaaS

Tell HN: Anthropic should make Claude the Author and me the Co-Author

Ask HN: UK Rescue Rocket Sheds/Houses Information

Ask HN: Looking for an old submission about sneering culture

Ask HN: Connecting Kubernetes dependencies to application telemetry

Ask HN: What are other "uncrackable benchmarks"?

Ask HN: Any Software Engineers here who enjoy their AI-native dev workflow?

Ask HN: (Why) Was the LLM breakthrough useful for images, audio, etc.?

Ask HN: Perplexity Pro vs. Claude – what are you paying for?

Ask HN: Who is using MCP in production?

Ask HN: Resources to get good at soldering?

Ask HN: Why were OpenAI, Claude, and Grok simultaneously down?

Ask HN: How safe are our password managers in face of LLM cyber attacks?

Ask HN: What is your biggest struggle while learning?

Ask HN: Why don't we bring back old school OkCupid?

Ask HN: What are you working on?

Tell HN: I want to see the same moon as you

Ask HN: Model sycophancy benchmarks?

Tell HN: Both recent GCP outages caused by fiber optic maintenance

Ask HN: Are we normalizing surveillance in the name of safety?

Astra still fails at creating acceptable Rotational Ambigram

Ask HN: Are others seeing Google's reCAPTCHA rejecting Firefox users?

Ask HN: 30, burnt out, with 6-10 months of runway. What would you do?

Ask HN: Do you route only certain websites through a VPN?

APAC Alpha – Zero-lag Intel terminal for Asian markets

Ask HN: What's the best place to learn about skills and AI for coding in general

Ask HN: Any Alternatives to Archive.ph & co?

Ask HN: (Why) Was the LLM breakthrough useful for images, audio, etc.?

Comments