frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

fp.

Open in hackernews

Why does SSH send 100 packets per keystroke?

https://eieio.games/blog/ssh-sends-100-packets-per-keystroke/
140•eieio•1h ago

Comments

snowmobile•1h ago
> That 20ms is a smoking gun - it lines up perfectly with the mysterious pattern we saw earlier!

Speaking of smoking guns, anybody else reckon Claude overuses that term a lot? Seems anytime I give it some debugging question, it'll claim some random thing like a version number or whatever, is a "smoking gun"

Hikikomori•1h ago
It's a smoking gun of Claude usage.
lloydatkinson•1h ago
smoking gun, you're absolutely right, good question, em dash, "it isn't just foo, it's also bar", real honest truth, brutal truth, underscores the issue, delves into, more em dashes, <20 different hr/corporate/cringe phrases>.

It's nauseating.

cubano•1h ago
Come on...haven't we all had to deal with the crazy smart lead who was loaded with those same types of annoying tics?

Considering what these LLMs bring to the table, I think a little tolerance for their cringe phrases is in order.

jcynix•1h ago
It's what they read on The Internets when training, so don't expect them to generate new phrases, other than what they learned from it?
Terretta•58m ago
### The answer that fits everything (and what to do about it)
calvinmorrison•39m ago
cant wait for chatgpt to make me read about grandmas secret recipe and scroll through 6 ads to see the ingredients for my chicken teriyaki dinner
jcynix•17m ago
Maybe we need a real AI which creates new phrases and teaches the poor LLMs?

Looking back we already had similar problems, when we had to ask our colleagues, students, whomever "Did you get your proposed solution from the answers part or the questions part of a stackoverflow article?" :-0

hamdingers•21m ago
You might find this a fun read: https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
Telemakhos•1h ago
I've love to delve into that.

https://pshapira.net/2024/03/31/delving-into-delve/

bdamm•1h ago
Without knowing how LLM's personality tuning works, I'd just hazard a guess that the excitability (tendency to use excided phrases) is turned up. "smoking gun" must be highly rated as a term of excitability. This should apply to other phrases like "outstanding!" or "good find!" "You're right!" etc.
yread•1h ago
ChatGPT too. And "lines up perfectly" when it doesnt actually line up with anything
dave78•1h ago
Same with Gemini.
MonkeyClub•27m ago
You can absolutely see this pattern in Gemini in 2026.

Btw, is the injection of "absolutely" and "in $YEAR" prevalent in other LLMs as well, or is it just in Gemini's dialect?

cristoperb•22m ago
It's just Gemini. I'm guessing they changes the system prompt for the new year or something, but it's pretty annoying.
locallost•1h ago
I chuckled out loud. It's funny cause it's true.
redwall_hp•1h ago
"You're so right, that nice catch lines up perfectly!"
smallmancontrov•1h ago
It's not just a coincidence, it's the emergence of spurious statistical correlations when observations happen across sessions rather than within sessions.
f1shy•55m ago
You can add an M-dash, and we completed the bs-bingo. :)
cipehr•1h ago
I don't think claude has even once used this in my conversations (Claude Desktop, Claude Code, Voice conversations...) Sycophancy, yes absolutely!

Maybe it has something to do with your profile/memories?

eieio•1h ago
Yes! While this post was written entirely by me, I wouldn't be surprised if I had "smoking gun" ready to go because I spent so much time debugging with Claude last night.
gf000•1h ago
Reminds me of ethimology nerd's videos. He has some content about how LLMs will influence human language.
grim_io•1h ago
The "maybe" of yesterday is the "you're absolutely right!" of tomorrow.
hinkley•57m ago
Some day in the future we will complain about AIs with a 2015 accent because that’s the last training data that wasn’t recursive.
ranger_danger•53m ago
shouldn't it be "human language influences human language"?
rubslopes•39m ago
It's interesting how LLMs influence us, right? The opposite happened to me: I loved using em dashes, but AI ruined it for me.
andai•36m ago
I still love using emdashes, and people already thought I was a robot!

https://xkcd.com/3126/

Soon the Andy 3000 will finally be a reality...

thadt•4m ago
I used to love using em dashes.

I still do - but I used to, too.

jcynix•1h ago
You might see certain phrases and mdashes ;-) rather often, because … these programs are trained on data written by people (or Microsoft's spelling correction) which overused them in the last n years? So what should these poor LLMs generate instead?
simonjgreen•51m ago
I see it from GPT5 too a lot
Fnoord•49m ago
> Speaking of smoking guns

Oh shoot! A shooting.

So the TL;DR of this post is: don't change this setting unless you know what you're doing.

kevin_thibedeau•43m ago
Chastise it with a reminder that you're using smokeless powder.
observationist•30m ago
Or the "Eureka! That's not just a smoking gun, it's a classic case of LLMspeak."

Grok, ChatGPT, and Claude all have these tics, and even the pro versions will use their signature phrases multiple times in an answer. I have to wonder if it's deliberate, to make detecting AI easier?

nurettin•28m ago
At this I'm just so glad that "you're absolutely right!" phase is over.
layer8•21m ago
Yes, it’s kind of a corpus delicti. ;)
jcims•18m ago
I'm working on a little SRE agent to pre-load tickets with information to help our on-call and I'm already tired of Claude finding 'smoking guns'.
HPsquared•14m ago
They love clichés, and hate repeating the same words for something (repetition penalty) so they'll say something like "cause" then it's a "smoking gun" then it's something else
ycombinatrix•1h ago
You can also use TCP_CORK to reduce the number of packets without any increased latency.

Disabling TCP_NODELAY would also reduce number of packets + be portable & simpler to implement - but would incur a latency penalty.

eieio•1h ago
Oh wow - I've never heard of TCP_CORK before. Without disabling pings I'd still pay the cost of receiving way more packets, but maybe that'd be tolerable if I didn't have to send so many pongs. This is super handy; excited to play around with it.

I am aware of TCP_NODELAY (funny enough I recently posted about TCP_NODELAY to HN[1] when I was thinking about it for the same game that I wrote about here). But I think the latency hit from disabling it just doesn't work for me.

[1] https://news.ycombinator.com/item?id=46359120

cheschire•1h ago
I enjoyed this write up as it touched on several topics I enjoy reading about.

Also I was unfamiliar with SSH being vulnerable in the past to keystroke timing!

pixl97•1h ago
https://news.ycombinator.com/item?id=37307708

2023 discussion about it here.

dathinab•1h ago
> Keystroke obfuscation can be disabled client-side.

please never do that (in production)

if anyone half way serious tries they _will_ be able to break you encryption end find what you typed

this isn't a hypothetical niche case obfuscation mechanism, it's a people broke SSH then a fix was found case. I don't even know why you can disable it tbh.

lazypenguin•1h ago
They literally explain the mechanism in the post and then explain why the security tradeoff made sense for their ssh game………
shadowgovt•1h ago
But they'd have to be on the same network as me to do that attack, right?
benlivengood•14m ago
Yep, like ECHELON and friends are. The metadata recorded about your (all of our) traffic is probably enough to perform the timing attack.
advisedwang•1h ago
That doesn't sound right to me. This obfuscation isn't about a side-channel on a crypto implementation, this is about literally when your keystrokes happen. In the right circumstances, keystroke timing can reduce the search space for bruteforcing a password [1] but it's overstating to describe that as broken encryption.

[1] https://people.eecs.berkeley.edu/~daw/papers/ssh-use01.pdf

eikenberry•55m ago
It is to prevent timing attacks but there are many ssh use cases where it is 100% computer to computer communications where there is no key based timing attack possible.
OneDeuxTriSeiGo•42m ago
There is an argument that if:

- you are listening to an SSH session between devices

- and you know what protocol is being talked over the connection (i.e. what they are talking about)

- and the protocol is reasonably predictable

then you gain enough information about the plaintext to start extracting information about the cipher and keys.

It's a non-trivial attack by all means but it's totally feasible. Especially if there's some amount of observable state about the participants being leaked by a third party source (i.e. other services hosted by the participants involved in the same protocol).

eikenberry•13m ago
I agree it is more nuanced than a simple 'good for computer-to-computer' and 'bad for person-to-computer'. I'm sure there are cases where both are wrong but I don't think that necessarily changes that it makes a reasonable baseline heuristic.
Romario77•10m ago
this only works for manually typed text, not computer to computer communication where you can't deduce much from what is being "typed" as it's not typed but produced by a program to which every letter is the same and there is no different delay in sending some letters (as people have when typing by hand)
PhilipRoman•40m ago
I haven't given this more than 5 seconds of thought, but wouldn't it make sense to only enable the timing attack prevention for pseudo-terminal sessions (-t)?
simplicio•32m ago
The fix seems kind of crazy though, adding so much traffic overhead to every ssh session. I assume there's a reason they didn't go that route, but on a first pass seems weird they didn't just buffer password strokes to be sent in one packet, or just add some artificial timing jitter to each keystroke.
bot403•22m ago
I'm just guessing but this chaff sounds like it wouldn't actually change the latency or delivery of your actual keystrokes while buffering or jitter would.

So the "real" keystrokes are 100% the same but the fake ones which are never seen except as network packets are what is randomized.

It's actually really clever.

kevin_thibedeau•16m ago
SSH has no way of knowing when a password is being typed. It can happen any time within the session after SSH auth.
swiftcoder•1h ago
> Obviously forking go’s crypto library is a little scary, and I’m gonna have to do some thinking about how to maintain my little patch in a safe way

This should really be upstreamed as an option on the ssh library. Its good to default to sending chaff in untrusted environments, but there are plenty of places where we might as well save the bandwidth

eikenberry•59m ago
+1... Given how much SSH is used for computer-to-computer communication it seems like there really should be a way to disable this when it isn't necessary.
jacquesm•17m ago
In practice I've never felt this was an issue. But I can see how with extremely low bandwidth devices it might be, for instance LoRa over a 40 km link into some embedded device.
mkj•5m ago
It looks like it is only applied for PTY sessions, which most computer-computer connections wouldn't be using.

https://github.com/openssh/openssh-portable/blob/d7950aca8ea...

BoppreH•15m ago
Yes, but I wouldn't be surprised if the change is rejected. The crypto library is very opinionated, you're also not allowed to configure the order of TLS cipher suites, for example.
pixl97•1h ago
>very confidently told me that my tcpdump output was normal ssh behavior:

I mean, for modern version of Openssh it's not exactly wrong. The failure was to tell you why that is the normal behavior.

raggi•1h ago
> I am working on a high-performance game that runs over ssh.

WAT. Please no.

shitter•1h ago
Why not? If it's high-performance, it's fine.
pseidemann•1h ago
Performing with highly elevated privileges? (Joke)
jabedude•1h ago
ssh the protocol doesn't imply any privileges of any kind
zamadatix•1h ago
Very interesting, I hadn't heard of this obfuscation before so it was well worth clicking.

Another good trick for debugging ssh's exact behavior is patching in "None" cipher support for your test environment. It's about the same work as trying to set up a proxy but lets you see the raw content of the packets like it was telnet.

For terminal games where security does not matter but performance and scale does, just offering telnet in the first place can also be worth consideration.

Veserv•1h ago
The really mysterious part is how ~10,000 packets per second costs ~20% of a core. That would mean SSH is bottlenecking in its code at ~500 Mbps per core (assuming full packets) which is ludicrously slow. It is trivial to do 10x that packet per second rate. Is SSH really that poorly designed?
diath•34m ago
> It is trivial to do 10x that packet per second rate.

When making this statement, are you taking into account that SSH encrypts the traffic by default?

Veserv•11m ago
I do not know where people get the idea that encryption is that slow. Standard AES hardware acceleration instructions do ~25 Gbps per core (on a 2023 CPU) which is ~50x that rate [1]. I have heard modern cores can do ~40-50 Gbps, but I have not been able to find any independent benchmarks of that. Even the Intel i5-2500, a CPU from 2011, averages ~10 Gbps which is ~20x that rate. Even unaccelerated encryption can do ~2-5 Gbps in pure software which is 4-10x the SSH rate.

And in this situation, the amount of encrypted payload in each packet is 36 bytes which is ~40x less than a full packet of ~1500 bytes. You would almost surely hit packet per second limits before you hit payload throughput limits at these small sizes.

Encryption is slow when compared to data throughput you can get with a properly designed transport stack, but that is because it is in comparison to 100 Gbps per core even with no hardware offload. Anything less than ~10 Gbps/1 million packets per second (ignoring other bottlenecks, so only the software transport is the limit) is not merely unoptimized, it is pessimized.

[1] https://calomel.org/aesni_ssl_performance.html

svnt•1h ago
> I am working on a high-performance game that runs over ssh.

Found your problem.

But it is an interesting world where you can casually burrow into a crypto library and disable important security features more easily than selecting the right network layer solution.

ycombinatrix•1h ago
Yea UDP is technically more performant, but then you need a crypto layer + reliable message delivery layer + bespoke client. Using a plain old SSH client is cool.

However, there are existing libraries for exactly this use case - see https://github.com/ValveSoftware/GameNetworkingSockets

I guess QUIC libraries would also work.

convolvatron•1h ago
its not really a question of 'udp performs better'. in tcp we have to live to head-of-line blocking on losses and congestion control. if you don't care about receiving every packet, but only the most recent, then udp is a good choice.

running without congestion control means that you avoid slowstart. but at a certain rate you run into poorly defined 'fairness' issues where you can easily negatively impact other flows. past that point, you can actually self-interfere and cause excessive losses for yourself.

quic uses congestion control, but uses latency estimates and variance as a signal to back off. it still imposes an ordering on a per-stream basis. so it might not be ideal either.

sctp has a mode which supports reliable and unordered, which might be something to consider

so really - if you care about latency and have a different reliability model, its worth unpacking all these considerations and using them to select your transport layer or even consider writing a minimal one yourself

ycombinatrix•1h ago
>in tcp we have to live to head-of-line blocking on losses and congestion control.

Is this not a performance consideration?

Either way, using plain old SSH means a metric bajillion computers have a client for your game built in.

eieio•1h ago
the obtuseness is the point! This is true of a lot of my work[1][2][3].

The problems you run into when doing things you shouldn't do are often really fun.

[1] https://news.ycombinator.com/item?id=42342382

[2] https://news.ycombinator.com/item?id=37810144

[3] https://news.ycombinator.com/item?id=42674116

PaulHoule•1h ago
I find it disturbing.

One thing you notice if you have ADSL is that some services are built as if slower connections matter and others are not. Like Google's voice and audio chat services work poorly but most of the others work well. Uploading images to Mastodon, Bluesky, Facebook, LinkedIn, Instagram and Nextdoor is reliable, but for Tumblr you have to try it twice. I don't what they are doing wrong but they are doing something wrong and not finding out what they're doing wrong because they're not testing and they're not listening to users.

Nobody consulted me about their decision not to run fiber by my house. If some committee decides to make ssh bloated they are, together with the others, conspiring to steal my livelihood and I think it would be fair for me to sue them for the $50k it would take to run that fiber myself.

It's OK if you work for Google where there is limitless dark fiber but what about people in African countries?

It's the typical corporate attitude where latency never matters: Adobe thinks it is totally normal that it takes 1-5s for a keystroke to appear when you are typing into Dreamweaver.

starttoaster•51m ago
There's a good chance you have other options. Regardless of how you feel about the company's head, Starlink would probably be one of them, with likely better performance than you're dealing with on ADSL.

But you cannot just sue a company because their network connected software doesn't work well on slow networks. Let alone a project like OpenSSH. It would be like me suing a game studio because my PC doesn't meet their listed minimum requirements to play the game.

PaulHoule•7m ago
Hey, it is one thing to buy a new computer, it is another thing to ask people to move. A better analogy is a bank redlining neighborhoods.
Refreeze5224•50m ago
You're not ok with a security/privacy tool using defensive techniques because of ... the lack of fiber in Africa?
PaulHoule•4m ago
My backyard but people will take Africa more seriously than anywhere in the US 2 miles from the end of cable.
gucci-on-fleek•42m ago
I agree with your general point that most companies/projects do a terrible job optimizing for slow computers/networks, but OpenSSH is from the OpenBSD people, who are well-known for supporting ancient hardware [0]. Picking a random architecture, they fully support a system with only 64MB of memory [1], and the base install includes SSH. So I suspect that OpenSSH is fairly well tested on crappy computers/networks.

[0]: https://www.openbsd.org/plat.html

[1]: https://www.openbsd.org/landisk.html#hardware

lokar•27m ago
The openssh team does not owe you anything.

If you want a “1990s” mode, add it yourself or pay some to do it for you.

bergen•23m ago
You just opened a huge nostalgia portal, never thought that Dreamweaver would still be around, I used that somewhere around 2003 I believe. Good memories
PaulHoule•4m ago
Frankly I wish there was an HTML editor that delivers on what it promised. I mean, markdown is almost as rife with edge cases as YAML and somehow the link syntax still eludes me. If we could “just” template by merging at the DOM level and had decent HTML editors the world would be a different place. But yeah, Adobe probably thinks Dreamweaver isn’t worth maintaining just as they seem to think Photoshop is barely worth maintaining (they keep adding AI features that sorta work but the foundations seem to be much worse than Illustrator)
layer8•13m ago
> One thing you notice if you have ADSL

This is funny to me, because ADSL used to be the fast thing, as opposed to dialup modems.

idontwantthis•54m ago
If security doesn’t matter then why not use telnet or something else besides ssh instead of forking a security library?
layer8•7m ago
Telnet nowadays typically isn’t available by default for security reasons, and OP wants people to be able to play the game just by typing “ssh thegamehost”.
kenmacd•51m ago
@eieio: whatever email protection you're running is triggering on the extension info. For example I see:

> And they’re sent to servers that advertise the availability of the [email protected] extension. What if we just…don’t advertise [email protected]?

eieio•45m ago
Is it possible that this is on your end?

The extension is "ping@openssh.com." It shows up in the blog reliably for me across several browsers and devices.

wizzwizz4•4m ago
No, it's Cloudflare munging the HTML. Cloudflare then provides JavaScript to un-munge it, but that's not reliable.
Animats•8m ago
In 2023, ssh added keystroke timing obfuscation. The idea is that the speed at which you type different letters betrays some information about which letters you’re typing. So ssh sends lots of “chaff” packets along with your keystrokes to make it hard for an attacker to determine when you’re actually entering keys.

Now that's solving the problem the wrong way. If you really want that, send all typed characters at 50ms intervals, to bound the timing resolution.

davidhyde•4m ago
I wonder if this is the same reason why Microsoft's Remote SSH plugin on VS Code is so flaky even with a decent internet connection. Every couple of months I try to give it another go and give up due to the poor keyboard latency I inevitably experience. And the slow reconnects whenever I glance away from my computer monitor briefly. This is on a fiber connection with a 20ms ping to the remote machine.
JohnLeitch•54s ago
The reliance on LLMs is unfortunate. I bet this mystery could gave been solved much quicker by simply looking at the packet capture in Wireshark. The Wireshark dissectors are quite mature, SSH is covered fairly well.

Scaling PostgreSQL to power 800M ChatGPT users

https://openai.com/index/scaling-postgresql/
1•mustaphah•1m ago•0 comments

Microsoft Slop 365

https://www.youtube.com/watch?v=BuUDhHuc7Rg
1•flockyflock•2m ago•0 comments

Show HN: Presidents Flights – map of world leaders' trips

http://presidentsflights.com/
1•Neodelf•2m ago•0 comments

Beyond Vendor Lock-In – A Framework for LLM Sovereignty

https://nezhar.com/blog/llm-sovereignty-framework/
1•nezhar•2m ago•0 comments

Show HN: Website template for iOS apps

https://appview.dev
1•harmash•3m ago•0 comments

Capital One to buy fintech firm Brex in $5.15B deal

https://www.reuters.com/legal/transactional/capital-one-buy-fintech-firm-brex-515-billion-deal-20...
1•personjerry•3m ago•0 comments

Capital One Strikes $5.15B Deal for Fintech Brex

https://www.wsj.com/finance/banking/capital-one-to-buy-fintech-brex-e2dacc67
1•cj•3m ago•0 comments

Show HN: pgedge-loadgen for realistic open-source PostgreSQL workload simulation

https://github.com/pgEdge/pgedge-loadgen
2•pgedge_postgres•4m ago•0 comments

White House Posts Digitally Altered Image of Woman Arrested After ICE Protest

https://www.theguardian.com/us-news/2026/jan/22/white-house-ice-protest-arrest-altered-image
12•TigerUniversity•5m ago•0 comments

Iran's 14-Day Blackout: The New, Silent Reality

https://mastodon.social/@netblocks/115939719155570618
2•us321•5m ago•0 comments

Google study finds DeepSeek, Alibaba models mimic human collective intelligence

https://www.scmp.com/tech/tech-trends/article/3340690/google-study-finds-deepseek-alibaba-ai-mode...
1•maxloh•6m ago•1 comments

Show HN: Human vs. AI Tetris Arena

https://tetrisbench.com/
1•ykhli•6m ago•0 comments

Lisp Coding Standards v1.0

https://franz.com/~jkf/coding_standards.html
1•swatson741•7m ago•0 comments

Noahpinion on Fertility – Let's Save the Human Species

https://www.noahpinion.blog/p/lets-save-the-human-species
1•A_D_E_P_T•7m ago•0 comments

Rewards Are Everywhere

https://agenticposts.substack.com/p/rewards-are-everywhere
1•brandonb•8m ago•0 comments

Ask HN: Which Wireless Router in 2026

1•helij•9m ago•0 comments

To flexibly organize thought, the brain makes use of space

https://news.mit.edu/2026/to-flexibly-organize-thought-the-brain-makes-use-of-space-0120
2•gmays•9m ago•0 comments

GM to Move Production of Buick SUV from China to US

https://www.reuters.com/business/autos-transportation/gm-bring-china-built-buick-us-2026-01-22/
1•tokyobreakfast•11m ago•0 comments

Social rank and social environment combinedly affect REM sleep in mice

https://www.nature.com/articles/s41598-025-32402-2
1•PaulHoule•11m ago•0 comments

Building a vintage Macintosh web server [video]

https://www.youtube.com/watch?v=XkLFHN273zk
1•geerlingguy•12m ago•0 comments

Beginner's Guide to Vibetrading

https://blog.everstrike.io/beginners-guide-to-vibetrading/
1•mo3rew4r•12m ago•0 comments

Ask HN: How do you keep hardware requirements "verified" throughout development?

3•joshguggenheim•12m ago•0 comments

Moderna curbing investments in vaccine trials due to US backlash

https://www.reuters.com/business/healthcare-pharmaceuticals/moderna-curbing-investments-vaccine-t...
2•_DeadFred_•12m ago•0 comments

Vnccc: VNC Wrapped Claude Code

https://github.com/pgray/vnccc
2•pgray•12m ago•1 comments

Exoplanets Beyond the Conservative Habitable Zone. I. Habitability – IOPscience

https://iopscience.iop.org/article/10.3847/1538-4357/ae21d7
1•rbanffy•14m ago•0 comments

Engineer at Elon Musk's XAI Departs After Spilling the Beans in Interview

https://gizmodo.com/engineer-at-elon-musks-xai-departs-after-spilling-the-beans-in-podcast-interv...
1•rbanffy•17m ago•0 comments

We got an AI agent to read a config file and email it to an external address

1•exordex•17m ago•0 comments

SSDs now cost 16x more than HDDs due to AI supply chain crisis

https://www.tomshardware.com/pc-components/storage/ssds-now-cost-16x-more-than-hdds-hybrid-ssd-hd...
2•rbanffy•17m ago•0 comments

2.8 Days to Disaster: Low Earth Orbit Could Collapse Without Warning

https://scitechdaily.com/2-8-days-to-disaster-low-earth-orbit-could-collapse-without-warning/
3•g-b-r•17m ago•0 comments

The Conditionally Open Web

https://www.coryd.dev/posts/2026/the-conditionally-open-web
1•cdrnsf•17m ago•0 comments