But even a curl test on localhost was showing 1,2, or more seconds longer slowdown.s
After a lot of digging, realized that I was victim of some sort of DOS attack, which appeared to be a SYN flood attack.
In a normal SYN flood attack the SYN packets are sent from one IP address then never reply, leaving the server in a state with multiple connections stuck in the "SYN_RECVD" state.
In this attack, the SYN packets are actually being "sent" from multiple IPs - and one theory is that these are forged IPs, so the attack is really a reflection attack, where the attacker sends a forged IP SYN to my server, which replies (several times with SYN ACK)
I blocked the IP block /16 at the firewall and all was well, but this made me wonder:
How are modern OS's so vulnerable to this? On my macOS server, I could run netstat -anp tcp | grep '\.443 ' and see about 128 entries stuck in "SYN_RCVD" at which point the server just went to pieces.
In other words, if my server received 128 "SYN" packets, it would die for about 75 seconds.
Is this a macOS only problem? Are other OS's susceptible to this?
for this question, please consider "Use CloudFlare" off limits. I'm more interested in why modern OSs can be so fragile to a few (forged) TCP packets.