I thought I might've introduced a regression in the application layer, but the pattern I saw was strange. The latency would only spike on weekdays from around 8am-6pm PST i.e. peak hours. Note that the site is still a WIP and has 0 real-user traffic.
After some digging I've confirmed the issue isn't happening at the application layer, but rather at the network/Cloudflare level. When I check `mydomain.com/cdn-cgi/trace` during peak hours, the Cloudflare data centre processing the request is very far from both me and the origin server (e.g. LHR London). And then it changes to something closer during off-hours. I've also confirmed latencies returns to normal when I disable Cloudflare proxying.
Some Cloudflare community members have described that the issue is actually a business problem with ISP-Cloudflare peering agreements rather than a technical problem: https://community.cloudflare.com/t/very-slow-server-response-time/611853/3 . But they say the fix is to go from a free plan to a pro plan, and I've tried that but the latency still persists...
I'm not really sure where to go from here. Has anyone gone through anything similar before? Am I missing anything obvious? I would prefer not to remove Cloudflare, but if the user experience is at the whims of Cloudflare's ISP agreements (which seemed to be fine a month ago!), then I'm seriously considering to just handle security on my own.
dc396•5mo ago
Sounds like you're experiencing vagaries of somebody (maybe Cloudflare, maybe some other ISP Cloudflare is peering with) doing traffic engineering, probably to reduce congesting on particular paths. The recommendation to go with the Pro plan is likely just the first step, the next step is to open a ticket and get them to fix it -- that's what you're paying them for.
Dropping Cloudflare is, of course, an option as most of the security stuff they do can be handled by competent security folks, but you (may?) need to find someone similar if your site is at risk of DDoS.
hnroo99•5mo ago