I just burned hours trying to figure out why basic D1 SQL queries against a 100MB database are taking 20-30 seconds wall time. Turns out there's an ongoing D1 issue, but it's only being acknowledged in Cloudflare's D1 Discord channel, while cloudflarestatus.com says everything is a-ok.
Posting here in case it saves someone else the debugging rabbit hole.
So after having used GCP, AWS, and all the orchestration baggage that came with it, developing and testing user-facing apps with Miniflare, workerd, D1, and wrangler has been a breeze... until I discovered some of my D1 calls take 20 seconds to resolve on a table with 10 rows. Query takes sub-millisecond.
Status page looking good, all systems operational. Digging through issues, pulling my hair out. Then stumbled upon this post to discover there is indeed an issue, and it is acknowledged on Discord... Discord? Why?
Honestly, I've been the fool here. Discouraged every single soul I know from vendor lock-in, only to vendor lock-in myself because I really liked the product.
Cloudflare is in a great position to dominate the ecosystem. They understand our needs. However, feature creep, half-baked releases, and improper communication don't give me the confidence to continue using their services at scale.
Incident is marked as resolved... tracing outliers tell a different story. Guess it's time to move on. Can't claim you offer a reliable database at scale if a trivial lookup on a 10-row table resolves in 20 seconds.
We do have an ongoing issue that is causing some customers to experience elevated long tail latencies. These are generally occurring above p999 and are usually on the order of single digit seconds. Queries taking 20+ seconds would be outside the norm and unexpected especially if this is happening more frequently.
Our team is working quickly to resolve it and we are expecting improvements to be released Monday. If you are seeing latencies that are substantially worse than this please drop me an email with your account and database identifiers byule at cloudflare
eis•2w ago
Our company opened an issue but we have not heard back in 3 days. The issue has been going on for more than the 3 days, we've noticed it nearly a week ago.
You can also find people in the CF community forums or in other places complaining about D1 latency and networking issues going back as much as a year. Nobody from CF replies in their community forums.
I think this is 1. not an acceptable level of service 2. not acceptable level of support 3. very surprising for a company who is all about fast and reliable networking.
Unfortunately I experienced all kinds of problems with Cloudflare and their non-CDN services over the past couple of months and I've also never seen a major cloud service provider suffering from so many dashboard errors and bugs.
Cloudflare folks, if you read this: you need a Snow Leopard year. You've launched many services and products but they are riddled with bugs, performance issues and outright outages. The recent launch of your data warehouse offering which in our testing (and I think even in your presentation) showed data processing speeds measured in the kilobytes/s while not even supporting aggregations at launch would have been embarassing even 20 years ago.
Please please step up your quality game because you do have some interesting offerings with good potential. The two huge outages last year were just surfacing deeper issues that seem to permeate throughout the stack.