The only effective way I've found to combat "unwanted traffic" (bots, botnets, crawlers, search engines, etc) is to a) make request/responses as cheap as possible (heavy caching, rate limiting, etc) and b) measure cohorts, fingerprints and behaviour and block patterns that can't possibly look like "normal human traffic" (tricky to get right, but this is to a large extent what Cloudflare is doing I believe, and I've had great success with this technique and building out an OSS Caddy module for this...)
jackpriceburns•1h ago