All started with 4 million requests / day on rocketgit.com (AGPL git hosting). Instead of thousands. Enough is enough!
Presentation (8 pages): https://rocketgit.com/user/catalinux/ai-drop/artefacts/downl...
So, I built AI-drop free-software project (AGPL) to block this kind of crawlers. rocketgit.com has a very restrictive robots.txt (only around 10 pages are allowed). But the crawlers are not respecting it. So, who is not respecting it will be blocked.
Limiting by IP or user-agent is not working anymore. The crawlers are using millions of IPs and it takes days to reuse an IP. And they use fake user-agents.
I am using honeypots to build a list of misbehaving IPs which are streamed to downstream instances. Right now I have around 1.573.000 IPv4 and 6.000 IPv6 misbehaving addresses. I keep them in the list only for 7 days to not block somebody legit. I am watching the traffic on rocketgit.com and the solution is working nicely.
It uses an eBPF program to fast drop the packets.
You can try it easily on your Linux servers:
# Check https://rocketgit.com/user/catalinux/ai-drop/pkg # If your distribution is already there, follow the instructions. # It is the best way to keep you up-to-date.
# Look inside 'ai-drop.spec' or 'debian/control' and install the dependencies. git clone https://rocketgit.com/user/catalinux/ai-drop cd ai-drop ./configure make install # edit /etc/ai-drop.conf # Very important: # set 'interface' (non 'eth' are not supported now) # set 'safe_mode = 0' if you only want to evaluate the product.
Then:
systemctl enable --now ai-drop journalctl -f -u ai-drop
# The list of misbehaving IPs will be streamed from the main server.
# Look at the statistics, especially 'drops': ai-drop dump-stats # Output example: # now=1794482762 ipv4_total=0 ipv6_total=0 total=0 \ # ipv4_adds=0 ipv6_adds=0 adds=0 \ # ipv4_updates=0 ipv6_updates=0 updates=0 \ # ipv4_drops=0 ipv6_drops=0 drops=0 \ # ipv4_real_drops=0 ipv6_real_drops=0 real_drops=0 \ # ipv4_expires=0 ipv6_expires=0 expires=0 \ # ipv4_deletes=0 ipv6_deletes=0 deletes=0 \ # last_load=0 daemon_start=0
Please note that no AI was used to build this software.
Thank you for your time!
Catalin(ux) C.M.T. BOIE