I was already building the vast majority of it from source because I enable CPU optimizations for the specific microarchitecture in the machine (nixpkgs.hostPlatform.gcc.arch and nixpkgs.hostPlatform.gcc.tune), so once I learned about the risk of supply chain attack on the nix cache, disabling it entirely was a pretty small change.

So far, I'd say the biggest negative (aside from the build times that I was already experiencing due to the optimizations) is that GNU savannah will temporarily IP ban you when you download too much from them. For example, building the grub that is used for the ISOs downloads like 70+ patches from GNU Savannah which frequently triggers the IP ban.