As a learning project, this is absolutely awesome.
I run Kubernetes via Kind on Docker on NixOS.
There's a ton of other ways to get a development environment on your NixOS developer PC.
I don't pretend this one is very good, I just copy what my colleagues have come up with (+ NixOS).
For production workloads, I wouldn't run the kubelet using this much custom wiring.
I'd run Talos. It's vastly simpler, you can run them in NixOS VMs, it's declarative and lowers the surface area of things that need interaction, no SSH'ing in.
It seems like the author is torn between where to put control: In NixOS, or in Kubernetes?
You can move stuff, e.g. CoreDNS, out of Kubernetes for a "simpler" setup.
But the point of running workloads inside Kubernetes is that you get redundancy between nodes.
So if a single node dies, your services don't die.
Embracing Kubernetes, I certainly haven't let go of NixOS. My personal servers still just run NixOS.
It's much simpler, much cheaper, and resilient in its own way.
Selling Kubernetes and Cloud Native users on using NixOS, I'd probably go another way, e.g. via dev environments.
You're right, it's very much a trade-off and preference where you put control, NixOS or Kubernetes. I'm not so much torn, but more believe you always have to weigh pros and cons.
For CoreDNS specifically, this setup adds CoreDNS to every node, and every node does DNS locally, so there's no redundancy benefit to using a Kubernetes deployment for CoreDNS. It does become a benefit as soon as you can't have a CoreDNS per node. I guess the obvious downsides to CoreDNS per node are that cache becomes very spread out in larger setups, and you may end up hammering your API server and upstream DNS servers more.
darkwater•2mo ago
anttiharju•2mo ago
darkwater•2mo ago
[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/applicatio...
clvx•2mo ago
darkwater•2mo ago