HN removes a "how" if the post starts with it, sometimes making it hilarious.

Big question is which feature subset you want to replicate.

Kubernetes means everything to everyone. At its core, I think it’s being able to read/write distributed state (which doesn’t need to be etcd) and being able for all the components (especially container hosts) to follow said state. But the ecosystem has expanded significantly beyond that.

IMO this is what keeps people from building systems that might challenge kubernetes. Everyone wants to say Kuberentes is too complex, so we built something that does much less. I respect that! But I think it usually fails to grok what Kubernetes is and why it's such an interesting and vital rallying point, that's so thoroughly captured our systems-making. Let's look at the premise:

> That’s why I like to think of Kubernetes as a runtime for declarative infrastructure with a type system.

You can go build a simple way to deploy containers or ship apps: but you are missing what I think allows Kubernetes to be such a big tent, thats a core useful platform for so many. Kubernetes works the same for all types, for everything you want to manage. It's the same desired state management + autonomic systems patterns, whatever you are doing. An extensible platform with a very simple common core.

There are other takes and other tries, but managing desired state for any kind of type is a huge win that allows many people to find their own uses for kube, that is absolutely the cornerstone to it's popularity.

If you do want less, the one project I'd point to that is kubernetes without the kubernetes complexity is KCP. It's just the control plane. It doesn't do anything at all. This to me is much simpler. It's not finding a narrowly defined use case to focus on, it's distilling out the general system into it's simplest parts. Rebuilding a good simple bespoke app container launching platform around KCP would be doable, and maintain the overarching principles that make Kube actually interesting.

I seriously think there is something deeply rotten with our striving for simplicity. I know we've all been burned, and there's so often we want to throw up our hands, and I get it. But the way out is through. I'd rather dance the dance & try to scout for better further futures, than reject & try to walk back.

Everything in infrastructure is a set of trade-offs that work in both directions.

If you want better monitoring, metrics, availability, orchestration, logging, and so on, you pay for it with time, money, and complexity.

If you can't justify that cost, you're free to use simpler tools.

Just because everyone sets up a Kubernetes / Prometheus / ELK stack to host a web app that would happily run on a single VPS doesn't mean you need to do the same, or that nowadays this is the baseline for running something.

Of course things can be simpler.

Remove abstractions like CNI, CRI, just make these things built-in.

Remove unnecessary things like Ingress, etc, you can always just deploy nginx or whatever reverse proxy directly. Also probably remove persistent volumes, they add a lot of complexity.

Use some automatically working database, not separate etcd installation.

Get rid of control plane. Every node should be both control plane and worker node. Or may be 3 worker nodes should be control plane, whatever, deployer should not think about it.

Add stuff that everyone needs. Centralised log storage, centralised metric scrapping and storage, some simple web UI, central authentication. It's reimplemented in every Kubernetes cluster.

The problem is that it won't be serious enough and people will choose Kubernetes over simpler solutions.

> What's the really simple K8s?

It's k3s. You drop a single binary onto the node, run it, and you have a fully functional one-node k8s cluster.

> What's the really simple K8s?

I think K8s couples two concepts: the declarative-style cluster management, and infrastructure + container orchestration. Keep CRDs, remove everything else, and implement the business-specific stuff on top of the CRD-only layer.

This would give something like DBus, except cluster-wide, with declarative features. Then, container orchestration would be an application you install on top of that.

Edit: I see a sibling mentioned KCP. I’ve never heard of it before, but I think that’s probably exactly what I’d like.

In ascending order of functionality and how much complexity you need:

  - Docker Compose running on a single server
  - Docker Swarm cluster (typically multiple nodes, can be one)
  - Hashicorp Nomad or K3s or other light Kubernetes distros

One of the issues is that the open source Helm charts (or whatever) for something like Grafana do not come out-of-the-box with good config. I spent a significant amount of time a little while ago reading blogs to get Grafana to use up to date indexing algorithms, and better settings, etc.

Considering these companies make money when you use their hosted solution, this is not surprising, and it just goes to show TANSTAFL.

It would have helped if you tell us why you don’t like this approach.

I feel like the author has a good grasp of the Kubernetes design... What about the approach is problematic? And why don't you think that is how Kubernetes was designed to be used?

But then you need two different provisioning tools, one for infra in k8s, and one for infra outside k8s. Or perhaps using non-native tools or wrappers.

Kubernetes is explicitly designed to do what the article describes. In that respect the article is just describing what you can find in the standard Kubernetes docs.

> it is super important to treat a container orchestrator a container orchestrator.

Which products do you think are only “container orchestrators”? Even Docker Compose is designed to achieve a desired state from a declarative infrastructure definition.

If you're not Google, please for the love of god, please consider just launching a monolith and database on a Linux box (or two) in the corner and see how beautifully simple life can be.

You can literally get a Linux box (or two) in the corner and run:

  curl -sfL https://get.k3s.io | sh -
  cat <<EOF | kubectl apply -f -
  ...(json/yaml here)
  EOF

How am I installing a monolith and a database on this Linux box without Kubernetes? Be specific, just show the commands for me to run. Kubernetes that will work for ~anything. HNers spend more tokens complaining about the complexity than it takes to setup.

The mental gymnastics required to express oneself in yaml, rather than, say, literally anything else

Like, brainfuck? Like bash? Like Terraform HCL puppet chef ansible pile-o-scripts? The effort required to output your desired infrastructure's definition as JSON shouldn't really be that gargantuan. You express yourself in anything else but it can't be dumped to JSON?

Sometimes the interfaces are clear and consistent (loadbalancers, for example) and the line between, e.g., an AWS NLB and your ingress gateway, is clear-cut.

Other times there is a significant degree of portability pains and sparse feature matrices (CSI, IME).

This more direct interaction by using kubectl happens on a development cluster? Or is it on the same that runs production workloads?

When you say infrastructure in this context, are you referring to the actual K8s cluster infra?

Or is it what I tend to call "intra-cluster infra" - DBs / Prometheus / Kafka etc. Infra that support apps?

Writing go in yaml and forgetting everything else we learned software engineering. Proper ide's, being able to make abstractions, not copy pasting, structured templating and thus not string based templating, should I go on?

Well, a group of processes... With some separation between them. Like you can't listen on localhost and call from another pod. But with processes you can do that. Same thing with UDS.