I agree with respect to admiring it from afar. I've gone through large chunks of the source many times and always have an appreciation for what it does and how it accomplishes it. It has a great, supportive community around it as well (if not a tiny bit proselytizing at times, which doesn't bother me really).
With all that said, while I have no "hate" for the stack, I still have no plans to migrate our container infrastructure to it now or in the foreseeable future. I say that precisely because I've seen the source, not in spite of it. The net ROI on subsuming that level of complexity for most application ecosystems just doesn't strike me as obvious.
Not to be rude, but K8s has had some very glaring issues, especially early on when the hype was at max.
* Its secrets management was terrible, and for awhile it stored them in plaintext in etcd.
* The learning curve was real and that's dangerous as there were no "best practice" guides or lessons learned. There are lots of horror stories of upgrades gone wrong, bugs, etc. Complexity leaves a greater chance of misconfiguration, which can cause security or stability problems.
* It was often redundant. If you're in the cloud, you already had load balancers, service discovery, etc.
* Upgrades were dangerous and painful in its early days.
* It initially had glaring third party tooling integration issues, which made monitoring or package management harder (and led to third party apps like Helm, etc).
A lot of these have been rectified, but a lot of us have been burned by the promise of a tool that google said was used internally, which was a bit of a lie as kubernetes was a rewrite of Borg.
Kubernetes is powerful, but you can do powerful in simple(r) ways, too. If it was truly "the most amazing" it would have been designed to be simple by default with as much complexity needed as everybody's deployments. It wasn't.
It's only useful for the degenerate "run lots of instances of webapp servers running slow interpreted languages" use case.
Trying to do anything else in it is madness.
And for the "webapp servers" use case they could have built something a thousand times simpler and more robust. Serving templated html ain't rocket science. (At least compared to e.g. running an OLAP database cluster.)
Yeah, they basically spent a shitload of effort developing their own cluster management platform that turns off all the Kubernetes functionality in Kubernetes.
Must be some artifact of hosting on Azure, because I can't imagine any other reason to do something this contorted.
