I generally agree with you, though I want to comment on this line:
> Code that you don't touch doesn't suddenly change its behaviour one day.
This can and does happen all the time, when the platforms and abstractions your code builds on change underneath you. This is why a compelling environment / dependency management story is so important. "Code rot" is real. =P
I thought about this, but technically that is still the code changing, it just happens to be in your dependencies rather than your codebase. The only reason you really have to change dependency versions is security fixes, and they should be infrequent enough that you could do manual testing. So I don't think it's a compelling reason to write unit tests, although it is certainly an added value.
- Stretch of freeway by my old house that looks and feels like any freeway you've ever been on, except they limit the speed to 45mph. Try to go 45 at your own peril as car and semi alike go 65 past you. Then the cop camps there. I avoid that freeway now.
- Shopping center exit onto a small side street had a "no left turn" sign, but trying to obey this meant a massive detour around if you needed to go left. The side street was empty most of the time (and low speed!), so people tended to just go for it. Cop camped out there. They removed the sign a year later.
My company has deployed Kubeflow for production model training. Early on, we used their big deployment, but we got frustrated trying to manage it with kfctl, so we started using kustomize directly and deploying only what we need, like KFP. So no istio for us! YMMV with your use case.
Yeah my plan is to use it the same way we use kubernetes: everything via kubernetes configs/justo use is friends and nobody using CLI’s, which I’m convinced are the text version of “click ops” hahaha.
So Istio isn’t strictly required? Can I ask which components you deploy? At minimum I’m planning on just the deployment and serving components and just use straight Polyaxon for training.
Yeah, there are quite a few applications that actively discourage the use of anything other than their CLI, which I find profoundly bizarre. Istio had a big warning in their docs that their Helm charts were deprecated, even though istioctl uses them under the hood. Funny that Kubeflow has its own Istio deployment, and they make you use kfctl to deploy it, haha.
It's not strictly required, but might be for you, since A/B deployment actually is a thing Istio does. We only use Kubeflow Pipelines currently; looking into Katib.
My feeling about Kubeflow is that it's a package of a lot of things that exist, with nice things on top, and an easy way to deploy those things. Only, it ends up not being that easy, and not easy at all to configure, and various features of the underlying tools are hard to get to or completely unavailable.
If you deployed your own Istio, you'd understand it end-to-end and could solve problems with it when it goes south, and you'd even understand what exactly you're using it for and why. The biggest problem I have with kfctl is that it basically asks for system:masters privilege to install everything and then you get to figure out what it did on your own. I don't want Istio, Cert-Manager, Knative, Argo, etc. deployments that I don't understand and can't easily configure because they're buried 5 levels deep in Kustomize overlays. These are all things I can install from elsewhere and with more documentation.
Kubeflow Pipelines is still a little mysterious, but the footprint is much smaller and not as invasive.
> Code that you don't touch doesn't suddenly change its behaviour one day.
This can and does happen all the time, when the platforms and abstractions your code builds on change underneath you. This is why a compelling environment / dependency management story is so important. "Code rot" is real. =P