hi friends, Flux maintainer here. Stefan gave this talk at FluxCon EU

He maps out some "levels of detachment" for software & platform engineering that we've all been watching happen. Copilot autocomplete -> chat interfaces -> spec-driven development -≥ agent teams -≥ fully autonomous dark factories. Same progression applies to ops: agents that go into your clusters, diagnose issues, and present resolution plans without human intervention.

GitOps still matters in this world of agents: 1. Git as the interface to production (agents open PRs instead of running kubectl directly) 2. Git as persistent memory (incident RCA gets committed so the next similar event resolves faster) 3. Desired state that includes agent definitions (recover your AI orchestration the same way you recover a cluster: point Flux at a repo)

He also announced fluxcd/agent-skills, which are OCI packaged, verifiable skills for Flux that should progressively discovery instead of loading everything into context. We've seen a 32% improvement on knowledge tasks and 29% on repo audits vs. baseline models running against real public repos.

Stefan's anecdotes on developing agent tooling are relevant beyond Flux. Every harness defines agents differently (markdown, TOML, JSON), there's no portability, and reproducibility and measurement is expensive and time-consuming.

I think the team who runs this infra will be around KubeCon and FluxCon next week -- will try to ask

This is an awesome writeup of the tools and culture issues you run into maintaining dev environments.

From post, the problems that justified central dev boxes are roughly: 1. dependency / config mgmt / env drift on laptops 2. collaboration / debugging between engineers 3. compute scaling + optimization 4. supporting devs with updates and infra changes

The last one is particularly interesting to me, because supporting the dev env is separate engineering role/task that starts small and grows into teams of engineers supporting the environment.

I'm helping build Flox. We're working on these pain points by making environments (deps, vars, services, and builds) workable across all kinds of Mac/Linux laptops and servers. 1) a. Virtualize the pkg manager per-project b. Nix packages can install across OS/arch pretty well 2) Imperative actions like `flox install`/`upgrade` always edit a declarative env manifest.toml -- share it via git 3) less Docker VM's -- get more out of devteam Macbooks 4) reduce toil with a versioned, shareable envs --> less sending ad-hoc config and brew commands to people (as mentioned in the post.) Just `git pull && flox activate`.

I think on problem point #2, collab tools are advancing to where, pairing on features, bugs, and env issues can be done without central SSH. (ex: tmate, vscode liveshare, screensharing, etc) -- however, that does sort of fall apart on laptops for async debugging of env issues (ex: when devprod is in the US, and eng is in London). Having universal telemetry on ephemeral cloud dev-boxes with a registry and all of the other DNS and SSH goodies could be the kind of infra to aspire to as your small teams run into more big-team problems.

In the Stripe anecdote, adopting the centralized infra created new challenges that their devprod teams were dedicated to supporting: - international latency from central, US-based VM's - syncing code to the dev boxes (https://facebook.github.io/watchman/) - linting, formatting, generating configs (run it locally or serverside?) - a dev workflow CLI tool dedicated to dev-box workflows and sync'ing with watchman's clock - IaaS, registry, config, glue for all the servers

This is all very non-trivial work, but maybe there's a future where people can win some portability with Flox when they are small and grow into those new challenges when it's truly needed -- now their laptop environments just get a quick `flox activate` on some new, shiny servers or Cloud IDE's.

I really like the notes from the author on how useing Language Server Protocol across a high latency link has great optimizations that work along side the watchman sync for real-time code editing.

That's a huge win -- has your team written about or spoke on this anywhere?

No but I'd be happy to (I maintained the docker-compose stack, our CLI, and did the transition to Nix).

I'm curious about the # of svc's / stack / company / team size -- if you have your own blog -- would love to read it when you publish

could be a cool lightning talk (or part of something longer)

maybe it's a good piece for https://nixinthewild.com/ ?

I'm @capileigh on twitter and hachyderm.io if you wanna reach out separately -- here is good tho too

Hey, I plan to reach out when I get some time :)

I can see there’s interest in the topic.

sounds good! it's a cool anecdote :)

Hi Jason! Like many others here I'm looking forward to that blog post! :-)

For now, could you elaborate on what exactly you mean by transitioning from docker-compose to Nix? Did you start using systemd to orchestrate services? Were you still using Docker containers? If so, did you build the images with Nix? Etc.

When we used docker-compose we had a CLI tool which developers put in their PATH which was able to start/stop/restart services using the regular compose commands. This didn’t accomplish much at the time other than being easy to remember and not requiring folks to know where their docker-compose files were located. It also took care of layering in other compose files for overriding variables or service definitions.

Short version of the Nix transition: the CLI tool would instead start services using nix-shell invocations behind pm2. So devs still had a way to start services from anywhere, get logs or process status with a command… but every app was running 100% natively.

At the time I was there, containers weren’t used in production (they were doing “App” deploys still) so there was no Docker target that was necessary/useful outside of the development environment.

Besides the performance benefit, microservices owning their development environment in-repo (instead of in another repo where the compose configs were defined) was a huge win.

Thanks for elaborating!

By pm2 you mean https://www.npmjs.com/package/pm2 ?

Yep!

the pm2 thing via a custom cli is interesting

several nixy devtools do some process management now

something we're trying in Flox is per-project services run /w process-compose. they automatically shutdown when all your activated shells exit, and it feels really cool

I’d like to learn more about switching compose to nix. We will hit a wall with compose at some point.