Then... it wasn't. The more I read the less I ever want to see this again. The LLM train has got to end at some point.
But, the marketing heavily focuses on LLM stuff to the point of making everyone confused.
Then they completely abandoned not just the CUE frontend, but CUE altogether (while strenuously denying that they were doing so) for a GraphQL-based rewrite that focused on letting people use popular general-purpose languages to construct their workflows. The initial rollout of this was not feature complete and only supported imperative languages (Python and TypeScript, IIRC), which I didn't like.
Instead of porting everything over to all their new interfaces, I hopped off the train and rewrote all of our portable pipeline scripts in Nix, via Devenv. At the time, I'd never used Devenv before, but getting the work done that time still took maybe a tenth of the time or less. More than anything else, this was due to not having to fuck around with the additional overhead Docker entails (fussing with mount points, passing files from one stage to another, rebuilding images, setting up VMs... all of it). I got the reproducibility without the extra bullshit, and got to work with interfaces that have proven much more stable.
I still think there's a place for something like Dagger, focused just on CI, perhaps even still using Docker as a deployment/distribution strategy. But I no longer trust Dagger to execute on that. I think a proper external DSL (probably special-purposw but still Turing-complete, e.g., Nickel) is the right fit for this domain, and that it should support multiple means of achieving repeatability rather than just Docker (e.g., Nix on bare metal and Podman, to start). An option to work on bare metal via reproducible environment management tools like Nix, Guix, or Spack is a valuable alternative to burdensome approaches like containers.
I haven't looked at Dagger in several months, but the other big piece that is missing for portable CI workflows is a library that abstracts over popular CI platforms so you can easily configure pull/merge request pipelines without worrying about the implementation details like what environment variables each platform exposes to indicate source and target branch.
Idk anything about all the AI horseshit; I was off the Dagger bandwagon before they took that turn. I don't know if it's serious or a nominal play to court investors. But that kind of pivot is another reason not to build core infra on top of the work of startups imo. If the product is 70% of what you want, you have no way of knowing whether filling that 30% gap is something the maintainers will suddenly pivot away from, even if their current direction looks aligned with yours.
I'd recommend considering tools in this space only if (a) they're already close to 100% of what you need and (b) they're open-source. Maybe you can relax (a) if it's really easy to extend the codebase (I find this to be true for Devenv's Nix modules, for example.)
I currently manage my development environments via NixOS and Devenv, so if I could just keep that and achieve the same functionality, that sounds good to me.
Without the LLM bits, this is basically like Bazel or buck2, right?
Out of curiosity, would it be feasible to take a big cmake project and generate thousands of compile rules into dagger and use it as a substitute for make with sandboxing? I’ve never seen builkit used with such many nodes, how would it fare?
I ended up opting for CUE and GitHub Actions, and I'm glad I did as it made everything much, much simpler.
Here are a few links to whet your appetite:
- https://cue.dev/docs/getting-started-with-github-actions-cue...
- https://cue.dev/docs/drying-up-github-actions-workflows/
- https://cue.dev/docs/spotting-errors-earlier-github-actions-...
Definitely read through the CUE documentation (https://cuelang.org/docs/), watch their YouTube videos (https://www.youtube.com/@cuelang/videos), and join the community Slack channel (https://cuelang.org/community/). I've gotten a lot of help in the Slack from both enthusiastic community members and from the developers themselves whenever I've gotten stuck.
To some extent yes. If all you have is 2 GitHub Actions YAML files you are not going to reap massive benefits.
I'm a big fan of CUE myself. The benefits compound as you need to output more and more artifacts (= YAML config). Think of several k8s manifests, several GitHub Actions files, e.g. for building across several combinations of OSes, settings, etc.
CUE strikes a really nice balance between being primarily data description and not a Turing-complete language (e.g. cdk8s can get arbitrarily complex and abstract), reducing boilerplate (having you spell out the common bits once only, and each non-commit bit once only) and being type-safe (validation at build/export time, with native import of Go types, JSON schema and more).
They recently added an LSP which helps close the gap to other ecosystems. For example, cdk8s being TS means it naturally has fantastic IDE support, which CUE has been lacking in. CUE error messages can also be very verbose and unhelpful.
At work, we generate a couple thousand lines of k8s YAML from ~0.1x of that in CUE. The CUE is commented liberally, and validation imported from native k8s types, and sprinkled in where needed otherwise (e.g. we know for our application the FOO setting needs to be between 5 and 10). The generated YAML is clean, without any indentation and quoting worries. We also generate YAML-in-YAML, i.e. our application takes YAML config, which itself is in an outer k8s YAML ConfigMap. YAML-in-YAML is normally an enormous pain and easy to get wrong. In CUE it's just `yaml.Marshal`.
You get a lot of benefit for a comparatively simple mental model: all your CUE files form just one large document, and for export to YAML it's merged. Any conflicting values and any missing values fail the export. That's it. The mental model of e.g. cdk8s is massively more complex and has unbounded potential for abstraction footguns (being TypeScript). Not to mention CUE is Go and shipped as a single binary; the CUE v0.15.0 you use today will still compile and work 10 years from now.
You can start very simple, with CUE looking not unlike JSON, and add CUE-specific bits from there. You can always rip out the CUE and just keep the generated YAML, or replace CUE with e.g. cdk8s. It's not a one-way door.
The cherry on top are CUE scripts/tasks. In our case we use a CUE script to split the one-large-document (10s of thousands of lines) into separate files, according to some criteria. This is all defined in CUE as well, meaning I can write ~40 lines of CUE (this has a bit of a learning curve) instead of ~200 lines of cursed, buggy bash.
re: the cloud specifically see these GitHub issues:
https://github.com/dagger/dagger/issues/6486
https://github.com/dagger/dagger/issues/8004
Basically if you want consistently fast cached builds it's a PITA and/or not possible without the cloud product, depending on how you set things up. We do run it self-hosted though, YMMV.
We used Dagger, and later Nix, mostly to implement various kinds of security scans on our codebases using a mix of open-source tools and clients for proprietary ones that my employer purchases. We've been using Nix for years now, and still haven't set up any of our own binary cache. But we still have mostly-cached builds thanks to the public NixOS binary cache, and we hit that relatively sparingly because we run those jobs on bare metal in self-hosted CI runners. Each scan job typically finishes in less than 15 seconds once the cache is warm, and takes up to 3 minutes when the local cache is cold (in case we build a custom dependency).
Some time in the next quarter or two I'll finish our containerization effort for this so that all the jobs on a runner will share a /nix/store and Nix daemon socket bind-mounted from the host, so we can have relatively safe "multi-tenant" runners where all jobs run under different users in rootless Podman containers while still sharing a global cache for all Nix-provided dependencies. Then we get a bit more isolation and free cleanup for all our jobs but we can still keep our pipelines running fast.
We only have a few thousand codebases, so a few big CI boxes should be fine, but if we ever want to autoscale down, it should be possible to convert such EC2 boxes into Kubernetes nodes, which would be a fun learning project for me. Maybe we could get wider sharing that way and stand up fewer runner VMs.
Somewhere on my backlog is experimenting with Cachix, so we should get per-derivation caching as well, which is finer-grained than Docker's layers.
If you have questions about Dagger, I encourage you to join our Discord server, we will be happy to answer them!
That is, of course, a self-fulfilling prophecy (or, perhaps, a self-inflicted wound). As soon as Dagger's "multi-language support" came out (actually a bit before), the CUE SDK was rendered abandonware. Development only happened on the new backend, and CUE support was never ported over to the new one.
We shipped multi-language support because we had no choice. It was a major engineering effort that we hadn't originally planned for, but it was painfully obvious that remaining a CUE-only platform was suicide.
What else could be used to abstract away your CICD from the launcher (Jenkins, Argo Workflows, GitHub Actions, etc.)?
They don't seem to have jumped for AI hype (yet?)...
I think a more interesting point of comparison is the Claude Code Github Action, Co-Pilot code reviews, etc.
Although I'm not sure if that's so much a value-added? It's not so hard to just create a container and launch an agent in it.
The whole interesting thing was to use actual programming languages for Docker build, which I think was what they initially tried to do, but now it's a bit incomprehensible... I guess conceptually Dagger relates to Dockerfile a bit like Pulumi related to Terraform?
I.e. declaratively setup a web of CI / deployment tasks, based on docker, with a code-first DSL, instead of the morass of copy-pasted (and yes orbs) CircleCI yaml files we have strewn about our internals repos.
But their DSL for defining your pipelines is ... golang? Like who would pick golang as "a friendly language for setting up configs".
The underlying tech is technically language-agnostic, just as aws-cdk's is (you can share cdk constructs across TypeScript/Python), but it's rooted in golang as the originating/first-class language, so imo will never hit aws-cdk levels of ergonomics.
That technical nit aside, I love the idea; ran a few examples of it a year or so ago and was really impressed with the speed; just couldn't wrap my around "how can I make this look like cdk".
/s I've never heard of Dagger the DI framework but I have heard of this Dagger. Names will overlap sometimes and it's not a big deal.
If I already have a Dockerfile that doesn’t need composition, how does this help me vs being a small cosmetic improvement over ”docker build” command line?
But one flaw (IMO) that it can’t export artifacts and import into other steps without breaking the cache.
Eg if you provide monorepo as input, and then on some step narrow your build to one specific dir, then even when files change outside of that dir then caching still is invalidated.
Which makes it extremely verbose and maintenance nightmare to keep multiple narrow inputs and keep all those paths up to date.
oulipo2•7h ago
jiehong•7h ago