When the configuration grows complex, and you feel the need to abstract and generate things, switch to a configuration language like Cue or RCL, and render to json. The application doesn't need to force a format onto the user!
At work we generate both k8s manifests as well as application config in YAML from a Cue source. Cue allows both deduplication, being as DRY as one can hope to be, as well as validation (like validating a value is a URL, or greater than 1, whatever).
The best part is that we have unit tests that deserialize the application config, so entire classes of problems just disappear. The generated files are committed in VCS, and spell out the entire state verbatim - no hopeless Helm junk full of mystery interpolation whose values are unknown until it’s too late. No. The entire thing becomes part of the PR workflow. A hook in CI validates that the generated files correspond to the Cue source (run make target, check if git repo has changes afterwards).
The source of truth are native structs in Go. These native Go types can be imported into Cue and used there. That means config is always up to date with the source of truth. It also means refactoring becomes relatively easy. You rename the thing on the Go side and adjust the Cue side. Very hard to mess up and most of it is automated via tooling.
The application takes almost its entire config from the file, and not from CLI arguments or env vars (shudder…). That means most things are covered by this scheme.
One downside is that the Cue tooling is rough around the edges and error messages can be useless. Other than that, I fully intend to never build applications differently anymore.
We did a prototype at work to try different configuration languages for our main IaC repository, and Cue was the one I got furthest with, but we ended up just using Python to configure things. Python is not that bad for this: the syntax is light, you get types, IDE/language server support, a full language. One downside is that it’s difficult to inspect a single piece of configuration, you run the entry point and it generates everything.
As for RCL, I use it almost daily as a jq replacement with easier to remember syntax. I also use it in some repositories to generate GitHub Actions workflows, and to keep the version numbers in sync across Cargo.toml files in a repository. I’m very pleased with it, but of course I am biased :-)
I think the post is hurt by the desire to sort of… “have a theory” or take a new stance. The configuration file is obviously not a user interface, it is data. It is data that is typically edited with a text editor. The text editor is the UI. The post doesn’t really justify the idea of calling the configuration file, rather than the program used to edit it, the UI. Instead it focuses on a better standard for the data.
The advancement of standards that make the data easier to handle inside the text editor is great, though! Maybe the slightly confusing (I dare say confused) blog title will help spread the idea of kson, regardless.
Edit: another idea, one that is so obvious that nobody would write a blog post about it, is that configuring your program is part of the UX.
- The key-value pair. Maybe some section marker (INI,..). Easy to sed.
- The command kind. Where the file contains the same command that can be used in other place (vim, mg, sway). More suited to TUI and bigger applications.
With these two, include statement are nice.
Curiously, LLMs have made it a lot easier. One step away from an English adapter that routes through an LLM to generate the config.
Here is an example I made in a few minutes:
ports:
- 80
- 8000 - 10000
- 12000 -
- 14000
{"ports":[80,8000,10000,12000,[14000]]}When I think about it, any language should come with a strict, non-configurable built-in formatter anyways.
Would that be on the language, or the IDEs that support it? Seems out of scope to the language itself, but maybe I'm misunderstanding.
If you supposedly human writable config file format is unusable without external tools, there is something wrong with it.
Just use a formatter and stop bikeshedding please.
And KSON is pretty unique in how bad is it. For example, go to playground and remove a single dot in line 11 (after the books list). Look how the parse tree changes - this 1-character edit means the file is still valid, but the rest of the file is now filed under one of the books.
Do you know of any other config language with this property - a single character removed yields a syntactically correct file which yet parses absolutely incorrectly? I don't. Worst I can think of is maybe one section parses weirdly, or next element of a list. But something like _entire rest of file_ is broken is unique to KSON.
Btw, I love JetBrain's SSH sync for keeping my config files in VCS and editing them with proper tooling.
This is roughly equivalent to saying that a linter can transform the AST of the language into a canonical representation, and the syntax will be rejected unless it matches the canonical representation (modulo things like comments or whitespace-for-clarity).
I don't think warnings will be part of the spec, though classes of errors may be specified (TBD). This will allow, for instance, implementations that prioritize pure speed by leaving all the guardrails off and simply replying "NO." when they are unable to parse some KSON (and then possibly falling back to a parse with richer validations?).
But languages are more than syntax, they are also the ecosystem of tools around the language. And for a language to be useful and reliable, that ecosystem of tools needs to be excellent, and it needs to be available on any platform you want to use that language. That's a really important motivator for KSON's multiplatform architecture, which allows us to reach all these platforms[2] and editors[3] (with even more to come!)
[1] https://github.com/kson-org/kson/blob/857d585ef26d9f73e080b5... [2] https://kson.org/docs/install/#languages [3] https://kson.org/docs/install/#editor-support
i.e. until that "formal grammar" which is currently a comment in a Kotlin source file ;) becomes a versioned and reference-able spec that can be implemented by third-parties this is all a bit of a cart/horse dilemma i think maybe
To see misleading indentation warnings in action, you can try the following snippet in the playground (https://kson.org/playground/) and will properly get a warning:
ports:
- 80
- 8000
- 10000
- 12000
- 14000
If your config format requires autoformatter and/or linter to detect trivial mistakes, it is junk.
IMO this way of thinking disqualifies lots of legitimate configuration formats, but to each their own I guess.
I’ll stick to JSON. When JSON isn’t enough it usually means the schema needs an update.
The idea of configuring something but not actually having any sort of assurances that what you're configuring is correct is maddening. Building software with nothing but hopes and dreams.
Example with Terraform HCL: if everything was just declared as plain resources, you can modify them trivially. But HCL has functions, so people try to DRY their code by declaring some custom input variable, running maps/filters/reshaping it until it's unrecognizable and them generating a dynamic set of resources out of it. There's no way a program can work through the reverse of all the input reshaping logic to e.g. add another resource without knowledge of the custom input structure.
However, reality shows that embedded code in configs is already widespread - think SQL queries in dbt, bash scripts in CI/CD configs, or regex patterns in routing rules. These aren't going away. KSON's embed blocks acknowledge this reality while making it safer and more maintainable through proper syntax highlighting, validation, and clear boundaries. We're not encouraging bad practices, we're providing better tooling for existing patterns.
The comment option gives enough space for devs to explain new options flags and objects included to those familiar enough to be using it.
For customer facing configurations we build a UI.
> "backwardsCompatible": "with JSON",
But in that same example, they have a comment like this:
> // comments
Wouldn't that make it not compatible with JSON?
From what I understand, it's "backwards-compatible" with JSON because valid JSON is also valid JSON5.
But it's not "forwards-compatible" precisely because of comments etc.
If people tended to interpret English correctly and not be susceptible to misinterpreting written statements, that would be nice. Reality is a bugger!
"compatible with existing valid JSON"
Forwards-compatible means the old thing can handle the new things. Here JSON is not forwards-compatible with JSON5.
1: I pulled that out of my butt, there's no factual data to it.
I like the language embedding feature in KSON - we would use that. Have you thought about having functions and variables? That is something you get in Pkl and Dhall which are useful.
This sounds like the kind of question for Daniel himself to chime in, since he has the best overview of the language's design and vision. He's not super active on HN, but I'll give him a heads up! Otherwise feel free to join our Zulip (https://kson-org.zulipchat.com) and we can chat over there.
We have a design for KSON constants that we're looking forward to sharing soon, but they will be constant and KSON will still be JSON/YAML-equivalent. For the places where someone wants more programming power in their config, I'm glad people like the folks behind Pkl and Dhall are serving that use case.
That’s why 90% of each iOS update is just another menu or a reorganization of menus and why there are 3 different ways to access the same info/menus on iOS.
Configuration is difficult, the tooling is rarely the problem (at least in my experience).
But we allow it for files that tend to make production changes usually without any unit tests!
I'd prefer something syntaxed like a programming language but without turing completeness.
And don’t forget Ansible playbooks!
It's not telling, it's impossible. Neither JSON or YAML are Turing complete. That said, the JS in JSON is for Javascript, whose syntax it is derived from, so at least one major language uses it in its syntax.
This is because the only implementation is written in Kotlin. There are Python and Rust packages, but they both just link against the Kotlin version.
How do you build the Kotlin version? Well, let's look at the Rust package's build.rs:
https://github.com/kson-org/kson/blob/main/lib-rust/kson-sys...
It defaults to simply downloading a precompiled library from GitHub, without any hash verification.
You can instead pass an environment variable to build libkson from source. However, this will run the ./gradlew script in the repo root, which… downloads an giant OpenJDK binary from GitHub and executes it. Later in the build process it does the same for pixi and GraalVM.
The build scripts also only support a small list of platforms (Windows/Linux/macOS on x86_64/arm64), and don't seem to handle cross-compilation.
The compiled library is 2MB for me, which is actually a lot less than I was expecting, so props for that. But that's fairly heavy by Rust standards.
Edit: point taken about verifying checksums, just created an issue for it (https://github.com/kson-org/kson/issues/222)
Both to bootstrap making a parser in a new language, and also as a kind of living spec document.
All in all, I'm confident that KSON can become ubiquitous despite the limitations of the current implementation (provided it catches on, of course).
But I'd strongly encourage everybody to think about whether that deep configurability is really needed.
Where KSON diverges from hjson is enabling a YAML-like presentation while improving on the YAML experience. And by making the content type of embed blocks a first-class citizen of the syntax, with clear boundaries for the embedded content, editors and IDEs can provide full tooling support (syntax highlighting, validation, etc.) for the embedded code. Importantly though, KSON remains 'plain data'---it's not a runtime that executes code in your configs.
Configuring TypeScript applications with the `defineConfig` pattern that takes asynchronous callbacks allowing you to code over settings is very useful. And it's fully typed and programmable.
It's particularly useful because it also allows you to trivially use one single configuration file where you only set what's different between environments with some env === "local" then use this db dependency or turn on/off sentry, etc.
Zig is another language that shows that configuration and building should just be code in the language itself.
Note that this implies a Turing incomplete language. Which makes sense - our goal is to make dangerous programs unrepresentable, so naturally we can't implement every algorithm in our restricted language.
This depends on the trust-model for who is doing the configuration, especially if you're trying to host for multiple tenants/customers. Truly sandboxing arbitrary code is much harder than reading XML/JSON/etc.
Yes, there are bad consequences that can happen. No, you don't dodge having problems by picking a different data format. You just pick different problems. And take away tools from the users to deal with them.
Definitely more control than guessing the right JSON, or breaking that YAML file. Plus, you get completion, introspection, and help while editing the config because you're in a code-writing environment. Bonus for having search and text manipulation tools under your fingertips instead of clicking checkboxes or tabbing through forms.
The data model is closer to XML than JSON, though, so unfortunately it's not a drop-in replacement for yaml.
Small sample:
package {
name my-pkg
version "1.2.3"
dependencies {
// Nodes can have standalone values as well as // key/value pairs.
lodash "^3.2.1" optional=#true alias=underscore
}
}As an example, a friend of mine introduced TOML to a reasonably big open source project a while ago. Recently, he mentioned there were some unexpected downsides to TOML. I've asked him to chime in here, because I think he's more qualified to reply (instead of a best-effort attempt from my side to faithfully reproduce what he told me).
Think about a Github Action being written in TOML ... would probably not look great!
Another fun thing about configuration: it's a great indicator of poor software design. If the configuration can be very complicated, in one single format, in one big file, look at the source code and you'll find a big ball of mud. Or even worse, it's lots of very different software that all shares one big config file. Both are bad for humans and bad for computers.
I use a lot of openbsd and one of the things I really like about it is that they care about the user interface(note 1) and take the time to build custom parsers(note 2)
Compare pf to iptables. I think, strictly speaking, ip tables is the more feature-full filter language. but the devs said "we will just use getopt to parse it" and called it a day. leading to what I find one of the ugliest, more painful languages to use. pf is an absolute joy in comparison. I would pick pf for that reason alone.
note 1. Not in the sense of chasing graphical configurators. An activity I find wastes vast amounts of developer time leading to neutered interfaces. But in the sense of taking the time to lay out a syntax that is pleasant to read and edit and then write good documentation for it.
note 2. If I am honest, semi custom parsers, the standard operating procedure appears to start with a parse.y from a project that is close and mold it into the form needed. which lends itself to a nice sort of consistency in the base system.
As for Apple Pkl, I think we share the goal of robustness and "toolability", but pkl seems way more complex. I personally think it's more useful to keep things simple, to avoid advancing in the configuration complexity clock (for context, see https://mikehadlow.blogspot.com/2012/05/configuration-comple...).
price_in_€: 42
"price_in_€": 42
And code is also user interface.
So you probably don't really want a separate language for configuration, but rather a programming language that (a) has a reasonable way of creating complex object literals and (b) can be, but does not have to be, interpreted.
And you want to be able to subset and/or sandbox the language so that the power you give to those parts of the program that does the configuring is under control.
I saw this from one side when working on the BBC SportStats project, where I eliminated all the configuration files. All of them. The deployment was a single jar. That doesn't mean that there wasn't configuring going, but it was done with Java code inside the jar.
One reason for doing it this way was that deployment was just as complex/arduous for a configuration file change as it was for a code change. And the code we could easily test, abstract etc. So there were few if any benefits.
At a later stage we did have multiple instances of the app, all still using the same jar, but now delivered with different 1-2 line shell scripts that selected one of the built-in configurations.
It can be converted to JSON, yaml, and a bunch of other formats, and whilst it looks a bit like JSON it's actually a full blown functional programming language designed for creating config trees. The nice thing about Pkl is that it's got full IDE support and the standard library is patterned after Kotlin. So all the method names are very obvious and intuitive especially if you know Java or Kotlin. It has lots of features for validation like range types and so on, so you can get a lot of semantic checking before any config is emitted. It's the first config-as-language language I've seen that is actually good.
KSON looks more like an alternate syntax than a language, which is fine, but because JSON/YAML are just data structures sometimes you really want to do a for loop.
server_config:
port_mapping:
# Expose only ssh and http to the public internet.
- '22:22'
- '80:80'
- '443:443'
serve:
- '/robots.txt'
- '/favicon.ico'
- '*.html'
- '*.png'
# Do not expose our Git repository to the entire world.
- '!.git'
geoblock_regions:
# The legal team has not approved distribution in the Nordics yet.
- dk
- fi
- is
- no
- se
flush_cache:
on: [push, memory_pressure]
priority: background
.
allow_postgres_versions:
- '9.5.25'
- '9.6.24'
- '10.23'
- '12.13'
name: 'Leonardo Bonacci'
favorite_books:
- title: Metaphysics
author: Aristotle.
favorite_numbers:
- 1
- 2
SSE2 assembler. Fast and self explanatory.
For well over a decade, I've felt we missed the boat with XML, XSD, and friends to make a generic, literal user interface for configuration such that you didn't need to rely on referencing documentation for valid options and such and usable interfaces could be presented instead of editing the text directly.
Spivak•4mo ago
I agree that it's not perfect but worse is better and familiar is a massive win over making your users look up a new file format or set their editor up for it. If you truly hate YAML that's fine, there's plenty of other familiar formats: INI, toml, JSON.
NeutralForest•4mo ago
ksenzee•4mo ago
NeutralForest•4mo ago
hk1337•4mo ago
Why would you be doing that?
NeutralForest•4mo ago
hk1337•4mo ago
Like, are you _literally_ scripting in the value of an item or are you just equating that they are similar?
Literal being:
get_number_of_lines:
NeutralForest•4mo ago
Invariably, ppl will write inline scripts instead of actual scripts in their own files. There are also some SDKs for most of these operations that would let you do it in code but they are not always 100% the same as the CLI, some options are different, etc.
Spivak•4mo ago
kiitos•4mo ago
chromalchemy•4mo ago
https://yamlscript.org/