Maven, by default, does not check your transitive dependencies for version conflicts. To do that, you need a frustrating plugin that produces much worse error messages than NPM does: https://ourcraft.wordpress.com/2016/08/22/how-to-read-maven-....

How does Maven resolve dependencies when two libraries pull in different versions? It does something insane. https://maven.apache.org/guides/introduction/introduction-to....

Do not pretend, for even half a second, that dependency resolution is not hell in maven (though I do like that packages are namespaced by creators, npm shoulda stolen that).

This article appears to be talking about lockfiles for libraries - and I agree, for libraries you shouldn't be locking exact versions because it will inevitably pay havoc with other dependencies.

Or maybe I'm missing something about the JavaScript ecosystem here? I mainly understand Python.

The reason we have dependency ranges and lockfiles is so that library a1.0 can declare "I need >2.1" and b1.0 can declare "I need >2.3" and when you depend on a1.0 and b1.0, we can do dependency resolution and lock in c2.3 as the dependency for the binary.

Most of us, fortunately, don't post these thoughts to the internet for anybody to read.

I assume java gets around this by bundling libraries into the deployed .jar file. That this is better than a lock file, but doesn't make sense for scripting languages that don't have a build stage. (You won't have trouble convincing me that every language should have a proper build stage, but you might have trouble convincing the millions of lines of code already written in languages that don't.)

If every dependency was a `=` and cargo allowed multiple versions of SemVer compatible packages.

The first impact will be that your build will fail. Say you are using `regex` and you are interacting with two libraries that take a `regex::Regex`. All of the versions need to align to pass `Regex` between yourself and your dependencies.

The second impact will be that your builds will be slow. People are already annoyed when there are multiple SemVer incompatible versions of their dependencies in their dependency tree, now it can happen to any of your dependencies and you are working across your dependency tree to get everything aligned.

The third impact is if you, as the application developer, need a security fix in a transitive dependency. You now need to work through the entire bubble up process before it becomes available to you.

Ultimately, lockfiles are about giving the top-level application control over their dependency tree balanced with build times and cross-package interoperability. Similarly, SemVer is a tool any library with transitive dependencies [0]

[0] https://matklad.github.io/2024/11/23/semver-is-not-about-you...

When I, the owner of an application, choose a library (libuseful 2.1.1), I think it's fine that the library author uses other libraries (libinsecure 0.2.0).

But in 3 months, libinsecure is discovered (surprise!) to be insecure. So they release libinsecure 0.2.1, because they're good at semver. The libuseful library authors, meanwhile, are on vacation because it's August.

I would like to update. Turns out libinsecure's vulnerability is kind of a big deal. And with fully hardcoded dependencies, I cannot, without some horrible annoying work like forking/building/repackaging libuseful. I'd much rather libuseful depend on libinsecure 0.2.*, even if libinsecure isn't terribly good at semver.

I would love software to be deterministically built. But as long as we have security bugs, the current state is a reasonable compromise.

.NET doesn't have lock files either, and its dependency tree runs great.

Using fixed versions for dependencies is a best practice, in my opinion.

If a transient dependency (not directly referenced) updates, this might introduce different behavior. if you test a piece of software and fix some bugs, the next build shouldn't contain completely different versions of dependencies. This might introduce new bugs.

> “But Niki, you can regenerate the lockfile and pull in all the new dependencies!”

> Sure. In exactly the same way you can update your top-level dependencies.

how does updating top-level deps help with updating leaf packages? Is the author assuming that whenever a leaf package is updated, every other package in the dep chain gets immediately new release? That is fundamentally impossible considering that the releases would need to happen serially.

This works with minimal coordination between authors of the dependencies. It becomes a big deal when you have several unrelated dependencies, each transitively requiring that libpupa. The chance they converge on the same exact version is slim. The chance a satisfying version can be found within specified ranges is much higher.

Physical things that are built from many parts have the very same limitation: they need to specify tolerances to account for the differences in production, and would be unable to be assembled otherwise.

For server-side or other completely controlled environments the only good reason to have lock files is if they are actually hashed and thus allow to confirm security audits. Lock files without hashes do not guarantee security (depending on the package registry, of course, but at least in Python world (damn it) the maintainer can re-publish a package with an existing version but different content).

The algorithm can be deterministic, but fetching the dependencies of a package is not.

It is usually an HTTP call to some endpoint that might flake out or change its mind.

Lock files were invented to make it either deterministic or fail.

Even with Maven, deterministic builds (such as with Bazel) lock the hashes down.

This article is mistaken.

The author of this piece doesn't understand why a top level project might want control of its dependencies dependencies.

That's the flaw in this whole article, if you can't articulate why it's important to be able to control those... don't write an article. You don't understand the problem space.

Semantic versioning isn't perfect, but it's more than a "hint", and it sure as hell beats having to manually patch (or fork) an entire dependency chain to fix a security problem.

> Our dependency resolution algorithm thus is like this:

> 1. Get the top-level dependency versions

> 2. Look up versions of libraries they depend on

> 3. Look up versions of libraries they depend on

...would fail in languages like Python where dependencies are shared, and the steps 2, 3, etc. would result in conflicting versions.

In these languages, there is good reason to define dependencies in a relaxed way (with constraints that exclude known-bad versions; but without pins to any specific known-to-work version and without constraining only to existing known-good versions) at first. This way dependency resolution always involves some sort of constraint solving (with indeterminate results due to the constraints being open-ended), but then for the sake of reproducibility the result of the constraint solving process may be used as a lockfile. In the Python world this is only done in the final application (the final environment running the code, this may be the test suite in for a pure library) and the pins in the lock aren't published for anyone to reuse.

To reiterate, the originally proposed algorithm doesn't work for languages with shared dependencies. Using version constraints and then lockfiles as a two-layer solution is a common and reasonable way of resolving the dependency topic in these languages.

No; in fact it's perfectly reasonable, and at the core of what the author doesn't seem to get. Developers have motivations other than reproducibility. The entire reason we have version number schemes like this is so that we can improve our code while also advertising reasonable expectations about compatibility. If we have dependents, then hopefully this also improves their UX indirectly — whether by taking advantage of optimizations we made, not encountering bugs that were actually our fault, etc. Similarly, if we have dependencies, we can seek to take advantage of that.

Upgrading environments is an opportunity to test new configurations, and see if they're any better than what's in existing lockfiles.

> But this is what version ranges essentially are. Instead of saying “libpupa 1.2.3 depends on liblupa 0.7.8”, they are saying “libpupa 1.2.3 depends on whatever the latest liblupa version is at the time of the build.”

But also, developers aren't necessarily using the latest versions of their dependencies locally anyway. If I did pin a version in my requirements, it'd be the one that I tested the build with, not necessarily the one that was most recently released at the time of the build. Not everyone runs an industrial-strength CI system, and for the size of lots of useful packages out there, they really shouldn't have to, either. (And in the pathological case, someone else could re-release while I'm building and testing!)

> But... why would libpupa’s author write a version range that includes versions that don’t exist yet? How could they know that liblupa 0.7.9, whenever it will be released, will continue to work with libpupa? Surely they can’t see the future? Semantic versioning is a hint, but it has never been a guarantee.

The thing about this is that "work with [a dependency]" is not really a binary. New versions also fix things — again, that's the main reason that new versions get released in the first place. Why would I keep writing the software after it's "done" if I don't think there's anything about it that could be fixed?

For that matter, software packages break for external reasons. If I pin my dependency, and that dependency is, say, a wrapper for a third-party web API, and the company operating that website makes a breaking change to the API, then I just locked myself out of new versions of the dependency that cope with that change.

In practice, there are good reasons to not need a guarantee and accept the kind of risk described. Lockfiles exist for those who do need a guarantee that their local environment will be set in concrete (which has other, implicit risks).

I see it as much like personal finance. Yes, investments beyond a HISA may carry some kind of risk. This is worthwhile for most people. And on the flip side, you also can't predict the future inflation rate, and definitely can't predict what will happen to the price of the individual goods and services you care about most.

> The funny thing is, these version ranges end up not being used anyway. You lock your dependencies once in a lockfile and they stay there, unchanged. You don’t even get the good part!

??? What ecosystem is this author talking about? Generating a lockfile doesn't cause the underlying dependency metadata to disappear. You "get the good part" as a developer by periodically regenerating a lockfile, testing the resulting environment and shipping the new lock. Or as a user by grabbing a new lockfile, or by just choosing not to use provided lockfiles.

> “But Niki, you can regenerate the lockfile and pull in all the new dependencies!” Sure. In exactly the same way you can update your top-level dependencies.

Has the author tried both approaches, I wonder?

Not to mention: the lockfile-less world the author describes, would require everyone to pin dependency versions. In practice, this would require dropping support for anything else in the metadata format. And (I did have to look it up) this appears to be the world of Maven that gets cited at the end (cf. https://stackoverflow.com/questions/44521542).

I like choice and freedom in my software, thank you.

> “But Niki, lockfiles help resolve version conflicts!” In what way? Version conflicts don’t happen because of what’s written in dependency files.

Perhaps the author hasn't worked in an ecosystem where people routinely attempt to install new packages into existing environments? Or one where users don't want to have multiple point versions of the same dependency downloaded and installed locally if one of them would satisfy the requirements of other software? Or where dependency graphs never end up having "diamonds"? (Yes, there are package managers that work around this, but not all programming languages can sanely support multiple versions of the same dependency in the same environment.)

The entire point of lockfiles is to let the user decide when the version resolution algorithm should execute and when it shouldn't. That's all they do and they do it exactly as promised.

Given that, I still see some consequences:

The burden for testing if a library can use its dependency falls back on the application developer instead of the library developer. A case could be made that, while library developers should test what their libraries are compatible with, the application developer has the ultimate responsibility for making sure everything can work together.

I also see that there would need to be tooling to automate resolutions. If ranges are retained, the resolver needs to report every conflict and force the developer to explicitly specify the version they want at the top-level. Many package managers automatically pick one and write it into the lock file.

If we don’t have lock files, and we want it to be automatic, then we can have it write to the top level package manager and not the lock file. That creates its own problems.

One of those problems comes from humans and tooling writing to the same configuration file. I have seen problems with that idea pop up — most recently, letting letsencrypt modify nginx configs, and now I have to manually edit those. Letsencrypt can no longer manage them. Arguably, we can also say LLMs can work with that, but I am a pessimist when it comes to LLM capabilities.

So in conclusion, I think the article writer’s reasoning is sound, but incomplete. Humans don’t need lockfiles, but our tooling need lockfiles until it is capable of working with the chaos of human-managed package files.

We need the metadata. Not a new container.

It doesn't work at all for something like Python. In Python, libpupa 1.2.3 depends on liblupa 0.7.8. But libsupa 4.5.6 depends on liblupa 0.7.9. Since the Python environment can only have one version of each module at a time, I need to decide on a universe in which libpupa and libsupa can both have their dependencies satisfied simultaneously. Version ranges give me multiple possible universes, and then for reproducibility (2) I use a lockfile to define one.

(1) npm's dependencies-of-dependencies design introduces its own risks and sharp edges. liblupa has a LupaStuff data structure in it. It changed very subtly between v0.7.8 and v0.7.9, so subtly that the author didn't think to bump the minor version. And that's okay, because both libpupa and libsupa should be wrapping their dependent data structures in an opaque interface anyway; they shouldn't be just barfing liblupa-generated structs into their client code. Oh, you think people actually encapsulate like that? You're hilarious. So eventually, a LupaStuff generated by libpupa is going to get passed to libsupa, which is actually expecting a subtly different struct. Will it work? Hahah, who knows! Python actually avoids this failure mode by forcing one coherent environment; since 'pupa and 'supa have to be depending on the same 'lupa (without very fancy module shenanigans), you can have some expectation that their LupaStuff objects will be compatible.

(2) I think the author is hitting on something real though, which is that semantic versioning is a convention, not a guarantee; nobody really knows if your code working with 0.7.8 implies it will work with 0.7.9. It should. Will it? "Cut yourself and find out." In an ideal world, every dependency-of-a-dependency pairing has been hand-tested by someone before it gets to you; in practice, individual software authors are responsible for one web of dependencies, and the Lockfile is a candle in the darkness: "Well, it worked on my machine in this configuration."

However: You absolutely do need a lock file to store a cryptographic hash of each dependency to ensure that what is fetched has not been tampered with. And users are definitely not typing a hash when adding a new dependency to package.json or Cargo.toml.