lsprotocol: To pin or not to pin?

[tombh]

Whether 'tis nobler in the mind to suffer
The slings and arrows of outrageous fortune,
Or to take arms against a sea of troubles

tl;dr

It's probably not that important, but there's a lot more to it than appears at first glance. I, the person responsible for maintaining this project, tacitly lean towards pinning, not because I believe that is correct, but to give voice to the depth and nuance of the conversation.

Context

This issue first arose in #317, where an upstream change in lsprotocol caused automatic breakages in downstream servers that depend on Pygls. Ultimately the root cause has been fixed by the fact that lsprotocol now includes Pygls' test suite in its CI build process. Which again highlights the fact that what is being discussed here is largely philosophical.

Arguments against pinning

This is already the current status. And even though I will argue against it, I don't think it's a problem to just continue this policy.

Before presenting the arguments against pinning I want to make some important distinctions. This issue is purely about pinning lsprotocol, not about Pygls overall approach to dependencies. We only actually have one other runtime dependency, typeguard, so again this isn't really a big deal right now. But it's important to note nevertheless because lsprotcol is unique in that it doesn't follow Semantic Versioning (SemVer).

The reasoning for lsprotocol's avoidance of Semver is actually illustrative of Python's general approach to dependencies as a whole. In #317 (comment), Brett Cannon, a significant contributor to the Python ecosystem, provides 2 insightful articles that give a comprehensive overview of Python's orientation amongst the global landscape of software's general approach to dependency management. The 2 articles are;

• https://snarky.ca/why-i-dont-like-semver
• https://iscinumpy.dev/post/bound-version-constraints

It's hard to give a summary of the arguments, but essentially they are providing concrete examples of how placing upper version limits on dependencies causes real world problems.

These real world problems are indeed ... real.

Arguments for pinning

Now, as I've said, I don't think this is really that big of deal, especially since lsprotocol is now running Pygls' test suite in its CI. So the reason I'm arguing for pinning is: Pygls is as much a universal software project as it is a Python-specific project. Which isn't really an argument at all! Nevertheless it is relevant.

Whereas Pygls and lsprotocol are written in Python, they are both projects fundamentally concerned with software as a whole. Namely the Language Server Protocol. Or to be blunt: not the Python Server Protocol. The arguments presented against version pinning are largely based on criticisms of language-agnostic practices such as; pinning, SemVer and lock files, as they apply to Python. However, those practices take on quite different identities in the light of other language ecosystems. The most relevant factor being Python's "flat" dependency tree, namely that multiple versions of a dependency cannot coexist. Which is not the case in say Javascript or Rust, where multiple versions of a dependency can coexist.

Pinning, SemVer and lock files are not in and of themselves flawed computer science concepts. As the maintainer and official voice of Pygls, I say to our downstream creators of non-Python Language Servers: we value the hard-fought wisdom of versioning best practices. We do not abandon their utility lightly. Where possible, Pygls strives for those universally and perennially proven practices of idempotency, reproducibility and reliability. Yet we are also pragmatic, and so equally where possible, we balance such idealism with the benefits of developer experience: a pragmatic and active project project is better than an ideal but inactive project.

The arguments against pinning are valid, though I would argue they are more valid for Python in general than Pygls specifically. Considering that Pygls only has 2 runtime dependencies, one of which (lsprotocol) is highly specific, if not exclusive to Pygls, and indeed unique (justifiably so) in its approach to versioning, then Python's versioning best practices can legitimately be held more lightly in order to benefit from the broader software ecosystem's collective wisdom around versioning.

Conclusion

This isn't a big deal. I'm totally happy not pinning lsprotocol. I present this issue more to reflect the nuances of the matter and to demonstrate that Pygls is both respectful to the particular benefits of Python and the broader wisdom of software as whole.

[augb]

I agree with your sentiments, @tombh.

In the first article you reference, this statement captures my rationale for favoring pinning of depenencies in general , and lsprotocol in particular:

Remember to pin your dependencies in your apps if you really don't want to have to worry about a dependency breaking you unexpectedly

Whether all pygls dependencies should be pinned, is another issue. The likelihood that lsprotocol will be imported by a project relying on pygls strikes me as exceedingly low. If the project is relying on pygls, then in essence, that project is constrained by our dependency on a particular version of lsprotocol anyway.

Whether or not semver should be used for pygls, is another topic for another time. Having said that, I do think it does communicate intent, even if imperfectly in the Python context. Stated another way, the pygls project and/or developers believe this to be a major, minor or patch change. Even if this turns out to be incorrect, it, hopefully, causes folks to give some consideration to what their changes actually are, and to communicate this to the broader community.

My 2 cents.

[tombh]

Thanks @augb. I'd be up for the full lockfile, pinning, semver, package-managed experience, say with something like Poetry. So I appreciate your 2 cents. In other language ecosystems, I've come to really value these approaches. Even if, as you say for example, maybe semver isn't perfect, but it does at least express intent.

I intend to write good software 🙇 Even if I don't always achieve that 😆

[karthiknadig]

@tombh I recommend pinning. There are some changes coming in the next version of lsprotocol that might break things (this is due to changes in LSP spec). Pinning is highly recommend. Also, 3.18 proposed features are coming in as well which means there will be regular updates to the proposed types.

[tombh]

Thanks Karthik, I've opened a PR for the pinning: #345

[tombh]

Addressed in #349