Can we use compatible release spec in environment.yml?

[MattF-NSIDC]

Hello all,

My usual practice with environment.yml is to use compatible release spec to make upgrades easier, and a lockfile for reproducibility. Since this repository has a lockfile, what do you think about modifying the environment.yml to not pin dependencies as strictly? This article speaks to the different use cases of these two types of spec files: One focuses on specifying a human-curated environment optimized for upgradeability, while the other optimizes for reproducibility and is generated by machine from a real environment.

Compatible release spec allows specifying an arbitrary number of version digits, e.g. ~=1, ~=1.2, or ~=1.2.3. The last digit specified is allowed to "move", and others are not. If you specify ~=1.2.3, version 1.2.4 would be allowed, but 1.3 would not. This provides some advantage over greater/less-than spec because in my experience a spec like >=3.0,<4.0 can allow a 4.0.0alpha1 package to be installed. Perhaps this issue has been long-since resolved, but it's the reason I stopped using that style.

In PR #45 I used compatibility spec out of habit and then realized the inconsistency. Happy to change it or not (or, if desired, overhaul the entire environment.yml file to use compat spec) depending on discussion outcome :)

Thanks all!

[weiji14]

Thanks Matt for raising this issue! In general, I'm in agreement with you on having a loosely pinned conda environment.yml file and a specifically pinned conda-lock-64.lock file (which needs to be improved in #14). That is generally what conda-lock is working towards in the conda ecoystem, much less so in the PyPI ecosystem.

On whether using ~= compatible release spec or >= minimum pinning is preferred, that depends on the situation. The general disagreement is that we can't predict what happens to a package in the future, see e.g. https://prefix.dev/blog/the_python_packaging_debate. If people followed semantic versioning perfectly, then ~= would be a good idea, but in practice, it can be a source of frustration when people want the latest version 4.0 of some library and they know it works, but some maintainer in the past decided to pin to >=3.0, <4.0. Another example is when packages like xarray switched from semver to calver (e.g. 0.19.0 to 2023.0.1) and all those libraries that set an upper bound or compatible release spec to >=0.16.0, <1.0.0 couldn't handle it. And yes, ~= doesn't really work so well with CalVer for that matter since you'd be stuck with yearly versions.

To be fair, I'm not against using ~=, I've actually used https://python-poetry.org/docs/dependency-specification/#tilde-requirements quite a bit with past projects. The key factor is that someone needs to actively maintain these pinnings, and with a long-term project like CryoCloud, it depends on whether someone can do that in 2 or 3 years time. This is perhaps just a long winded way of me saying, do you want to be added to this repo to implement and maintain this? 😉

Edit: Sorry, I just realized you're part of the cryoclouduser team already! In that case, I should ask if you're keen to help review future issues/PRs related to dependency upgrades.

[MattF-NSIDC]

https://prefix.dev/blog/the_python_packaging_debate

☝️ Excellent article! @betolink shared this with me earlier this week.

I don't like calver, personally; there's no specification method (aside from e.g. ~= 2023.01.1 would allow patch updates only) that protects you from breaking changes while still allowing implicit upgrades :(

The key factor is that someone needs to actively maintain these pinnings, and with a long-term project like CryoCloud, it depends on whether someone can do that in 2 or 3 years time.

This is always the challenge with complex environments. Sometimes I'm away from a project from 3 years and cleaning up the mess at that point is always a nightmare.

This is perhaps just a long winded way of me saying, do you want to be added to this repo to implement and maintain this? wink

Edit: Sorry, I just realized you're part of the cryoclouduser team already! In that case, I should ask if you're keen to help review future issues/PRs related to dependency upgrades.

I'm absolutely willing to help out with environment-related changes. I'm feeling fairly comfortable with the tools you're using for this repo after doing these two PRs today!

[weiji14]

Cool, we've decided to go ahead to relax the dependency pinnings, beginning with a few packages in #56. Currently the ~= compatible release spec is used, but we could also use >= for packages using CalVer if needed. Thanks again @MattF-NSIDC for the useful discussion, closing this issue now :D