Alternative to conda-lock for lockfiles?

[weiji14]

Originally posted by @mfisher87 in #138 (comment)

Longer term, we'll need a faster conda-lock (or alternative).

I was able to create a pixi lockfile in 1min 30sec!

I'd be really interested in hacking sometime on adding pixi support to repo2docker and figuring out how to leverage it in the hub. But there's also the problem of users being used to conda and introducing a new tool. My personal belief (I'm often wrong) is that pixi's user experience is superior and will eventually edge out conda in user mindshare, but I could be overly optimistic ;)

Perhaps it could be to CryoCloud's benefit to offer this superior user experience to users and help evolve their workflows. And perhaps we could offer an equivalent conda-based image for those who just need to get something done and ain't got time for that (yet).

Looks like someone has explored this already: https://www.youtube.com/watch?v=rjWtRFXRRko (haven't finished watching it yet ;) )

See also: jupyterhub/repo2docker#1339 , pangeo-data/pangeo-docker-images#571

[weiji14]

Pixi uses conda(-forge), so in theory, there should be ways to translate pixi.toml to conda-lock.yml and vice versa. I'm not sure if repo2docker has the development capacity to handle pixi.toml lockfiles natively (I've given up waiting for it to support poetry.lock), but if it does, we can then support it without re-writing most of the repo2docker-based workflow here.

The more fundamental difference is that conda focuses on 'environments', whereas pixi emphasizes 'projects'. You might face resistance getting users/scientists to switch from an all-in-one conda environment to a one-environment-per-project setup that pixi is designed for. While it is possible to build a JupyterHub environment from a pixi.toml lockfile, and even have global pixi environment, it is still going against the prevailing default, and I'm not sure how much to rely on this sort of pixi setup.

All that said, my stance (somehow hinted at #11) is that we want to aim for long-term reproducibility. The Python ecosystem has settled on a standard py.lock file in PEP 751 which is good, it represents a common denominator we can all depend on for years to come. As for multi-language environments, I'm not sure whether to bet on conda-lock or pixi in the long-term (there is a whole other argument around VC-funded projects Yuvi likes to rant on). All I can say is that there's still some 'innovation' left to do, and efforts like https://github.com/conda-incubator/conda-lockfiles popping up all the time that we just need to keep an eye on and hope that they stabilize into something reliable.

[mfisher87]

You might face resistance getting users/scientists to switch from an all-in-one conda environment to a one-environment-per-project setup that pixi is designed for

Agreed that this is a likely challenge. Though, again bright flashing 🚨 opinion 🚨 lights, I do feel this is the superior user experience. I can't count the number of times I've helped researchers troubleshoot their all-in-one conda environments that eventually become unusable! Moving to a project-based workflow could have huge payoffs for many users, but change can be sooo hard..

[weiji14]

This just released 🎉 - https://github.com/conda-incubator/conda-lockfiles/releases/tag/0.1.0. Supports both conda-lock-v1 (conda-lock.yml) and rattler-lock-v6 / pixi-lock-v6 (pixi.lock). Still in incubator status, but hopefully will become a part of conda at some point!