Conda package attempt #75

ilaflott · 2025-01-22T22:08:35Z

this PR:

adds meta.yaml and build.sh to define the conda build package recipe
adds environment.yml defining fully functional conda environment inwhich mkmf should work
puts all bin/ executables under a nested mkmf "package" directory for conda build recipe compatibility
adds a github workflow file to run conda build under github's CI/CD pipelines, conda build . --no-anaconda-upload version in .github/workflows/build_conda.yml, with uploading in .github/workflows/publish_conda.yml. run conditions identical to that of fre-cli
removes some old unused files (.travis.yaml, .gitlab-ci.yml)
adjust success condition of some tests, e.g. list_paths prints version regex reflect recent version strings
adjusts how t/t001-list_paths.sh does set-up, now creates t/src/file6.f90 symbolic link on-the-fly. removed at end of each of those tests
adjusts how every test does setup- checks PATH and adjusts only if needed. above file6.f90 link creation done on-the-fly for t/t003-mkmf.sh as well.
tests and package should work in the usual way with the usual level of functioning for all desired contexts

…define build recipe in meta.yaml and relevant fields. define conda environment.

ilaflott · 2025-01-22T22:12:17Z

green check, yes. note failures in pipeline...

…for test step. add git at dependency to see if it keeps the pipeline happy.

ilaflott · 2025-01-23T19:34:27Z

current testing state is odd.

In the pipeline, t/t001-list_paths.sh tests fail, tests #'s 1, 3, 5, 6, and 7 (see copied pipeline output). But, locally on my workstation, only the first test, 1 list_paths prints version fails.

+ bats t/t001-list_paths.sh
+ echo 'failure guard :-('

not ok 1 list_paths prints version
# (in test file t/t001-list_paths.sh, line 19)
#   `[[ "$output" =~ ^list_paths\ [0-9]+\.[0-9]+$ ]]' failed

not ok 3 list_paths using default out file
# (in test file t/t001-list_paths.sh, line 32)
#   `[ "$(wc -l < path_names)" -eq 6 ]' failed


not ok 5 list_paths find files in t and test_* directories
# (in test file t/t001-list_paths.sh, line 44)
#   `[ "$(wc -l < path_names)" -eq 8 ]' failed

not ok 6 list_paths find specific files in t or test_* directories
# (in test file t/t001-list_paths.sh, line 51)
#   `[ "$(wc -l < path_names)" -eq 7 ]' failed

not ok 7 list_paths with specified out file
# (in test file t/t001-list_paths.sh, line 59)
#   `[ "$(wc -l < $outFileName)" = "6" ]' failed

Additionally, in the pipeline, all t/t002-git-version-string.sh tests fail, and yet locally on my workstation, all of these tests pass. I suppose i'm not entirely surprised by this because i've seen github's CI/CD has been weird about calling git commands directly in the past.

+ bats t/t002-git-version-string.sh
+ echo 'failure guard :-('

…move most unneeded failure-guarding shell commands.

…st t/t001-list_paths.sh

…nge approach to resolving binDir in t/t001-list_paths- should possibly keep the pipeline happier. include mkmf in source files for test step of conda build. more echos, switch spot of echo output to avoid adjusting status of last-called shell command.

…he testing approach makes perfect sense locally- so why isnt the pipeline happy?

…tween string and ints in test success condition checks

…ia cat and wc -l

…slowing this pipeline down for its own good!! (maybe...)

…acing.

ilaflott · 2025-01-23T21:27:57Z

AHA! the failure in githhub's pipeline arises because the command there, for some reason, resolves the symbolic link file file6.linked in t/src to file file6.f90 with one of the targeted extensions.

a little more simply: locally, the test list_paths using default out file in t/t001-list_paths.sh should ignore a link and only find 6 files. In github CI/CD, the symbolic link is resolved and it finds 7 files resulting in failure.

… answer -_-

ilaflott · 2025-01-23T22:13:11Z

relatively sure that once conda build . enters the test phase, the source_files key specifying t/ likely copies the directory with a command ala cp -L -r t/ <target_dir>, and is de-referencing the link to file6.f90, changing the answers in the list_files tests. i adjusted the answers accordingly, but this is honestly bad practice.

the git based tests in t still do not work in the pipeline but work fine for me locally.

ceblanton · 2025-01-23T22:31:14Z

It looks great, thanks!

The noaa-gfdl channel is not needed for building, is it?

What do you think about throwing in the noaa-gfdl anaconda channel key and having this pipeline publish as well as build. Too soon?

It's odd that the tests needing adjusting (but then again I hadn't realized there were mkmf tests. :)

ilaflott · 2025-01-23T22:34:48Z

The noaa-gfdl channel is not needed for building, is it?

nope!

What do you think about throwing in the noaa-gfdl anaconda channel key and having this pipeline publish as well as build. Too soon?

no we can do it- i'll maybe say it should stick with an alpha version tag for now... at most, beta

It's odd that the tests needing adjusting (but then again I hadn't realized there were mkmf tests. :)

100% conda build is derefencing links found in specified source files for the testing step! idk what the best way to deal with that is and not lose true quality control.

put conda-relevant things in README- adjust markdown, update dependencies discussed. put disclaimer+license at the bottom of the readme to emphasize usage, installation, featurres etc.

update badge

… pipeline again.

…pty repo for pipeline-compatibility

ilaflott · 2025-01-24T16:34:31Z

All tests now pass without caveat in all contexts.

ceblanton

Nice work, Ian. Bonus for the the documentation updates.

The templates still have to be installed somehow, right?

.github/workflows/build_conda.yml

environment.yml

meta.yaml

README.md

ilaflott · 2025-01-24T18:07:32Z

Nice work, Ian. Bonus for the the documentation updates.

The templates still have to be installed somehow, right?

RE: these templates- mkmf needs you to specify a template target as an argument- i think. are these templates just supposed to be "found"? are they expected to be "nearby" ?

EDIT:
https://github.com/NOAA-GFDL/fre-cli/blob/a9730e039274739e5dc8710d82dcc09578735d18/fre/make/gfdlfremake/buildDocker.py#L31

ceblanton · 2025-01-24T18:26:36Z

It's odd that mkmf includes the code and the templates for compiling FMS models on selected platforms, but they're been used together like this for 20 years.

I think it's a little more consistent to only conda install the mkmf and list-paths tools, but by tradition, we should probably include the templates in the conda package also.

CatalogBuilder does this.. I'm not sure how it works, but e.g. the templates are installed alongside the bin and lib files. @Ciheim could explain it. For proof though, the GFDL default template (gfdl_template.json) is installed in fre-cli:

c2b:/nbhome/fms/conda/envs/fre-test%>find . -name gfdl_template.json 

./lib/python3.9/site-packages/catalogbuilder/cats/gfdl_template.json

ilaflott · 2025-01-24T18:34:07Z

It's odd that mkmf includes the code and the templates for compiling FMS models on selected platforms, but they're been used together like this for 20 years.

using shell to drive workflows for decades will incentivize this kind of approach. unfortunately, given the way conda packages function... do we really want to be doing things like this? targeting the templates/ dir within a conda environment i guess like:

mkmf -t /home/Ian.Laflotte/conda/envs/mkmf_test/......mkmf/templates

maybe we should consider de-coupling the templates and the code?

I think it's a little more consistent to only conda install the mkmf and list-paths tools, but by tradition, we should probably include the templates in the conda package also.

i'm sympathetic, i wanna figure it out.

CatalogBuilder does this.. I'm not sure how it works, but e.g. the templates are installed alongside the bin and lib files. @Ciheim could explain it. For proof though, the GFDL default template (gfdl_template.json) is installed in fre-cli:
c2b:/nbhome/fms/conda/envs/fre-test%>find . -name gfdl_template.json 

./lib/python3.9/site-packages/catalogbuilder/cats/gfdl_template.json

It's not quite 1-1: mkmf is not a python package and there is no pip install procedure here.

ilaflott · 2025-01-24T18:40:10Z

catalogbuilder is using it's own pkg_dir property: https://github.com/search?q=repo%3ANOAA-GFDL%2FCatalogBuilder%20gfdl_template&type=code

i get the executables in the environment to $PREFIX/bin, because conda adds that to one's PATH for you. I could add a similar step where i cp mkmf/templates/* $PREFIX/lib or something. but fre-cli might have to do that pkg_dir path wrangling on it's own, and adjust the -t option, making the directory resolving fre make's problem to solve

ilaflott · 2025-01-24T18:46:28Z

i think the ideal solution would be that
mkmf -t /absolute/or/relative/path/to/template.mk works as it does now, but e.g. specifying mkmf -t ncrc-gcc without the .mk extension says to mkmf "hey, get this template from your include dir".

ceblanton · 2025-01-24T18:50:10Z

Yes!! I agree that would be a natural solution. I was thinking of a dummy list-templates tool to be added that reports the template directory but your idea is better.

singhd789 · 2025-01-24T20:58:19Z

Nice work, Ian. Bonus for the the documentation updates.
The templates still have to be installed somehow, right?

RE: these templates- mkmf needs you to specify a template target as an argument- i think. are these templates just supposed to be "found"? are they expected to be "nearby" ?

EDIT: https://github.com/NOAA-GFDL/fre-cli/blob/a9730e039274739e5dc8710d82dcc09578735d18/fre/make/gfdlfremake/buildDocker.py#L31

Pertaining to the templates for fre make functions, currently we have the path specified in the platforms.yaml:
https://github.com/NOAA-GFDL/fre-cli/blob/a9730e039274739e5dc8710d82dcc09578735d18/fre/make/tests/null_example/platforms.yaml#L6

Ryan mentioned this in another issue as well (in relation to mkmf being a submodule but still applies): NOAA-GFDL/fre-cli#303

If the template is in a set location in the conda package, that would pretty nice to refer to that location and just have the name of the template in the platforms.yaml (for fre make things at least)

ilaflott · 2025-01-24T21:52:29Z

My main problem with me implementing that in this PR is that i do not know perl and am unsure if i'm the best person to implement that functionality on a short time-scale.

ilaflott · 2025-01-24T21:53:16Z

Let's chat for a minute in-person on this when we can

ilaflott · 2025-01-27T20:05:51Z

The templates should be a separate repository, as they're not really used by this repository out of necessity- they are a convenience for prospective users.

We're informing this view point with the following:

all mkmf does with the value given to -t is stick the value as-is into the output Makefile, and the template is not read nor it's contents used programmatically by mkmf.

> mkmf -t FOO
. Makefile is ready.

> cat Makefile | grep TEMPLATE
MK_TEMPLATE = FOO
include $(MK_TEMPLATE)

no one to our knowledge is relying on the templates in this repo as being in a particular spot directory-structure wise.

… package directory.

slightly adjust current README instructions

test-run publish_conda workflow by asking it to run on push to `conda_package` branch, see what happens!

…gitlab/travis pipeline workflow files

singhd789 · 2025-06-25T16:13:21Z

.github/workflows/build_conda.yml

Just curious, mkmf isn't really touched that much I don't think...right? If it was at all, Mikyung had that really nice addition in the fre-cli build_conda script:
https://github.com/NOAA-GFDL/fre-cli/blob/main/.github/workflows/build_conda.yml#L8-L10

it's not- but those bits that @mlee03 stuck in there are about redundant CI/CD workflows, not about the install. It's to use CI/CD minutes efficiently.

singhd789 · 2025-06-25T16:15:45Z

.github/workflows/publish_conda.yml

+          conda install conda-build conda-verify
+
+          # set token and upload preference
+          export ANACONDA_API_TOKEN=${{ secrets.ANACONDA_TOKEN }}


For my own clarification, what were the token and secrets for again?

upload to the noaa-gfdl channel

singhd789 · 2025-06-25T16:17:31Z

README.md

+Usage as a `conda` package requires `conda`.
+
+For testing, we recommend [`bats-core`](https://github.com/bats-core/bats-core),
+a current and up-to-date (as of January 2025) fork of the original 


looks like there was a new release as of May 18th

singhd789 · 2025-06-25T16:18:47Z

README.md

-Kushner's suggestion).
+Usage as a `conda` package requires `conda`.
+
+For testing, we recommend [`bats-core`](https://github.com/bats-core/bats-core),


Might be nice to include what bats stands for (Bash Automated Testing System)..because I had no idea what it was 😅

and maybe a quick little blurb about why it's recommended?

bats is standard bash testing infrastructure- given the link is there for a curious person to find bats-core doc with, i'm gonna disagree here.

and you're right, it's not recommended- it's required until someone comes up with something different for testing :-p.

singhd789 · 2025-06-25T16:20:05Z

README.md

+git clone https://github.com/noaa-gfdl/mkmf.git mkmf && cd mkmf
+conda env create -y -f ./environment.yaml
+conda activate mkmf
+export $PATH=$PWD/mkmf/bin:$PATH


Wait, why would it be necessary to add it to the path if there is a conda package for mkmf?

this is like the "developer install" of fre-cli- local code responds to local edits, so add the local mkmf/bin to PATH.

singhd789 · 2025-06-25T16:26:09Z

README.md

A bit unrelated, but do you know if anyone has made a simplified conda packaging guide at the lab or is it really, just follow along with tutorials online?

this is really good development, so I'm just curious

none that i know of. generally, we should be consulting official packaging guides and encouraging others to do so. we should be very cautious about writing our own packaging guides on top of the official ones- that invites some pretty nasty potential mistakes/misunderstandings

... that being said i have some personal notes i can share that may shortcut you, just don't spread them around as a "packaging guide" :-p... follow up on that offline

I wholeheartedly agree, just wondering

Conda packaging is great candidate for the 5-10 minute MSD training topics, I agree.

I'm afraid yours truly attempted to do exactly this, building off of a Seth guide, as part of a conda MSD seminar a year or two ago, but the conda packaging parts were not that well-developed.

underwoo

Some changes, or additional information needed.

underwoo · 2025-06-25T16:40:23Z

meta.yaml

+requirements:
+  host:
+    - conda-forge::perl
+    - conda-forge::tcsh


tcsh is not used in any of the mkmf tools anymore. This dependency can be removed.

underwoo · 2025-06-25T16:40:32Z

meta.yaml

+    - conda-forge::bats-core
+  run:
+    - conda-forge::perl
+    - conda-forge::tcsh


underwoo · 2025-06-25T16:40:45Z

meta.yaml

+test:
+  requires:
+    - conda-forge::perl
+    - conda-forge::tcsh


Not needed.

underwoo · 2025-06-25T16:46:27Z

t/t000-basic.sh

+   binDir=$(readlink -f ${BATS_TEST_DIRNAME}/../mkmf/bin)
+   do_we_have_mkmf=$(which mkmf) || echo "no we do not!"
+   if [ $do_we_have_mkmf ]; then
+	   echo 'likely conda case'


It would be better to make this more clear. I think I understand why this is done, but I could be wrong. Add comments or a better string to help someone who comes along later to know why this check is here.

Also, it is better to do something like:

if [ $(command -v mkmf) ] then echo "mkmf already in PATH. Not adding \"${binDir}\" to PATH." else PATH=${binDir}:$PATH fi export PATH

Thinking about this more, I don't think you will want to do this check. This is a test to verify that the mkmf in the repository passes the tests. If mkmf is anywhere in PATH, even if the user adds their own copy, then the test will run that copy, and likely hide any issues the test will attempt to find. I strongly suggest removing this, unless you have a good reason otherwise.

underwoo · 2025-06-25T16:47:57Z

t/t001-list_paths.sh

+   # set PATH if needed
+   binDir=$(readlink -f ${BATS_TEST_DIRNAME}/../mkmf/bin)
+   do_we_have_mkmf=$(which mkmf) || echo "no we do not!"
+   if [ $do_we_have_mkmf ]; then


Same as above.

underwoo · 2025-06-25T16:50:35Z

t/t001-list_paths.sh

 }

 teardown() {
+   rm -f ${BATS_TEST_DIRNAME}/src/file6.f90


If this file is in testDir, then you won't need this specific file removal.

underwoo · 2025-06-25T16:51:46Z

t/t001-list_paths.sh

   [ "$status" -eq 0 ]
   [ -e path_names ]
-   [ "$(wc -l < path_names)" -eq 6 ]
+   num_paths=$(cat path_names | wc -l)


Consider making a function to do this check.

underwoo · 2025-06-25T16:52:00Z

t/t002-git-version-string.sh

+   # set PATH if needed
+   binDir=$(readlink -f ${BATS_TEST_DIRNAME}/../mkmf/bin)
+   do_we_have_mkmf=$(which mkmf) || echo "no we do not!"
+   if [ $do_we_have_mkmf ]; then


underwoo · 2025-06-25T16:52:49Z

t/t003-mkmf.sh

+   # set PATH if needed
+   binDir=$(readlink -f ${BATS_TEST_DIRNAME}/../mkmf/bin)
+   do_we_have_mkmf=$(which mkmf) || echo "no we do not!"
+   if [ $do_we_have_mkmf ]; then


underwoo · 2025-06-25T16:53:14Z

t/t004-templates.sh

+   # set PATH if needed
+   binDir=$(readlink -f ${BATS_TEST_DIRNAME}/../mkmf/bin)
+   do_we_have_mkmf=$(which mkmf) || echo "no we do not!"
+   if [ $do_we_have_mkmf ]; then


ilaflott and others added 2 commits January 22, 2025 17:05

ham-fisted packaging approach- put everything in nested mkmf folder, …

833a850

…define build recipe in meta.yaml and relevant fields. define conda environment.

build_conda workflow

7791ad0

move test dir up one level and adjust testing commands, source files …

523f8e6

…for test step. add git at dependency to see if it keeps the pipeline happy.

ilaflott marked this pull request as ready for review January 23, 2025 19:19

Merge branch 'main' into conda_package

a962698

ilaflott added 11 commits January 23, 2025 14:42

remove git dependency, add verbose run flag to bats testing calls. re…

691fce0

…move most unneeded failure-guarding shell commands.

update success condition in regex-based check for version-printing te…

a8bf0c1

…st t/t001-list_paths.sh

torturing the pipeline and the test to see if i can get more intel. t…

42e73a8

…he testing approach makes perfect sense locally- so why isnt the pipeline happy?

comment out most commands to test build- focus on certain things...

7db3edf

torture test more. somewhat suspicious of insonsistent comparisons be…

d8c1fb2

…tween string and ints in test success condition checks

another attempt... set shell variable to number of lines determined v…

4ad9e29

…ia cat and wc -l

remove failure guard and add a "no-parallelize-within-files" flag... …

c487371

…slowing this pipeline down for its own good!! (maybe...)

that option doesnt work for bats. specify only ONE job. add -x for tr…

0904c22

…acing.

what are you doing pipeline????

47f4e8b

what are you doing pipeline????

54007b8

ilaflott added 4 commits January 23, 2025 16:32

adjust and torture tests again

13712d6

adjust and torture tests again because i forgot to change an expected…

e4c20ab

… answer -_-

lets see what we have won...

76b3b32

ok- throwing a towel in to focus on other things

9fb952c

ilaflott requested review from ceblanton and underwoo January 23, 2025 22:16

ilaflott added 2 commits January 24, 2025 09:16

Update README.md

19aa357

put conda-relevant things in README- adjust markdown, update dependencies discussed. put disclaimer+license at the bottom of the readme to emphasize usage, installation, featurres etc.

Update README.md

1832388

update badge

ilaflott added 3 commits January 24, 2025 11:00

adjust (which mkmf) check in bats setup. lets try the git test in the…

a04500d

… pipeline again.

clean up commented out shell

53d15ab

have the setup function in t/t002-git-version-string configure the em…

25a5cc5

…pty repo for pipeline-compatibility

ilaflott requested a review from singhd789 January 24, 2025 16:35

ceblanton requested changes Jan 24, 2025

View reviewed changes

.github/workflows/build_conda.yml Outdated Show resolved Hide resolved

environment.yml Outdated Show resolved Hide resolved

meta.yaml Outdated Show resolved Hide resolved

remove noaa-gfdl channel where not needed. update tag

8dad61a

singhd789 reviewed Jan 24, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

singhd789 mentioned this pull request Jan 24, 2025

mkmf not included in installation NOAA-GFDL/fre-cli#229

Closed

ilaflott added 2 commits January 24, 2025 13:13

replace some_channel in README

bf0eb7f

forget the Jinja for now

616d2cf

ilaflott and others added 4 commits January 28, 2025 11:42

move the templates back to base of the repo directory, outside of the…

c79f958

… package directory.

Update README.md

6d7b5e1

slightly adjust current README instructions

Update publish_conda.yml

93e6c8f

test-run publish_conda workflow by asking it to run on push to `conda_package` branch, see what happens!

restore publish_conda workflow run condition. remove uneeded and old …

45fac6d

…gitlab/travis pipeline workflow files

ilaflott closed this May 8, 2025

ilaflott reopened this Jun 24, 2025

Merge branch 'main' into conda_package

fb1e646

singhd789 reviewed Jun 25, 2025

View reviewed changes

underwoo requested changes Jun 25, 2025

View reviewed changes

Conda package attempt #75

Are you sure you want to change the base?

Conda package attempt #75

Uh oh!

Conversation

ilaflott commented Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilaflott commented Jan 22, 2025

Uh oh!

ilaflott commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilaflott commented Jan 23, 2025

Uh oh!

ilaflott commented Jan 23, 2025

Uh oh!

ceblanton commented Jan 23, 2025

Uh oh!

ilaflott commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilaflott commented Jan 24, 2025

Uh oh!

ceblanton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ilaflott commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ceblanton commented Jan 24, 2025

Uh oh!

ilaflott commented Jan 24, 2025

Uh oh!

ilaflott commented Jan 24, 2025

Uh oh!

ilaflott commented Jan 24, 2025

Uh oh!

ceblanton commented Jan 24, 2025

Uh oh!

singhd789 commented Jan 24, 2025 • edited by ilaflott Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilaflott commented Jan 24, 2025

Uh oh!

ilaflott commented Jan 24, 2025

Uh oh!

ilaflott commented Jan 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

ilaflott commented Jan 22, 2025 •

edited

Loading

ilaflott commented Jan 23, 2025 •

edited

Loading

ilaflott commented Jan 23, 2025 •

edited

Loading

ilaflott commented Jan 24, 2025 •

edited

Loading

singhd789 commented Jan 24, 2025 •

edited by ilaflott

Loading