GLIBC_2.27 not found

[samhh]

Have I misinterpreted the instructions regarding viability of local invocation or is this a bug?:

$ sam local invoke
<home>/.local/lib/python3.9/site-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.3) or chardet (3.0.4) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
Invoking NOT_USED (provided)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-provided:rapid-1.17.0.

Mounting <project>/.stack-work/docker/_home/.local/bin as /var/task:ro,delegated inside runtime container
START RequestId: 5495b4de-5771-4cf8-b15c-47aa02989eb9 Version: $LATEST
/var/task/bootstrap: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/bootstrap)
time="2021-02-07T17:22:47.318" level=error msg="Init failed" InvokeID= error="Runtime exited with error: exit status 1"
time="2021-02-07T17:22:47.318" level=error msg="INIT DONE failed: Runtime.ExitError"

This is on Arch with everything up to date. The build appears to succeed and stack run gives no such error (just missing AWS environment as you'd expect).

[IamfromSpace]

You should be able to run locally, just like you did with “sam local invoke.”

My shot in the dark guess is that the build environment has a later version of GLIBC than sam local’s emulator image based on this SO answer:

In your case, you built on a newer (GLIBC-2.28) system, and are trying to run on an older one (GLIBC-2.27). That is not guaranteed to work (although it might for sufficiently simple programs).

Can you tell me more about how you built the bootstrap executable? Which stack LTS (or otherwise) are you using? Are you building inside a docker image? If so which one?

Hopefully we can track it down quickly.

[samhh]

Here's my stack.yaml:

resolver: lts-16.31

extra-deps:
  - envy-1.5.1.0
  - hal-0.4.6
  - megaparsec-9.0.1

docker:
  enable: true

Built with stack build --copy-bins. Doing that, I can see .stack-work/docker/_home/.local/bin/bootstrap, which my template points to.

I'm not interacting with Docker at all beyond the configuration flag to Stack above.

If it helps, some versions on my host machine:

• SAM: 1.17.0
• Stack: 2.5.1
• glibc: 2.33

[IamfromSpace]

All that makes sense, in that case the build image should be fpco/stack-build:lts-16.31 as I think it will default to that unless you explicitly specify one (I think that will print to console when building, so you can sanity check it if you’d like)

I think the next question is what version of glibc is on that image. The host machine shouldn’t matter based on the config you’ve shown.

Can you try:

docker run fpco/stack-build:lts-16.31 ldd --version

I can give it a go myself later, but I’m restricted to my phone for the moment.

[samhh]

$ docker run fpco/stack-build:lts-16.31 ldd --version                                                                                   ~
ldd (Ubuntu GLIBC 2.27-3ubuntu1.2) 2.27
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

[samhh]

As a sanity check I've deleted dist-newstyle and .stack-work and rerun, no change.

Is this part of the sam error relevant?

Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-provided:rapid-1.17.0

[samhh]

stack build --verbose printout includes the string "fpco/stack-build:lts-16.31" so that's probably being used as expected.

[IamfromSpace]

Is this part of the sam error relevant?

Ah, yes, so that’s the image that the lambda is then running in, which sam local is selecting automatically. We should then run the same command for that image and see if it’s a mismatch—specifically if it’s a lower version.

[samhh]

$ docker run amazon/aws-sam-cli-emulation-image-provided:rapid-1.17.0 ldd --version                                                     ~
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

[samhh]

Tried this just in case:

$ docker pull amazon/aws-sam-cli-emulation-image-provided                                                                               ~
Using default tag: latest
latest: Pulling from amazon/aws-sam-cli-emulation-image-provided
Digest: sha256:8784f36ae6f73a78479711556b4dac67c3c956df3707a46d45483a2d701ef4d2
Status: Image is up to date for amazon/aws-sam-cli-emulation-image-provided:latest
docker.io/amazon/aws-sam-cli-emulation-image-provided:latest

[IamfromSpace]

Ah, yeah, that’s very likely our culprit then. Thank you for checking all this btw!

For next steps, we’ll need to see about setting up an image that’s a better match. First, to validate the issue, and second to get some guidance and docs on how to resolve it. That’s unfortunate though, definitely an unfun wrinkle.

I think the best bet is to use the amazon image as a base and then add stack. Trying to downgrade the fpco image probably just leaves the door open for the next mismatch.

I’ve been steadily working to get hal more broadly compatible and get it into current stack LTSes, so definitely a problem I’ll be prioritizing. If you want to continue working on it and tag team a branch, the help would of course be welcome!

[IamfromSpace]

Also worth noting that if you just need a quick fix and you can use an older stack LTS, you might try downgrading. I haven’t seen this issue before with older LTS images.

Not a recommendation I’m happy with, haha, but if it unblocks you, it might be worth it until there’s a cleaner solve.

[IamfromSpace]

Much downloading later I've (unsurprisingly) replicated this issue. I was able to create a Dockerfile that seems to build with the same version of glibc. This seems to get beyond that error and execute the binary. I'm getting some new error in decoding Context, but I'm not totally sure yet why.

You can try it out on this branch #fix/sam-local-glibc-version-error now.

1. build the image via docker build -t "${NAME}:${TAG}" . from the root
2. Add image: ${NAME}:${TAG} to your docker settings in stack.yaml
3. stack build

Still needs some digging and then quite a bit of polish, but some progress.

[samhh]

Really appreciate your support on this mate, cheers.

Yeah, I'd be perfectly content to downgrade my Stack LTS for now, do you have a known good version I can target?

[IamfromSpace]

I’m beginning to think this is more an issue with sam/its version rather than the stack image.

My Linux host has glibc 2.23 and runs just fine on actual lambda when compiled without docker.

On MacOS, it seems to work without issue with lts-13.22, where the stack build also uses glibc 2.23. I seem to be using an older lambda image, back from lambci/lambda:provided.

Since the current Amazon images only use 2.17, it doesn’t seem like a downgrade does the trick. I’ll dig more into sam though and see what I find. Definitely a priority to support its newest versions.

[IamfromSpace]

Hmm, so unfortunately there seems to be a string of issues. I don't think sam has been supported since 1.0--though I'm not sure on the exact timeline. Even if the glibc issue where resolved (which looks a bit more complicated, due to the docker images being built somewhat dynamically), then sam currently doesn't pass in all the seemingly required values to build the context.

Notably, the officially supported Rust runtime seems to have the same issues and not support sam after 1.0, so there seems to be a bit of the right hand not knowing what the left is doing. Other users have a similar glibc issue #awslabs/aws-lambda-rust-runtime#17. And both this and the Rust runtime expect that the traceId, functionArn, and deadline are passed in as headers, but sam does not send them. I'm not sure they've even realized that shortcoming yet. For both, it would be a breaking change or special branching when running locally.

Notably, Rust's docs don't mention sam, and steer users instead to directly use the image that sam<1.0 used to use. I think for the time being, this probably needs to recommend the same while chipping away at some of the above issues. Here's their README section on docker usage: https://github.com/awslabs/aws-lambda-rust-runtime#docker

You should be able to use the following to as a local invocation:

echo $MY_EVENT_JSON | docker run -i \
    -e DOCKER_LAMBDA_USE_STDIN=1 \
    --rm \
    -v ${PATH_TO_UNZIPPED_BOOTSTRAP_DIR}:/var/task \
    lambci/lambda:provided