Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AWS Lambda Fails On Require #1885

Closed
DeanBDean opened this issue Sep 24, 2019 · 8 comments
Closed

AWS Lambda Fails On Require #1885

DeanBDean opened this issue Sep 24, 2019 · 8 comments
Labels
installation

Comments

@DeanBDean
Copy link

Hi,

I apologize in advance if I'm missing something simple, but I've read through a bunch of the issues that were similar to this one, and the solutions did not fix my issue. I am using sharp in an AWS Lambda, which I am packing up on a Mac. Whenever the Lambda runs, I get this error when the Lambda attempts to require('sharp')

Something went wrong installing the "sharp" module

libvips-cpp.so.42: cannot open shared object file: No such file or directory

- Remove the "node_modules/sharp" directory, run "npm install" and look for errors
- Consult the installation documentation at https://sharp.pixelplumbing.com/en/stable/install/
- Search for this error at https://github.com/lovell/sharp/issues

I have tried running the npm install with all the options mentioned in the installation guide, and using the docker run command. Both result in this same error message. I am able to use Sharp locally on my Mac with a docker run command of docker run -v "$PWD":/var/task lambci/lambda:build-nodejs10.x node index2.js where index2.js is just a little test file. I am on macOS 10.14.6 and using node10.x on the Lambda.

Did you see the documentation relating to installation?

Yes, I have followed the instructions for setting up sharp for a Lambda function with the npm install and zip occuring on a Mac

Have you ensured the platform and version of Node.js used for npm install is the same as the platform and version of Node.js used at runtime?

Yes, I tried both options available in the installation instructions

If you're (mis)using sudo npm install have you tried with the sudo npm install --unsafe-perm flag?

What is the complete output of running npm install --verbose sharp? Have you checked this output for useful error messages?

I don't see any errors on the npm install

What is the output of running npx envinfo --binaries --languages --system --utilities?

Would this be useful to run in the AWS Lambda?
@lovell
Copy link
Owner

lovell commented Sep 24, 2019

If this problem started within the last 24 hours then please see #1882 and check any lockfiles for broken versions of minipass prior to v2.8.5.

@DeanBDean
Copy link
Author

DeanBDean commented Sep 24, 2019 •
edited
Loading

I'm using vanilla npm, so my package-lock.json has the following minipass references. I believe they are either up to 2.8.5 or before the issues mentioned in #1882

EDIT: The only thing I have installed besides sharp is lodash.get, which has no dependencies

"tar": {
      "version": "4.4.11",
      "resolved": "http://verdaccio.<redacted>.com:4873/tar/-/tar-4.4.11.tgz",
      "integrity": "sha512-iI4zh3ktLJKaDNZKZc+fUONiQrSn9HkCFzamtb7k8FFmVilHVob7QsLX/VySAW8lAviMzMbFw4QtFb4errwgYA==",
      "requires": {
        "chownr": "^1.1.1",
        "fs-minipass": "^1.2.5",
        "minipass": "^2.6.4",
        "minizlib": "^1.2.1",
        "mkdirp": "^0.5.0",
        "safe-buffer": "^5.1.2",
        "yallist": "^3.0.3"
      }
    }

"fs-minipass": {
      "version": "1.2.7",
      "resolved": "http://verdaccio.<redacted>.com:4873/fs-minipass/-/fs-minipass-1.2.7.tgz",
      "integrity": "sha512-GWSSJGFy4e9GUeCcbIkED+bgAoFyj7XF1mV8rma3QW4NIqX9Kyx79N/PF61H5udOV3aY1IaMLs6pGbH71nlCTA==",
      "requires": {
        "minipass": "^2.6.0"
      }
    },

"minipass": {
      "version": "2.8.5",
      "resolved": "http://verdaccio.<redacted>.com:4873/minipass/-/minipass-2.8.5.tgz",
      "integrity": "sha512-D5+szmZBoOAfbLjOXY4Ve5bWylyTdrUOmbJPgYmgTF5ovCnCFFTY+I16Ccm/UjSNNAekXtIEDvoCDioFzRz18Q==",
      "requires": {
        "safe-buffer": "^5.1.2",
        "yallist": "^3.0.0"
      }
    },

"minizlib": {
      "version": "1.2.2",
      "resolved": "http://verdaccio.<redacted>.com:4873/minizlib/-/minizlib-1.2.2.tgz",
      "integrity": "sha512-hR3At21uSrsjjDTWrbu0IMLTpnkpv8IIMFDFaoz43Tmu4LkmAXfH44vNNzpTnf+OAQQCHrb91y/wc2J4x5XgSQ==",
      "requires": {
        "minipass": "^2.2.1"
      }
    },

@lovell
Copy link
Owner

lovell commented Sep 24, 2019

Great, thanks for confirming.

In the zip file you upload to Lambda, what are the contents of node_modules/sharp/vendor/platform.json and what is the listing of the node_modules/sharp/vendor/lib directory?

@DeanBDean
Copy link
Author

The contents of platform.json are "linux-x64"

Here is a ls -l for that lib directory

drwxr-xr-x  4 <redacted>  staff      128 Sep 24 10:39 cmake
drwxr-xr-x  3 <redacted>  staff       96 Sep 24 10:39 gdk-pixbuf-2.0
drwxr-xr-x  3 <redacted>  staff       96 Sep 24 10:39 gio
drwxr-xr-x  3 <redacted>  staff       96 Sep 24 10:39 glib-2.0
lrwxr-xr-x  1 <redacted>  staff       29 Sep 24 10:39 libcairo-gobject.so -> libcairo-gobject.so.2.11702.0
lrwxr-xr-x  1 <redacted>  staff       29 Sep 24 10:39 libcairo-gobject.so.2 -> libcairo-gobject.so.2.11702.0
-rwxr-xr-x  1 <redacted>  staff    34728 Jul 27 03:36 libcairo-gobject.so.2.11702.0
lrwxr-xr-x  1 <redacted>  staff       21 Sep 24 10:39 libcairo.so -> libcairo.so.2.11702.0
lrwxr-xr-x  1 <redacted>  staff       21 Sep 24 10:39 libcairo.so.2 -> libcairo.so.2.11702.0
-rwxr-xr-x  1 <redacted>  staff  1066176 Jul 27 03:36 libcairo.so.2.11702.0
lrwxr-xr-x  1 <redacted>  staff       21 Sep 24 10:39 libcroco-0.6.so -> libcroco-0.6.so.3.0.1
lrwxr-xr-x  1 <redacted>  staff       21 Sep 24 10:39 libcroco-0.6.so.3 -> libcroco-0.6.so.3.0.1
-rwxr-xr-x  1 <redacted>  staff   256504 Jul 27 03:38 libcroco-0.6.so.3.0.1
lrwxr-xr-x  1 <redacted>  staff       17 Sep 24 10:39 libexif.so -> libexif.so.12.3.3
lrwxr-xr-x  1 <redacted>  staff       17 Sep 24 10:39 libexif.so.12 -> libexif.so.12.3.3
-rwxr-xr-x  1 <redacted>  staff   281448 Jul 27 03:28 libexif.so.12.3.3
lrwxr-xr-x  1 <redacted>  staff       17 Sep 24 10:39 libexpat.so -> libexpat.so.1.6.9
lrwxr-xr-x  1 <redacted>  staff       17 Sep 24 10:39 libexpat.so.1 -> libexpat.so.1.6.9
-rwxr-xr-x  1 <redacted>  staff   264152 Jul 27 03:32 libexpat.so.1.6.9
drwxr-xr-x  3 <redacted>  staff       96 Sep 24 10:39 libffi-3.2.1
lrwxr-xr-x  1 <redacted>  staff       15 Sep 24 10:39 libffi.so -> libffi.so.6.0.4
lrwxr-xr-x  1 <redacted>  staff       15 Sep 24 10:39 libffi.so.6 -> libffi.so.6.0.4
-rwxr-xr-x  1 <redacted>  staff    35136 Jul 27 03:25 libffi.so.6.0.4
lrwxr-xr-x  1 <redacted>  staff       23 Sep 24 10:39 libfontconfig.so -> libfontconfig.so.1.12.0
lrwxr-xr-x  1 <redacted>  staff       23 Sep 24 10:39 libfontconfig.so.1 -> libfontconfig.so.1.12.0
-rwxr-xr-x  1 <redacted>  staff   330624 Jul 27 03:32 libfontconfig.so.1.12.0
lrwxr-xr-x  1 <redacted>  staff       21 Sep 24 10:39 libfreetype.so -> libfreetype.so.6.17.1
lrwxr-xr-x  1 <redacted>  staff       21 Sep 24 10:39 libfreetype.so.6 -> libfreetype.so.6.17.1
-rwxr-xr-x  1 <redacted>  staff   881360 Jul 27 03:32 libfreetype.so.6.17.1
lrwxr-xr-x  1 <redacted>  staff       19 Sep 24 10:39 libfribidi.so -> libfribidi.so.0.4.0
lrwxr-xr-x  1 <redacted>  staff       19 Sep 24 10:39 libfribidi.so.0 -> libfribidi.so.0.4.0
-rwxr-xr-x  1 <redacted>  staff   116952 Jul 27 03:37 libfribidi.so.0.4.0
lrwxr-xr-x  1 <redacted>  staff       29 Sep 24 10:39 libgdk_pixbuf-2.0.so -> libgdk_pixbuf-2.0.so.0.3612.0
lrwxr-xr-x  1 <redacted>  staff       29 Sep 24 10:39 libgdk_pixbuf-2.0.so.0 -> libgdk_pixbuf-2.0.so.0.3612.0
-rwxr-xr-x  1 <redacted>  staff   180200 Jul 27 03:31 libgdk_pixbuf-2.0.so.0.3612.0
lrwxr-xr-x  1 <redacted>  staff       15 Sep 24 10:39 libgif.so -> libgif.so.7.0.0
lrwxr-xr-x  1 <redacted>  staff       15 Sep 24 10:39 libgif.so.7 -> libgif.so.7.0.0
-rwxr-xr-x  1 <redacted>  staff    39200 Jul 27 03:41 libgif.so.7.0.0
lrwxr-xr-x  1 <redacted>  staff       22 Sep 24 10:39 libgio-2.0.so -> libgio-2.0.so.0.5600.4
lrwxr-xr-x  1 <redacted>  staff       22 Sep 24 10:39 libgio-2.0.so.0 -> libgio-2.0.so.0.5600.4
-rwxr-xr-x  1 <redacted>  staff  1783792 Jul 27 03:27 libgio-2.0.so.0.5600.4
lrwxr-xr-x  1 <redacted>  staff       23 Sep 24 10:39 libglib-2.0.so -> libglib-2.0.so.0.5600.4
lrwxr-xr-x  1 <redacted>  staff       23 Sep 24 10:39 libglib-2.0.so.0 -> libglib-2.0.so.0.5600.4
-rwxr-xr-x  1 <redacted>  staff  1401784 Jul 27 03:26 libglib-2.0.so.0.5600.4
lrwxr-xr-x  1 <redacted>  staff       26 Sep 24 10:39 libgmodule-2.0.so -> libgmodule-2.0.so.0.5600.4
lrwxr-xr-x  1 <redacted>  staff       26 Sep 24 10:39 libgmodule-2.0.so.0 -> libgmodule-2.0.so.0.5600.4
-rwxr-xr-x  1 <redacted>  staff    14552 Jul 27 03:26 libgmodule-2.0.so.0.5600.4
lrwxr-xr-x  1 <redacted>  staff       26 Sep 24 10:39 libgobject-2.0.so -> libgobject-2.0.so.0.5600.4
lrwxr-xr-x  1 <redacted>  staff       26 Sep 24 10:39 libgobject-2.0.so.0 -> libgobject-2.0.so.0.5600.4
-rwxr-xr-x  1 <redacted>  staff   338912 Jul 27 03:26 libgobject-2.0.so.0.5600.4
lrwxr-xr-x  1 <redacted>  staff       20 Sep 24 10:39 libgsf-1.so -> libgsf-1.so.114.0.46
lrwxr-xr-x  1 <redacted>  staff       20 Sep 24 10:39 libgsf-1.so.114 -> libgsf-1.so.114.0.46
-rwxr-xr-x  1 <redacted>  staff   290216 Jul 27 03:28 libgsf-1.so.114.0.46
lrwxr-xr-x  1 <redacted>  staff       26 Sep 24 10:39 libgthread-2.0.so -> libgthread-2.0.so.0.5600.4
lrwxr-xr-x  1 <redacted>  staff       26 Sep 24 10:39 libgthread-2.0.so.0 -> libgthread-2.0.so.0.5600.4
-rwxr-xr-x  1 <redacted>  staff     6024 Jul 27 03:26 libgthread-2.0.so.0.5600.4
lrwxr-xr-x  1 <redacted>  staff       24 Sep 24 10:39 libharfbuzz.so -> libharfbuzz.so.0.20503.0
lrwxr-xr-x  1 <redacted>  staff       24 Sep 24 10:39 libharfbuzz.so.0 -> libharfbuzz.so.0.20503.0
-rwxr-xr-x  1 <redacted>  staff  1078720 Jul 27 03:34 libharfbuzz.so.0.20503.0
lrwxr-xr-x  1 <redacted>  staff       12 Sep 24 10:39 libjpeg.so -> libjpeg.so.8
lrwxr-xr-x  1 <redacted>  staff       16 Sep 24 10:39 libjpeg.so.8 -> libjpeg.so.8.2.2
-rwxr-xr-x  1 <redacted>  staff   600712 Jul 27 03:29 libjpeg.so.8.2.2
lrwxr-xr-x  1 <redacted>  staff       17 Sep 24 10:39 liblcms2.so -> liblcms2.so.2.0.8
lrwxr-xr-x  1 <redacted>  staff       17 Sep 24 10:39 liblcms2.so.2 -> liblcms2.so.2.0.8
-rwxr-xr-x  1 <redacted>  staff   490616 Jul 27 03:29 liblcms2.so.2.0.8
lrwxr-xr-x  1 <redacted>  staff       20 Sep 24 10:39 liborc-0.4.so -> liborc-0.4.so.0.28.0
lrwxr-xr-x  1 <redacted>  staff       20 Sep 24 10:39 liborc-0.4.so.0 -> liborc-0.4.so.0.28.0
-rwxr-xr-x  1 <redacted>  staff   635808 Jul 27 03:31 liborc-0.4.so.0.28.0
lrwxr-xr-x  1 <redacted>  staff       24 Sep 24 10:39 libpango-1.0.so -> libpango-1.0.so.0.4200.4
lrwxr-xr-x  1 <redacted>  staff       24 Sep 24 10:39 libpango-1.0.so.0 -> libpango-1.0.so.0.4200.4
-rwxr-xr-x  1 <redacted>  staff   310416 Jul 27 03:38 libpango-1.0.so.0.4200.4
lrwxr-xr-x  1 <redacted>  staff       29 Sep 24 10:39 libpangocairo-1.0.so -> libpangocairo-1.0.so.0.4200.4
lrwxr-xr-x  1 <redacted>  staff       29 Sep 24 10:39 libpangocairo-1.0.so.0 -> libpangocairo-1.0.so.0.4200.4
-rwxr-xr-x  1 <redacted>  staff    56632 Jul 27 03:38 libpangocairo-1.0.so.0.4200.4
lrwxr-xr-x  1 <redacted>  staff       27 Sep 24 10:39 libpangoft2-1.0.so -> libpangoft2-1.0.so.0.4200.4
lrwxr-xr-x  1 <redacted>  staff       27 Sep 24 10:39 libpangoft2-1.0.so.0 -> libpangoft2-1.0.so.0.4200.4
-rwxr-xr-x  1 <redacted>  staff    85856 Jul 27 03:38 libpangoft2-1.0.so.0.4200.4
lrwxr-xr-x  1 <redacted>  staff       21 Sep 24 10:39 libpixman-1.so -> libpixman-1.so.0.38.4
lrwxr-xr-x  1 <redacted>  staff       21 Sep 24 10:39 libpixman-1.so.0 -> libpixman-1.so.0.38.4
-rwxr-xr-x  1 <redacted>  staff   993976 Jul 27 03:35 libpixman-1.so.0.38.4
lrwxr-xr-x  1 <redacted>  staff       11 Sep 24 10:39 libpng.so -> libpng16.so
lrwxr-xr-x  1 <redacted>  staff       19 Sep 24 10:39 libpng16.so -> libpng16.so.16.37.0
lrwxr-xr-x  1 <redacted>  staff       19 Sep 24 10:39 libpng16.so.16 -> libpng16.so.16.37.0
-rwxr-xr-x  1 <redacted>  staff   252976 Jul 27 03:29 libpng16.so.16.37.0
lrwxr-xr-x  1 <redacted>  staff       19 Sep 24 10:39 librsvg-2.so -> librsvg-2.so.2.46.0
lrwxr-xr-x  1 <redacted>  staff       19 Sep 24 10:39 librsvg-2.so.2 -> librsvg-2.so.2.46.0
-rwxr-xr-x  1 <redacted>  staff  6953912 Jul 27 03:41 librsvg-2.so.2.46.0
lrwxr-xr-x  1 <redacted>  staff       16 Sep 24 10:39 libtiff.so -> libtiff.so.5.4.0
lrwxr-xr-x  1 <redacted>  staff       16 Sep 24 10:39 libtiff.so.5 -> libtiff.so.5.4.0
-rwxr-xr-x  1 <redacted>  staff   561344 Jul 27 03:30 libtiff.so.5.4.0
lrwxr-xr-x  1 <redacted>  staff       22 Sep 24 10:39 libvips-cpp.so -> libvips-cpp.so.42.11.0
lrwxr-xr-x  1 <redacted>  staff       22 Sep 24 10:39 libvips-cpp.so.42 -> libvips-cpp.so.42.11.0
-rwxr-xr-x  1 <redacted>  staff   216264 Jul 27 03:43 libvips-cpp.so.42.11.0
lrwxr-xr-x  1 <redacted>  staff       18 Sep 24 10:39 libvips.so -> libvips.so.42.11.0
lrwxr-xr-x  1 <redacted>  staff       18 Sep 24 10:39 libvips.so.42 -> libvips.so.42.11.0
-rwxr-xr-x  1 <redacted>  staff  4141032 Jul 27 03:43 libvips.so.42.11.0
lrwxr-xr-x  1 <redacted>  staff       16 Sep 24 10:39 libwebp.so -> libwebp.so.7.0.5
lrwxr-xr-x  1 <redacted>  staff       16 Sep 24 10:39 libwebp.so.7 -> libwebp.so.7.0.5
-rwxr-xr-x  1 <redacted>  staff   666152 Jul 27 03:30 libwebp.so.7.0.5
lrwxr-xr-x  1 <redacted>  staff       21 Sep 24 10:39 libwebpdemux.so -> libwebpdemux.so.2.0.6
lrwxr-xr-x  1 <redacted>  staff       21 Sep 24 10:39 libwebpdemux.so.2 -> libwebpdemux.so.2.0.6
-rwxr-xr-x  1 <redacted>  staff    18416 Jul 27 03:30 libwebpdemux.so.2.0.6
lrwxr-xr-x  1 <redacted>  staff       19 Sep 24 10:39 libwebpmux.so -> libwebpmux.so.3.0.4
lrwxr-xr-x  1 <redacted>  staff       19 Sep 24 10:39 libwebpmux.so.3 -> libwebpmux.so.3.0.4
-rwxr-xr-x  1 <redacted>  staff    43216 Jul 27 03:30 libwebpmux.so.3.0.4
lrwxr-xr-x  1 <redacted>  staff       16 Sep 24 10:39 libxml2.so -> libxml2.so.2.9.9
lrwxr-xr-x  1 <redacted>  staff       16 Sep 24 10:39 libxml2.so.2 -> libxml2.so.2.9.9
-rwxr-xr-x  1 <redacted>  staff   881240 Jul 27 03:28 libxml2.so.2.9.9
lrwxr-xr-x  1 <redacted>  staff       14 Sep 24 10:39 libz.so -> libz.so.1.2.11
lrwxr-xr-x  1 <redacted>  staff       14 Sep 24 10:39 libz.so.1 -> libz.so.1.2.11
-rwxr-xr-x  1 <redacted>  staff   113264 Jul 27 03:25 libz.so.1.2.11
-rw-r--r--  1 <redacted>  staff      220 Jul 27 03:28 xml2Conf.sh

@DeanBDean
Copy link
Author

When I unzipped the files, I did notice this error

warning:  deferred symlink (node_modules/sharp/vendor/include/png.h) failed:
          invalid placeholder file

Is it possible I am zipping in a way that is messing up symbolic links?

@lovell
Copy link
Owner

lovell commented Sep 24, 2019

Yes, broken symlinks sounds like a plausible reason for this error. Everything else you've pasted here looks OK.

@DeanBDean
Copy link
Author

So I finally got to the bottom of it, and it turns out it was unrelated to your library. So I apologize for using up your time, and appreciate you helping me get to the bottom of it. I was using terraform to zip the Lambda contents, but the archive_file provider from terraform has an old, nasty bug where it does not handle symlinks correctly.

hashicorp/terraform-provider-archive#6 (comment)

For anyone in the future who runs into this as a problem, I used the above to workaround terraform's inability to zip files with symlinks

@lovell
Copy link
Owner

lovell commented Sep 25, 2019

Glad to hear you got it working.

If you're able, I'm sure users of terraform-provider-archive would welcome a PR to either fix this problem or a PR to clearly document this limitation in the currently-blank "Fill in for each provider" section of its README.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
installation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lovell @DeanBDean