Gatsby build is timing out in production

[drewbietron]

Description

Gatsby build is timing out in production and is hanging on success write out redirect data. I am using the gatsby-source-s3 plugin to load images externally from S3. Currently there are about 100 images, all around 8-15mb a piece. I've seen multiple file download issues with either got or with gatsby-source-filesystem (#4293, #6385, #4666, #7855, #6654) and I believe the issue is due to it not allowing enough time for the files to download, as well as deleting the cache directory with each production build so that every S3 image must be downloaded during every build.

I am able to run gatsby build locally and it succeeds after about 150-200 seconds.

Steps to reproduce

Init a new gatsby project and include the gatsby-source-s3 plugin with a query for a ~100, 10mb images. I would post my own project but it is linked to a private S3 bucket.

Environment

System:
    OS: macOS High Sierra 10.13.3
    CPU: (8) x64 Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
    Shell: 5.0.7 - /opt/boxen/homebrew/bin/zsh
  Binaries:
    Node: 8.11.4 - /usr/local/bin/node
    Yarn: 1.12.3 - /usr/local/bin/yarn
    npm: 5.6.0 - /usr/local/bin/npm
  Browsers:
    Chrome: 71.0.3578.98
    Firefox: 57.0
    Safari: 11.0.3
  npmPackages:
    gatsby: 2.0.73 => 2.0.73
    gatsby-image: ^2.0.15 => 2.0.25
    gatsby-plugin-google-analytics: 2.0.8 => 2.0.8
    gatsby-plugin-google-fonts: ^0.0.4 => 0.0.4
    gatsby-plugin-manifest: 2.0.12 => 2.0.12
    gatsby-plugin-offline: 2.0.20 => 2.0.20
    gatsby-plugin-react-helmet: 3.0.4 => 3.0.4
    gatsby-plugin-sass: ^2.0.0-rc.2 => 2.0.7
    gatsby-plugin-sharp: ^2.0.7 => 2.0.15
    gatsby-plugin-sitemap: ^2.0.1 => 2.0.3
    gatsby-plugin-typescript: ^2.0.0 => 2.0.3
    gatsby-source-filesystem: ^2.0.0 => 2.0.12
    gatsby-source-s3: 0.0.0 => 0.0.0
    gatsby-transformer-sharp: ^2.1.4 => 2.1.9
  npmGlobalPackages:
    gatsby-cli: 2.4.6

[sidharthachatterjee]

Hey @drewbietron

Thank you for opening this.

Does the CI where it is running have some timeout configured?

Since it is running fine locally, I don't think it is timing out due to internal code but rather because of the environment. Can you share you some details about the environment that the production build is running on?

[drewbietron]

@sidharthachatterjee - Thanks for the reply. The site is built via Netlify with their default config.

[DSchau]

I believe Netlify's default timeout is ~15 minutes or so. That seems odd that there's such a difference between local (2-4 minutes or so?) vs. Netlify where it seems to be taking longer than 15 minutes.

Can you link to the Netlify build logs? Or paste them?

[drewbietron]

Turns out my public/static assets was very large since I was returning a local file in order to run image transformations on the images fetched from S3. The deploy times out due to it trying to upload a huge static folder.

I’ll need to figure out an updated build flow that uploads the assets to S3 or a cdn instead of returning them to the build’s static directory. Does a flow like that already exist with any Gastby plugins?

[gatsbot]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

Thanks again for being part of the Gatsby community!

[tito300]

hey, I'm facing the exact same problem with a ~100 photos from S3 processed through sharp. Except that mine is not timing out and just keeps running for hours. how were you able to fix this issue? note that I'm only running a development build so far.

[tito300]

Somehow my problem was fixed (randomly?). This is the steps I took, I created a new s3 bucket with fewer pictures (for testing) and then tried building and It built successfully very quicly. Then I decided to go back and try to pull from the original bucket and now all the sudden it built successfully in 49s when originally it would go on for hours. I don't know how the mere switch in bucket links fixed the stall but hope this helps someone figure it out. maybe it has to do with the cache?

[LpmRaven]

I'm having a similar issue. I have ~2000 images in an S3 bucket. When I build the production version it is timing out, I increased the memory and CPU of the build server but that didn't work. The website is for e-commerce and the number of images is only going to increase which makes me think I am not going to be able to use gatsby to process the images.

I welcome any suggestions.

[vulcanr]

I'm facing the same issue with ~1500 images... I just get localFile null