Large amount of locales/data enabled in contentful case "JavaScript heap out of memory"

[stoltzrobin]

Summary

Enable the user to filter out unnecessary locales from Contentful.

Basic example

 {
      resolve: `gatsby-source-contentful`,
      options: {
        [... other settings needed]
        localeFilter: locale =>
          locale.code === ACTIVE_LOCALE || locale.code === DEFAULT_LOCALE,
      },
},

By passing a filter function we can limit the amount of nodes created in graphQL.

Motivation

In our case we have over 50 locales in Contentful and over 2500 entries (this numbers are increasing every week as well). This create a lot of nodes in the Contentful plugin, which case the JS VM instance to run our of memory. In our application we only build one locale per build (to improve build speed) and all other nodes from different locales an useless and not needed.

In our application it reduces the JS VM runtime memory around 80%.

[wardpeet]

Thanks for filling an issue and fixing it yourself! 💪 that's just bonkers!

Could you provide a bit more info about where it fails exactly? Just making sure it's not a known issue with saving redux state as we have an issue open for sites with lots of pages.

#12343

Would be handy if it's the same issue that you can comment in that thread as well.

[thibautRe]

I can provide more info (Robin and I work together, and he's on holiday as of now ^{I'm super jealous btw! :D} so he might not be able to answer until he's back).

The errors we started to experience randomly are memory issues (not speed), and would lead to our JS runtime running out of memory and throwing a wonderful C++ error. Using --max-old-space-size was helping us for local development but were not fixing the issue for Cloud builds - where the machines have limited amount of memory.

The OutOfMemoryError would get thrown randomly - sometimes not at all, and at random times... Most of the time the error would happen between the node sourcing and the graphql schema generation steps. I don't remember us seeing the error during the graphql queries step, or HTML generation step.

Also, we tried using the lokijs feature flag but that didn't help.

[gatsbot]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[thibautRe]

Fixed in #12939