potential conflicts with cachebusted filenames?

[fool]

Hi folks,

I work on the Netlify tech support team. We recently had a customer whose ~2000 page build had reached ~140,000 pages due to your plugin. This site had grown so large that it was causing deploys to fail due to size. While I didn't do any further debugging, below I list the observed symptoms. His normally-~400MB site was swollen in size to 6GB. Removing your plugin caused things to return to normal.

• many copies of files with very similar names. Some might have identical contents, most changed in size and thus content.
• variable numbers of copies of various files. I'm guessing that the "multiplication" effect was consistent but files added at different times perhaps, so "older" files had more copies...

Anyway, it'd be great if you didn't try to cache changing filenames. Here's an example subset. Format is name - size

/35-05ff18b67d9ad4c3412c.js - 584440
/35-0eb8ffb7b74a627c25bf.js - 581606
/35-15108f8ebaa0bcaa2cf4.js - 584418
/35-15c2b9288cda436e3b25.js - 563998
/35-161ae57d3142c0cfed13.js - 582211
/35-215a7610964a3caebffe.js - 584419
/35-2324c7b8821ad86eb001.js - 582211
/35-2b6f5139c1dc2c5bb51f.js - 584441
/35-3138b8d2dc58744d7bde.js - 557872
/35-3ce74874fa7e20489a1d.js - 581279
/35-5484463bdcf9b6ddd250.js - 582212
/35-57e3ec947114e8f715e4.js - 581564
/35-5ce0ab9ba59dad89cd9c.js - 575976
/35-6786aa145a2494758293.js - 582193
/35-6ebd2f59ca6b41623f2e.js - 584724
/35-6fa7957eb94a77d6f5bd.js - 584441
/35-97511072e9a32223305c.js - 577177
/35-a63cb760595d80e9bee6.js - 577142
/35-b373a559b2ed30743ff6.js - 573563
/35-c93879da713a8da3e38e.js - 563949
/35-d636a90a55e30d35d631.js - 581561
/35-dc3c5f01b11408371ccc.js - 557976
/35-e4097cddcff2cb72f27b.js - 581511
/35-ee3b37869a96144afd85.js - 557587
/35-f524133b53e8d8b8e92f.js - 584770
/35-fa448180a421ada90f6a.js - 584416
/35-fdee0b5d441cfa8d05f2.js - 580441

while there is only one copy of the file "34-HASH.js".

I don't have good repro steps, but hopefully that gives you enough insight and maybe that customer will speak up here with more details about their setup.

[axe312ger]

I'll reach out to the Gatsby community to see if we can find a way for a better cache bust.

As far as I understand Gatsby v2, it cleans the public and .cache dir on it's own after a build. So there might be no reason at all to keep the cached files from the last build and just cache the latest version of the directory

[azamatsmith]

I can confirm that I am experiencing the same problem.

[axe312ger]

I just released v1.2.0-beta.0 which will now no more cache the public directory. You can use the new cacheExtraDirs option if you want to cache specific folders within your public directory.

Please check it out via npm i gatsby-plugin-netlify-cache@beta and let me know if this helps to solve your issues :)

[azamatsmith]

Thanks @axe312ger! I think that this will solve my issues.

[kentcdodds]

Would be sweet if a solution can be made to caching the public directory without over-caching because my build's bottleneck is gatsby-plugin-sharp generating the static images in the public directory. If I could cache those between builds that would be great.

[axe312ger]

@kentcdodds this should be possible with the new beta.

plugins: [
  {
    resolve: "gatsby-plugin-netlify-cache",
    options: {
      extraDirsToCache: [
        "public/static"
      ]
    }
  }
]

[kentcdodds]

Will that not suffer from the original issue?

[axe312ger]

It might till Gatsby cleans the public directory from old files on its own. I unfortunately found no option to detect obsolete files with Gatsby to avoid caching no more used files :(

[kentcdodds]

Any chance this plugin could detect files that are over a week or two old and delete those?

[axe312ger]

Hmm yeah, since the timestamps are preserved I could add a feature to delete old files. Good idea!

[axe312ger]

Any suggestion how the config could look like?

[kentcdodds]

I could envision a very low-level API like this:

plugins: [
  {
    resolve: "gatsby-plugin-netlify-cache",
    options: {
      shouldDelete: file => {/* return true if it should be deleted? */},
      extraDirsToCache: [
        "public/static"
      ]
    }
  }
]

And then a higher-level API/config like this:

plugins: [
  {
    resolve: "gatsby-plugin-netlify-cache",
    options: {
      expirationTime: '2 weeks', // you could feed this into npm.im/ms
      extraDirsToCache: [
        "public/static"
      ]
    }
  }
]

Thoughts?