gatsby-plugin-sharp / gatsby-remark-images: Set destination directory with destinationDir option

[karlhorky]

Summary

For the plugins gatsby-plugin-sharp and/or gatsby-remark-images (whichever makes sense and is necessary), an option destinationDir (string or function) that works similarly to the option in gatsby-remark-copy-linked-files...

// save `my-awesome-pdf.pdf` to `public/downloads/2a0039f3a61f4510f41678438e4c863a/my-awesome-pdf.pdf`
destinationDir: f => `downloads/${f.hash}/${f.name}`

// save `my-awesome-pdf.pdf` to `public/downloads/2a0039f3a61f4510f41678438e4c863a-my-awesome-pdf.pdf`
destinationDir: f => `downloads/${f.hash}-${f.name}`

// save `my-awesome-pdf.pdf` to `public/my-awesome-pdf/2a0039f3a61f4510f41678438e4c863a.pdf`
destinationDir: f => `${f.name}/${f.hash}`

...also with the new relativeDirectory property in the returned file object:

// save `my-awesome-pdf.pdf` to `public/original-path-in-src-pages/my-awesome-pdf.pdf`
destinationDir: f => `${f.relativeDirectory}/${f.name}`,

Basic example

See examples above.

Motivation

The structure of the files in the output public folder is more configurable and using the relativeDirectory option, can also match 1 to 1 the folder structure of the input src/pages directory. This makes the output easier to reason about and also easier to build tooling upon.

[sidharthachatterjee]

This makes a lot of sense. Happy to receive a PR adding this!

[karlhorky]

Nice! Could you weigh in on where the changes should be made? In gatsby-plugin-sharp? Or gatsby-remark-images / gatsby-images? Or all three?

[wardpeet]

@karlhorky could you tell us more about tooling that would be beneficial here? I feel that adding an option destinationDir might cause more problems than it fixes.

Users can easily set it to downloads/${f.name} and will have issues with caching as we tell people to make these files immutable.

[karlhorky]

@wardpeet in lieu of an Asset Manifest (see #20745), I would like the built assets to better match the inputs - eg. that output images be under the same paths as the input images:

input:
src/pages/original/path/my-awesome-image.jpg

output:
public/original/path/my-awesome-image-83j4g898hg98sheijf0239ijf9djsf9h39h98hre.jpg

This is in order to achieve server-side authentication and access control to the built assets - eg. #1100 (comment)

will have issues with caching as we tell people to make these files immutable

This is a good point, and maybe also something that should be applied to the destinationDir callback in gatsby-remark-copy-linked-files.

Maybe one way of doing this would be to require and validate that the content hash be used somewhere in the filename? We could do this by running the function and making sure that the argument is being used, à la:

gatsby/packages/gatsby-remark-copy-linked-files/src/index.js

Line 37 in 6270c3d

const isValidFunction = `${dir({ name: `n`, hash: `h` })}` !== `${dir({})}`

But instead of validating that EITHER the name OR the hash is used, the validation could be stricter - that the hash is used.

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[karlhorky]

not stale

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[karlhorky]

not stale

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[karlhorky]

lol such a stale bot

not stale

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[wardpeet]

But instead of validating that EITHER the name OR the hash is used, the validation could be stricter - that the hash is used.

That's interesting, I'm still hesitant to enable this behavior as I don't fully understand the complications that this will bring. Technically we could start with adding outputDir to fluid, fixed, ...

Would it help if you got a list of assets per route?

[karlhorky]

That's interesting, I'm still hesitant to enable this behavior as I don't fully understand the complications that this will bring.

Well, as long as the hash (and also versionHash, I'm just realizing) is included in the name somehow, I don't think this would cause issues. It would need to generate multiple different:

Original Path

pages/git-github/002-dvcs.png

Before

static/289ee13ff077319387caa5cb396c6cea/2a79a/002-dvcs.png
static/289ee13ff077319387caa5cb396c6cea/3b59c/002-dvcs.png
static/289ee13ff077319387caa5cb396c6cea/74866/002-dvcs.png

After (without relativeDirectory)

Here the first static part of the URL stays the same (cannot be changed through this API).

The name destinationDir doesn't actually make sense here, because it is actually ONLY changing the filename. But destinationDir also doesn't make sense in gatsby-remark-copy-linked-files...

Config: destinationDir: f => `${f.name}-${f.hash}-${f.versionHash}`

static/289ee13ff077319387caa5cb396c6cea/2a79a/002-dvcs-289ee13ff077319387caa5cb396c6cea-2a79a.png
static/289ee13ff077319387caa5cb396c6cea/3b59c/002-dvcs-289ee13ff077319387caa5cb396c6cea-3b59c.png
static/289ee13ff077319387caa5cb396c6cea/74866/002-dvcs-289ee13ff077319387caa5cb396c6cea-74866.png

After (with relativeDirectory)

Config: destinationDir: f => `${f.relativeDirectory}/${f.name}/${f.versionHash}/${f.name}-${f.hash}`

public/git-github/002-dvcs.png/2a79a/002-dvcs-289ee13ff077319387caa5cb396c6cea.png
public/git-github/002-dvcs.png/3b59c/002-dvcs-289ee13ff077319387caa5cb396c6cea.png
public/git-github/002-dvcs.png/74866/002-dvcs-289ee13ff077319387caa5cb396c6cea.png

[karlhorky]

Would it help if you got a list of assets per route?

Sure, that would solve it a different way. That would be my request over at #20745

[wardpeet]

I feel there are more use-cases for a manifest file than there might be for renaming files. This would indicate that these files are part of a route, so part of webpack or page-data

[karlhorky]

Yeah definitely, I think there are some use cases for sure for a full asset manifest.

Right now I'm using the information in webpack.stats.json, which gets me some things. But a unified file that contained everything in the webpack stats and also these new files would be great.

Back then I made a list of all the things that I found missing from webpack.stats.json.

[karlhorky]

The project that I'm using it in is this Gatsby Server-side Authentication here: https://github.com/upleveled/gatsby-serverside-auth0