Deploy does not follow symlinks

[george-steel]

Describe the bug

netlify deploy does not follow symlinks out of the publish directory and misses files.

To Reproduce

cd out/
ln -s ../index.html out/
cd ..
netlify deploy

Configuration

[build]
base = "."
publish = "out/"
command = "wasm-pack build --target no-modules"

Expected behavior

index.html should be included in the deploy. It isn't.

[erezrokah]

Thanks for opening the issue @george-steel, we might need to fix this in https://github.com/netlify/js-client/blob/d835711b9cb5a556870577d0b6c2ee27b565db12/src/deploy/hash_files.js#L8

Also related okdistribute/folder-walker#1

[janosh]

Any update on this? I believe my builds are crashing due to a similar problem also involving symlinks:

10:01:19 AM: > ENOENT: no such file or directory, stat '/opt/build/repo/site/static/assets'

which is a symlink from one part of the repo to another. It works locally with netlify build which I thought was meant to mirror the behavior on Netlify's build servers exactly?

[erezrokah]

Hi @janosh, this issue is about failure to deploy locally using to CLI - per your comment that works for you?
If the failure to build/deploy is happening on the production environment can you please open a new issue here?

If I got something wrong, can you please provide a reproduction for your issue?

[janosh]

@erezrokah Thanks, will do.

[sumodirjo]

I face this issue today. I did local build and deploy for my gohugo-based site. My public directory was symlink to directory in tmpfs.

When running deploy, deploy only detect one file, public as symbolic link and not following the symbolic link.

After moving the file to the disk in public directory. The deployment start successfully.

This actually causing problem since the deployment process successful but only deploy 0 file so the site become unavailable. But of course I can rollback to the previous deploy easily from web console.

Maybe this is separate issue but if the build directory like public is symlink netlify-cli might need to warn the user and fail the deployment.

[tinfoil-knight]

@erezrokah Inspected this today. Symlinks are getting filtered out since they're not assigned the type file by the folder-walker dependency.

See:

cli/src/utils/deploy/hash-files.js

Line 14 in dda5a49

const fileStream = walker([configPath, dir], { filter })

cli/src/utils/deploy/hasher-segments.js

Line 55 in dda5a49

const fileFilterCtor = objFilterCtor((fileObj) => fileObj.type === 'file')

The fs.Stats object returned by folder-walker has a isSymbolicLink method but there wouldn't be any way to check if the sym-link points to a file or folder (or if it even exists) until we use something like fs.readlink and then fs.lstat to get the fs.Stats object for the actual file or directory.

[erezrokah]

Thanks @tinfoil-knight for digging into it. Very useful.
Looking at the dependencies we use in that code (and the code itself 🙃 ), we can either use a sync call to get the file stats (and also replace its content with the linked file), or refactor that code to use newer dependencies like https://www.npmjs.com/package/folder-walker.

WDYT?

[tinfoil-knight]

@erezrokah

We can either use a sync call to get the file stats

Bit confused here. The stats for the linked file would already be present since those are returned from folder-walker. Do you want to remove the dependency and just do it in the cli codebase itself?

(and also replace its content with the linked file)

Not sure why replacing the symlink with the linked file would be required since while reading/uploading the file it would behave the same as a normal file AFAIK. Haven't tried this yet but I think that we would just need to check if the linked path exists and is a file so as to not filter it out.

or refactor that code to use newer dependencies like npmjs.com/package/folder-walker.

I didn't really find any alternative dependency which would resolve to the original file / directory from a symlink. They just report that the path is a symlink just like folder-walker (which is being used currently) does.

---

I think I might've misunderstood a few points in your comments. Do correct me if that's the case.

[erezrokah]

Ah, I think I overcomplicated this. How are you suggesting to fix the filtering issue?

[tinfoil-knight]

Ah, I think I overcomplicated this. How are you suggesting to fix the filtering issue?

@erezrokah I was just saying that here:

cli/src/utils/deploy/hasher-segments.js

Line 55 in dda5a49

const fileFilterCtor = objFilterCtor((fileObj) => fileObj.type === 'file')

We can check if the file object is a symbolic link. Get the path to the linked file/directory (through fs.readlink). Then check if that path is actually a file or not. (fs.stat)

If the linked path is a file, we just don't filter it out.

---

But now that I think about it more, there's also the case for symlinks to directories which have files under them.

The files under these directories will need to be "walked" too which further complicates this issue. (We'd actually need to resolve the symlinks first in this case before any filtering happens)

[tinfoil-knight]

Tried out fast-glob locally. It has a followsymboliclinks option which actually does what I described above. It follows symlinks to individual files and files under linked directories.

[ajschmidt8]

any updates on this? seems like it's still an issue with the latest CLI version

[ajschmidt8]

This is an issue because Netlify's auto-deploys do follow symlinks.

So this CLI bug leads to a discrepancy between auto-deploys and manual deploys via the CLI.

@tinfoil-knight, do you know if there are any plans to address this in the near future?

[ajschmidt8]

I've got this working locally with fast-glob like @tinfoil-knight suggested.

I will be opening a PR soon.

[ajschmidt8]

PR here: #5650