Fix windows long paths #2566
Merged
Fix windows long paths #2566
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
beckjake
force-pushed
the
fix/windows-long-paths
branch
3 times, most recently
from
2984db2 to
8337c6b
Compare
beckjake
force-pushed
the
fix/windows-long-paths
branch
from
9d3b858 to
eee4180
Compare
beckjake
force-pushed
the
fix/windows-long-paths
branch
from
6830c7d to
f4b93c8
Compare
… hit path length limits
drewbanin
reviewed
Jun 19, 2020
There was a problem hiding this comment.
There is much about the need for this change that is unfortunate, but from what I can see, the changes here look excellent. I suspect the integration tests will be the best indicator of success - this LGTM when the tests pass (though I did have one question about them!).
| ): | ||
| # all our hard work and the path was still too long. Log and | ||
| # continue. | ||
| logger.debug( |
| temp_dir = magic_prefix + temp_dir | ||
| outer = tempfile.mkdtemp(prefix='dbt-int-test-', dir=temp_dir) | ||
| # then inside _that_ directory make a new one that gets us to just | ||
| # barely 260 total. I picked 250 to account for the '\' and anything |
There was a problem hiding this comment.
is there a reason we want to just barely eclipse 260 chars instead of going for, say, 300 chars? I just want to make sure I understand the logic behind this test.
There was a problem hiding this comment.
Yeah! We have to os.chdir into the directory we build here, and the \\?\ doesn't work for os.chdir because the underlying windows APIs for that function don't handle it.
kwigley
approved these changes
Jun 19, 2020
drewbanin
approved these changes
Jun 19, 2020
This was referenced Jun 22, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #2558
Description
This PR makes long paths on windows work. The issue described in #2558 no longer occurs, though there are a couple caveats:
dbt depsand the total path length includingdbt_modulesis >= 260, I think dbt will fail, because we'll try tocdinto a directory duringdepsthat will have a total length >= 260 and that isn't allowed ever, apparently. It will tell you the file path is too long, which is nice.dbt cleansuccessfully removes things fromtarget, which satisfies me.How this works:
First, this code checks that it needs to run at all. If the system isn't running windows, the path is under 250 chars (python likes to append things like
'*.*'in various places), or the windows function that tells us long paths are enabled is set, we don't do anything to the path.If the path needs to be massaged, dbt will prepend
\\?\(escaped to'\\\\?\\') to paths that we'll read/write from, which means we access it as a win32 File Namespace. This tells windows to skip path parsing, which bypasses MAX_PATH. There are some restrictions: Paths must be absolute, paths must not have any/in them, and something about UNC paths that I hope I got right 😬 Those caveats actually aren't a big deal for dbt.I am happy to report that very new versions of windows 10/windows server 2019 with a certain registry key/group policy setting actually don't have this problem at all with long paths, because starting with 3.6, python opts out of
MAX_PATHwith a compile-time options, and those versions of Windows know how to handle that. So in 10 years or whatever it takes for us to drop support, we can just not care about this at all.I found this blog post from project zero helpful. They're approaching it from a security standpoint but there are some very helpful charts. Note that we actually don't want the "NT Path", (it's distinct from the "win32 file namespace", so ignore that part). And we don't care about "drive-relative paths" because that's a cmd.exe thing.
Checklist
CHANGELOG.mdand added information about my change to the "dbt next" section.