Enable filenames with colon (:) on drvfs #1514

doehrm · 2016-12-16T22:32:37Z

Insider build 14986

Trying to create a filename with a colon on a drive mounted with drvfs is not working whereas I can create the file on lxfs:

$ mount | grep home
home on /home type lxfs (rw,noatime)

$ touch foo+bar:0185.somext

$ ls -l
-rw-rw-rw- 1 doehrm doehrm 0 Dec 16 23:26 foo+bar:0185.somext

$ mount | grep \/e
E: on /mnt/e type drvfs (rw,noatime)

$ touch foo+bar:0185.somext
touch: setting times of 'foo+bar:0185.somext': No such file or directory

$ echo "Foo Bar" > foo+bar:0185.somext
bash: foo+bar:0185.somext: Invalid argument

$ echo "Foo Bar" > foo+bar\:0185.somext
bash: foo+bar:0185.somext: Invalid argument

The drive connected is an external USB drive formatted with NTFS

Am I missing something?

--
Markus

The text was updated successfully, but these errors were encountered:

therealkenc · 2016-12-17T00:02:34Z

Unfortunately NTFS defines a set of reserved characters, while Unix-ish filesystems (ext, ufs2, zfs, hfs), on which '/' is the only special character, does not. Part of that is because of Microsoft still being tied to the boat anchor that is DOS.

As an aside, Unix is tied somewhat to the boat anchor that is bash, and ":" is de-facto problematic also; because, there is no way to escape ':' in a PATH environment variable. Curious what your use case is, because you pretty much have to avoid ':' on Linux too (but by convention only).

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx

doehrm · 2016-12-17T00:35:14Z

The use case is "chromium-os" ;-)

I build a special version for an older laptop of mine. The official build host is an Ubuntu 16.04.1. The whole build is done in a chroot'ed environment so basically only a few tools need to be installed on the OS (python, git, curl and a few others), the rest is downloaded and compiled in the chroot.

While downloading the repository descriptions I come across a few files that cannot be sync'ed because of the colon:

error: unable to create file frontend/client/src/autotest/public/Open+Sans:300.woff (Invalid argument)
error: unable to create file frontend/client/src/autotest/public/Roboto+Bold:700.woff (Invalid argument)
error: unable to create file frontend/client/src/autotest/public/Roboto+Light:300.woff (Invalid argument)
error: unable to create file frontend/client/src/autotest/public/Roboto+Medium:500.woff (Invalid argument)
error: unable to create file frontend/client/src/autotest/public/Roboto+Regular:400.woff (Invalid argument)
error: unable to create file server/site_tests/display_EdidStress/test_data/edids/weekly/SCT_272_STEELCASE_m:s_HDMI.txt

The original sources really contain that colon:

https://chromium.googlesource.com/chromiumos/third_party/autotest/+/master/frontend/client/src/autotest/public

My aim was to have a "somewhat native" possibility to compile a new version on my laptop rather than to have Hyper-V and virtual network cards and switches messing with my non-trivial network/VPN setup (if you ever set up half a dozen VPNs (SSTP, OpenVPN and PPTP mixed with ipv4/ipv6) and you install Hyper-V and you wonder, why nothing is working any more, you'll understand ;-) )

But I understand that it'd be problematic allowing it, despite I'm not looking to have it as a directory name but "just a file name".

aseering · 2016-12-17T00:35:21Z

Tangent -- : was the Mac path separator prior to OS X. Graphical Mac applications still allow filenames to contain / and not :. Basically, : is weird everywhere.

Anyway, it's an interesting question whether DrvFs should (for compatibility purposes) be allowed to construct filenames that wouldn't be allowed for regular Windows applications, and if so, what the semantics of such files should be.

aseering · 2016-12-17T00:38:01Z

@doehrm -- can you not build in WSL's $HOME? As you mention, it supports :. And it should have better performance as well.

doehrm · 2016-12-17T00:58:29Z

I still work on an OS that has filenames such as NODE::DKA0:[FOO.BAR.BAR2]foobar.dat;5 where a colon is a separator, no possibility to create a filename with a colon there and I guess, that's also where deep in the Windows (NT 4.0) kernel it [was|is] forbidden too.

And yes, I'll try that next - as soon as I can get rid of some older insider builds still stored on the disk and make some space.

doehrm · 2016-12-17T01:06:19Z

Interestingly enough, I could clone the repository using GIT on Windows:

C:\Users\curry\test>..\Git\bin\git.exe clone https://chromium.googlesource.com/chromiumos/third_party/autotest
Cloning into 'autotest'...
remote: Sending approximately 434.56 MiB ...
remote: Counting objects: 9793, done
remote: Finding sources: 100% (16/16)
remote: Total 177259 (delta 128226), reused 177251 (delta 128226)
Receiving objects: 100% (177259/177259), 434.38 MiB | 698.00 KiB/s, done.
Resolving deltas: 100% (128226/128226), done.
Checking out files: 100% (9006/9006), done.

and it seems there is some sort of semantics already existing:

c:\Users\curry\test\autotest\frontend\client\src\autotest\public>dir /r
 Volume in drive C has no label.
 Volume Serial Number is 8203-5049

 Directory of C:\Users\curry\test\autotest\frontend\client\src\autotest\public

17.12.16  02:01    <DIR>          .
17.12.16  02:01    <DIR>          ..
[...]
17.12.16  02:01                 0 Open+Sans
                           21 744 Open+Sans:300.woff:$DATA
17.12.16  02:01                 0 Roboto+Bold
                           19 812 Roboto+Bold:700.woff:$DATA
17.12.16  02:01                 0 Roboto+Light
                           21 080 Roboto+Light:300.woff:$DATA
17.12.16  02:01                 0 Roboto+Medium
                           20 636 Roboto+Medium:500.woff:$DATA
17.12.16  02:01                 0 Roboto+Regular
                           21 132 Roboto+Regular:400.woff:$DATA
[...]
              21 File(s)         66 494 bytes
               2 Dir(s)  46 544 334 848 bytes free

A plain dir shows the first line (size 0, short filename up to the colon), dir /r reveals the complete name with a "$DATA" at the end.

fpqc · 2016-12-17T04:07:49Z

@therealkenc The real reason doesn't have anything to do with DOS iirc. The ":" character is reserved for alternate data streams in NTFS and actually appears to be related maybe to the ancient OS/2 subsystem.

doehrm · 2016-12-17T10:00:54Z

I'd not say OS/2 but rather VMS (OpenVMS). Windows and VMS share quite a few things (http://www3.sympatico.ca/n.rieck/docs/Windows-NT_is_VMS_re-implemented.html).

doehrm · 2016-12-17T10:25:29Z

I was now able to download all the code to $HOME. When I try to enter the chroot the system either freezes (and I have to hard reboot) or BSODs :-( with APC_INDEX_MISMATCH.

*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 1, {7fce84d0c357, 0, 1, ffffc701f5f0dc80}

Page 10ad38 not present in the dump file. Type ".hh dbgerr004" for details
Probably caused by : ntkrnlmp.exe ( nt!KiSystemServiceExitPico+194 )

*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

APC_INDEX_MISMATCH (1)
This is a kernel internal error. The most common reason to see this
bugcheck is when a filesystem or a driver has a mismatched number of
calls to disable and re-enable APCs. The key data item is the
Thread->CombinedApcDisable field. This consists of two separate 16-bit
fields, the SpecialApcDisable and the KernelApcDisable. A negative value
of either indicates that a driver has disabled special or normal APCs
(respectively) without re-enabling them; a positive value indicates that
a driver has enabled special or normal APCs (respectively) too many times.
Arguments:
Arg1: 00007fce84d0c357, Address of system call function or worker routine
Arg2: 0000000000000000, Thread->ApcStateIndex
Arg3: 0000000000000001, (Thread->SpecialApcDisable << 16) | Thread->KernelApcDisable
Arg4: ffffc701f5f0dc80, Call type (0 - system call, 1 - worker routine)

Debugging Details:
------------------

Page 10ad38 not present in the dump file. Type ".hh dbgerr004" for details

FAULTING_IP: 
+8c0
00007fce`84d0c357 ??              ???

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT
BUGCHECK_STR:  0x1
CURRENT_IRQL:  0
ANALYSIS_VERSION: 6.3.9600.17336 (debuggers(dbg).150226-1500) amd64fre
LAST_CONTROL_TRANSFER:  from fffff80084d60929 to fffff80084d55240

STACK_TEXT:  
ffffc701`f5f0dab8 fffff800`84d60929 : 00000000`00000001 00007fce`84d0c357 00000000`00000000 00000000`00000001 : nt!KeBugCheckEx
ffffc701`f5f0dac0 fffff800`84d6082b : ffffc701`f5f0dc00 ffffb703`396c6200 00000000`00000000 00000000`00000000 : nt!KiBugCheckDispatch+0x69
ffffc701`f5f0dc00 00007fce`84d0c357 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceExitPico+0x194
00007fff`ce8dcf98 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007fce`84d0c357

STACK_COMMAND:  kb

FOLLOWUP_IP: 
nt!KiSystemServiceExitPico+194
fffff800`84d6082b 4883ec50        sub     rsp,50h

SYMBOL_STACK_INDEX:  2
SYMBOL_NAME:  nt!KiSystemServiceExitPico+194
FOLLOWUP_NAME:  MachineOwner
MODULE_NAME: nt
IMAGE_NAME:  ntkrnlmp.exe
DEBUG_FLR_IMAGE_TIMESTAMP:  58427289
BUCKET_ID_FUNC_OFFSET:  194
FAILURE_BUCKET_ID:  0x1_SysCallNum_0_nt!KiSystemServiceExitPico
BUCKET_ID:  0x1_SysCallNum_0_nt!KiSystemServiceExitPico
ANALYSIS_SOURCE:  KM
FAILURE_ID_HASH_STRING:  km:0x1_syscallnum_0_nt!kisystemserviceexitpico
FAILURE_ID_HASH:  {48293086-2975-82e2-87d1-2bab5cd63ecf}

I switched on a full dump and will reproduce the problem again (if that helps).

fpqc · 2016-12-17T11:18:12Z

@doehrm Please email [email protected] ATTN: Ben Hillis ( @benhillis ) or Russ Alexander ( @russalex )with a copy of a minidump.

Also, yes, that's right, alternate data streams is a VMS-inspired thing. Extended attributes are the OS/2 legacy.

doehrm · 2016-12-17T12:22:40Z

Done so, thanx @fpqc for pointing me in the direction and sorry for messing this issue up with nonrelated BSOD's.

Since NTFS seems to have a way to handle those files, doesn't/couldn't/shouldn't that mean DrvFS should too?

fpqc · 2016-12-17T12:27:17Z

@doehrm The path "a:b" refers to an alternate data stream of the file "a" on NTFS. It is not a valid path in Win32. Powershell has a cmdlet (streams) that lets you see what ADS are attached to a file. The default streamtype is simply called "$DATA" Look familiar?

https://msdn.microsoft.com/en-us/library/windows/desktop/aa364404(v=vs.85).aspx

iirc, GeoHot demonstrated an attack on Windows using ADS a few years ago.

doehrm · 2016-12-17T12:50:06Z

or it's internally "mapped" to a Unicode character (U+F000 - U+F0FF)...

fpqc · 2016-12-17T13:03:57Z

The result of your dir /r indicates that alternate data streams were stored.

It says "I created an empty file Open+Sans, then created an alternate $DATA stream for that file called 300.woff that contains the actual desired content of the file". Git for Windows shouldn't be doing that. It is completely broken behavior.

I just tested it myself. That's what it's doing. That's also probably why it's going crazy with BSODs. Nobody (external to Microsoft, at least) has, to my knowledge, tested DrvFS functionality with Alternate named $DATA streams thrown into the mix.

It works in LXFS because the WSL drivers are intercepting all File I/O within the environment and escaping the character as necessary when it ends up on the disk.

benhillis · 2016-12-20T16:51:10Z

@doehrm - I received your dump and took a look. The cause of the bluescreen was a locking issue in a terminal ioctl. The fix just reached the rs_prerelease branch so it should be in the next Windows Insider build.

Thank you for reporting the issue and sending us the memory dump!

bitcrazed · 2017-04-07T17:01:08Z

Thanks for reporting. Closing since this issue should have been fixed in an insider build in early 2017, and there's been no recent activity on this issue.

SRGOM · 2017-05-05T05:06:07Z

@bitcrazed is this "fix" in the creators update too? If so, is it allowed to have a filename with colon on an NTFS partition and be accessible from WSL?

therealkenc · 2017-05-05T05:20:17Z

No, it was just closed incorrectly.

SRGOM · 2017-05-06T10:34:54Z

Thanks @therealkenc Maybe dev wouldn't mind if I open a new one to get some clarity here..

therealkenc · 2017-05-06T19:37:09Z

@SRGOM Repurpose your #2061 by changing the title - this issue was closed 9 days after #1828 was marked as a dupe. I don't know the chances/timeframe for this being supported because it will probably take some cooperation from the NTFS win32 side people which might or might not be forthcoming. Notwithstanding Cygwin style hacks anyway, which I suppose could be a reasonable enough approach in the absence of alternatives.

SRGOM · 2017-05-07T09:34:48Z

I created #2070 (and closed #2061). I'd love to have this feature but what I'm actually looking for is clarity- Since Rich says that this has already been implemented, maybe a reg switch is all that I need?

therealkenc · 2017-05-07T10:25:10Z

Rich probably just conflated the terminal ioctl related BSOD being squashed above with the issue title. A registry switch to turn on and off filename colon support in DrvFS would make no sense, and even if it did, it wouldn't be a secret switch. And... if it were a secret switch... well... those in the know would be sworn to secrecy... wouldn't they.

therealkenc · 2018-02-14T17:08:10Z

Reopening until the win32 reserved characters (: ? etc) fix lands so this can get a fixedinsiders tag. Which I gather might be forthcoming.

benhillis · 2018-02-14T17:10:42Z

@therealkenc - Thanks, didn't realize this one was closed.

Brian-Perkins · 2018-02-15T01:01:24Z

The reserved character issue should be resolved as of Insiders Build 17101.

feaber · 2018-02-25T18:14:21Z

I have Windows 10 Pro N (1803, build: 17101.1000)
Looks like now it is possible to create file with colon in name on / (rootfs) and on /mnt/<letter>

Still I have some problems with colons on /mnt/<letter>
I run simple script that connects to my remote host with Debian and performs backup. The script is based on rsync. On the server I have some services, one of them are Dovecot/Postfix servers.

The files that represent email have complex names like:
.1519283136.M726136P495.articode.pl,S=23068,W=23405:2,Sc.KXvFzt

When I run my script from rootfs all works fine. But when destination location to save is some folder inside /mnt/<letter> then rsync try to make some strange rename actions on not existing files.

Example:
rsync: rename "/mnt/e/articode.pl/current/mail/articode.pl/biuro/.Trash/cur/.1519283136.M726136P495.articode.pl,S=23068,W=23405:2,Sc.KXvFzt" -> "articode.pl/biuro/.Trash/cur/1519283136.M726136P495.articode.pl,S=23068,W=23405:2,Sc": No such file or directory (2)

Oryginal rsync parameters:
rsync -az -e ssh --delete --rsync-path="sudo rsync" "${REMOTE_LOGIN}@${REMOTE_SERVER}:/var/mail/" "${BACKUP_DIR}/current/mail/"

therealkenc · 2018-08-21T23:01:01Z

I guess this made 1803 after all (17101 < 17134)

stehufntdev mentioned this issue Feb 15, 2017

Java file access with separator #1703

Closed

therealkenc mentioned this issue Feb 20, 2017

Compiling latest GIT from source on WSL bash #1711

Closed

aseering mentioned this issue Mar 5, 2017

Cannot use colon ":" in filenames #1752

Closed

billziss-gh mentioned this issue Mar 23, 2017

POSIX filenames and invalid characters on DrvFs #1804

Closed

aseering mentioned this issue Mar 29, 2017

failed to repo sync aosp source code #1828

Closed

bitcrazed closed this as completed Apr 7, 2017

This was referenced Apr 14, 2017

conda-forge::perl installation error conda/conda#4819

Closed

Windows 10 Bash Ubuntu conda install perl failure conda/conda#4827

Closed

SRGOM mentioned this issue May 5, 2017

[Fun] Successfully managed to hose my Windows file system #2061

Closed

SRGOM mentioned this issue May 11, 2017

Enable filenames with colon (:) on drvfs (Copy of #1514) #2070

Closed

therealkenc mentioned this issue Dec 9, 2017

File naming incompatibilities in mounted folders #2741

Closed

therealkenc mentioned this issue Feb 12, 2018

checkout of a git repo with filenames containing ":" failes on /mnt/c/..... #2936

Closed

benhillis mentioned this issue Feb 14, 2018

[Bug] Not being able to use ':' in filenames causes issues #2952

Closed

therealkenc reopened this Feb 14, 2018

Brian-Perkins added bug fixedininsiderbuilds labels Feb 15, 2018

therealkenc mentioned this issue Mar 1, 2018

gcc make check in fixincludes fails because it can't create directory "*" #2991

Closed

therealkenc mentioned this issue Apr 15, 2018

mu4e - can't open "new" message due to rename file with : in name #3096

Closed

therealkenc added fixedin1803 and removed bug fixedininsiderbuilds labels May 24, 2018

therealkenc closed this as completed May 24, 2018

therealkenc added fixedininsiderbuilds and removed fixedin1803 labels Jun 20, 2018

therealkenc mentioned this issue Jun 20, 2018

filenames containing colon ':' fail to rsync and svn #3316

Closed

therealkenc added fixedin1803 and removed fixedininsiderbuilds labels Aug 21, 2018

therealkenc mentioned this issue Jan 19, 2020

wslpath does not translate some characters correctly #4682

Closed

PythonNut mentioned this issue Jun 8, 2020

Accessing files with special reserved symbols names maharmstone/btrfs#151

Closed

therealkenc mentioned this issue Jul 1, 2020

When browsing a WSL directory, some characters are not properly displayed #5516

Closed

ExE-Boss mentioned this issue Nov 21, 2021

sparse-checkout does not appear to work on windows. "error: invalid path " git-for-windows/git#2803

Open

klardotsh mentioned this issue Nov 23, 2022

Use ISO 8601 date format for backup filenames zulip/docker-zulip#364

Merged

narfa0215 mentioned this issue Dec 3, 2024

Revisiting Solution for Accessing Files with Special Reserved Symbols in WinBtrfs maharmstone/btrfs#709

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable filenames with colon (:) on drvfs #1514

Enable filenames with colon (:) on drvfs #1514

doehrm commented Dec 16, 2016

therealkenc commented Dec 17, 2016 •

edited

Loading

doehrm commented Dec 17, 2016

aseering commented Dec 17, 2016

aseering commented Dec 17, 2016 •

edited

Loading

doehrm commented Dec 17, 2016

doehrm commented Dec 17, 2016

fpqc commented Dec 17, 2016 •

edited

Loading

doehrm commented Dec 17, 2016

doehrm commented Dec 17, 2016

fpqc commented Dec 17, 2016 •

edited

Loading

doehrm commented Dec 17, 2016

fpqc commented Dec 17, 2016 •

edited

Loading

doehrm commented Dec 17, 2016

fpqc commented Dec 17, 2016 •

edited

Loading

benhillis commented Dec 20, 2016 •

edited

Loading

bitcrazed commented Apr 7, 2017

SRGOM commented May 5, 2017

therealkenc commented May 5, 2017

SRGOM commented May 6, 2017

therealkenc commented May 6, 2017 •

edited

Loading

SRGOM commented May 7, 2017 •

edited

Loading

therealkenc commented May 7, 2017 •

edited

Loading

therealkenc commented Feb 14, 2018

benhillis commented Feb 14, 2018

Brian-Perkins commented Feb 15, 2018

feaber commented Feb 25, 2018

therealkenc commented Aug 21, 2018

Enable filenames with colon (:) on drvfs #1514

Enable filenames with colon (:) on drvfs #1514

Comments

doehrm commented Dec 16, 2016

therealkenc commented Dec 17, 2016 • edited Loading

doehrm commented Dec 17, 2016

aseering commented Dec 17, 2016

aseering commented Dec 17, 2016 • edited Loading

doehrm commented Dec 17, 2016

doehrm commented Dec 17, 2016

fpqc commented Dec 17, 2016 • edited Loading

doehrm commented Dec 17, 2016

doehrm commented Dec 17, 2016

fpqc commented Dec 17, 2016 • edited Loading

doehrm commented Dec 17, 2016

fpqc commented Dec 17, 2016 • edited Loading

doehrm commented Dec 17, 2016

fpqc commented Dec 17, 2016 • edited Loading

benhillis commented Dec 20, 2016 • edited Loading

bitcrazed commented Apr 7, 2017

SRGOM commented May 5, 2017

therealkenc commented May 5, 2017

SRGOM commented May 6, 2017

therealkenc commented May 6, 2017 • edited Loading

SRGOM commented May 7, 2017 • edited Loading

therealkenc commented May 7, 2017 • edited Loading

therealkenc commented Feb 14, 2018

benhillis commented Feb 14, 2018

Brian-Perkins commented Feb 15, 2018

feaber commented Feb 25, 2018

therealkenc commented Aug 21, 2018

therealkenc commented Dec 17, 2016 •

edited

Loading

aseering commented Dec 17, 2016 •

edited

Loading

fpqc commented Dec 17, 2016 •

edited

Loading

fpqc commented Dec 17, 2016 •

edited

Loading

fpqc commented Dec 17, 2016 •

edited

Loading

fpqc commented Dec 17, 2016 •

edited

Loading

benhillis commented Dec 20, 2016 •

edited

Loading

therealkenc commented May 6, 2017 •

edited

Loading

SRGOM commented May 7, 2017 •

edited

Loading

therealkenc commented May 7, 2017 •

edited

Loading