-
Notifications
You must be signed in to change notification settings - Fork 533
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Profiled AOT not possible on Windows when there are spaces in project path #3407
Comments
@Mikilll94 can you post a diagnostic MSBuild log of the failure? I tried testing this scenario, but using a path with spaces was working for me: master...jonathanpeppers:profiledaot-spaces |
Here it is:
|
This is the old Escaping issue we always have when calling
However this is already using the Which should be returning a short path for those bits with spaces etc. |
Hmm, when I looked at a log of mine that is working, it did this:
So why isn't that happening here? @Mikilll94 that build log doesn't look like it has diagnostic verbosity, can you turn that on and send a new log? Or if you can get an MSBuild binlog that is even better: |
Hey, Look at my temp-path:
As you can see my project is localized in "MyApp - new". And this directory name has 2 SPACES. In temp-path which you have provided, I don't see any spaces. So maybe that's the reason that it is working for your project. |
@Mikilll94 a diagnostic log would be enormous, it looks like you may have posted a snippet of one? You should be able to zip up the full log and attach in a Github comment. Two spaces also works for me, which is why we need a full log to figure out the issue here. |
Here is a full log: |
@Mikilll94 reviewing what we do here: https://github.com/xamarin/xamarin-android/blob/ecc016a2157e2a048dfd08f57e5ce4952aae3127/src/Xamarin.Android.Build.Tasks/Tasks/Aot.cs#L176-L182 What path does your machine report if you run this sample? ShortPath.zip Mine outputs:
I tried |
For my C drive it prints: For my D drive it prints: Try also on your computer. You should have the same results. |
GetShortPathName is based on a 8dot3 parameter. It's possible that for certains users all partitions have this parameter activated so it always work. You can check it by running this command.
|
You are right! I have checked these two commands. On my computer 8.3 short file names are only enabled on C: drive. On D: drive short filenames are disabled. |
We really need to just kill these 8dot3 paths and not use them at all, the issue was that we didn't have a way to run the Mono AOT compilers with a "response file" as input. We started this effort here: #2062 If I remember correctly, we needed some further changes on the Mono side. @dellis1972 do you remember? |
@jonathanpeppers there still needs to be some changes on the mono side. If I remember rightly it was not parsing the response file correctly and did not deal will spaces correctly either. |
Hello, i confirmed i have the same trouble with my computer. The beginning of the path of the directories of all my projects is "E:\visual studio 2015\Projects..." and i have the error "[aot-compiler stderr] Cannot open assembly 'studio': No such file or directory." On E: volume, short file names isn't enabled only on C:. Thanks for your help. |
Fixes: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/609244 Fixes: https://developercommunity.visualstudio.com/content/problem/554407/vs2019-xamarinandroid-aot-command.html Fixes: #3407 Our AOT system has issues when a user uses non-ASCII characters and spaces on Windows. We used [`GetShortPathName()`][0] to get the old DOS 8.3 short names of paths to get around paths having spaces and unicode characters. However, short path name generation can be [disabled][1], in which case `GetShortPathName()` will return the long path name. Consequently, builds can fail when spaces appear in unexpected places: [aot-compiler stderr] Cannot open assembly 'Files': No such file or directory. Short path name generation can be disabled on a drive-by-drive basis, and our Azure DevOps build machines have short path name generation disabled on the `E:` drive used for builds. We really need to support paths with spaces and unicode characters. Rework the way we provide the `--aot` argument to the cross compilers so that it actually works in those scenarios. The first thing was how the arguments were parsed. `mono` uses the built-in system command line parser [`CommandLineToArgv()`][2] to parse arguments. Given the following `--aot` argument --aot=outfile="c:\Sandbox\repo\bin\BuildAotApplicationAndBundle AndÜmläüts_x86_True_True\obj\Release\libaot-Mono.Android.dll.so",asmwriter,mtriple=i686-linux-android,tool-prefix=C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\toolchains\x86-4.9\prebuilt\windows\bin\i686-linux-android-,ld-flags=C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\toolchains\x86-4.9\prebuilt\windows\lib\gcc\i686-linux-android\4.9.x\libgcc.a;C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\platforms\android-16\arch-x86\usr\lib\libc.so;C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\platforms\android-16\arch-x86\usr\lib\libm.so,temp-path="c:\Sandbox\repo\BuildAotApplicationAndBundle AndÜmläüts_x86_True_True\obj\Release\temp" This ends up as the following arguments, one per-line [0]: --aot=outfile="c:\Sandbox\repo\bin\BuildAotApplicationAndBundle [1]: AndÜmläüts_x86_True_True\obj\Release\libaot-Mono.Android.dll.so" [2]: ,asmwriter,mtriple=i686-linux-android,tool-prefix=C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\toolchains\x86-4.9\prebuilt\windows\bin\i686-linux-android-,ld-flags=C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\toolchains\x86-4.9\prebuilt\windows\lib\gcc\i686-linux-android\4.9.x\libgcc.a;C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\platforms\android-16\arch-x86\usr\lib\libc.so;C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\platforms\android-16\arch-x86\usr\lib\libm.so,temp-path="c:\Sandbox\repo\BuildAotApplicationAndBundle [3]: AndÜmläüts_x86_True_True\obj\Release\temp" As you can see the parameters have been split wherever there is a space. The solution to this is to double quote the ENTIRE argument and remove any quotes within the parameter list like: "--aot=outfile=c:\Sandbox\repo\bin\BuildAotApplicationAndBundle AndÜmläüts_x86_True_True\obj\Release\libaot-Mono.Android.dll.so,asmwriter,mtriple=i686-linux-android,tool-prefix=C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\toolchains\x86-4.9\prebuilt\windows\bin\i686-linux-android-,ld-flags=C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\toolchains\x86-4.9\prebuilt\windows\lib\gcc\i686-linux-android\4.9.x\libgcc.a;C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\platforms\android-16\arch-x86\usr\lib\libc.so;C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\platforms\android-16\arch-x86\usr\lib\libm.so,temp-path=c:\Sandbox\repo\BuildAotApplicationAndBundle AndÜmläüts_x86_True_True\obj\Release\temp" This allows Windows (and mac) to parse the parameters correctly as one value. There is another issue however. With the new argument line above, if the `temp-path=` path has a space in it then `mono` still has issues with that path. The good news is that we can use some domain knowledge to reduce not only the paths which need spaces but also the overall length of the argument. Because we know the cross compiler will be executed within `WorkingDirectory` we can shorten any path which is within that directory structure. `WorkingDirectory` is set to the directory of the projects `.csproj` file. So the following: E:\Some Project\My Project\obj\Release\aot\System.dll\ will become obj\Release\aot\System.dll\ This will fix the issue with the `temp-path=` argument. We end up with something like this "--aot=outfile=obj\Release\libaot-Mono.Android.dll.so,asmwriter,mtriple=i686-linux-android,tool-prefix=C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\toolchains\x86-4.9\prebuilt\windows\bin\i686-linux-android-,ld-flags=C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\toolchains\x86-4.9\prebuilt\windows\lib\gcc\i686-linux-android\4.9.x\libgcc.a;C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\platforms\android-16\arch-x86\usr\lib\libc.so;C:\ProgramData\Microsoft\AndroidNDK\android-ndk-r13b\platforms\android-16\arch-x86\usr\lib\libm.so,temp-path=obj\Release\temp" However we might also still start hitting the [command line length limit][3] on Windows. This is currently 8191 characters on Windows XP and later. So depending on where a user creates the project we might end up with a "too long" command line. To work around this issue we can make use of the new `mono --response=` argument. Instead of passing all the arguments to the cross compiler we can instead write them to a file and pass the path to that file as the `--response=` argument. This will reduce our command line length to be within an acceptable range unless the user creates a project in a very very deep directory structure. The final call will be like: ...\cross-arm --response="c:\Sandbox\repo\bin\BuildAotApplicationAndBundle AndÜmläüts_x86_True_True\obj\Release\Mono.Android.dll\response.txt" which works perfectly. This commit also updates the `BuildTest.BuildAotApplication()` and `BuildTest.BuildAotApplicationAndBundle()` unit tests to use paths with both spaces and non-ASCII characters to ensure we support both of those scenarios. [0]: https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getshortpathnamew [1]: https://support.microsoft.com/en-us/help/121007/how-to-disable-8-3-file-name-creation-on-ntfs-partitions [2]: https://docs.microsoft.com/en-us/windows/win32/api/shellapi/nf-shellapi-commandlinetoargvw [3]: https://support.microsoft.com/en-us/help/830473/command-prompt-cmd-exe-command-line-string-limitation
Release status update A new Preview version has now been published that includes the fix for this item. The fix is not yet included in a Release version. I will update this item again when a Release version is available that includes the fix. Fix included in Xamarin.Android 10.0.99.100. Fix included on Windows in Visual Studio 2019 version 16.4 Preview 1. To try the Preview version that includes the fix, check for the latest updates in Visual Studio Preview. |
Release status update A new Release version has now been published that includes the fix for this item. Fix included in Xamarin.Android 10.1.0.30. Fix included on Windows in Visual Studio 2019 version 16.4. To get the new version that includes the fix, check for the latest updates or install the latest version from https://visualstudio.microsoft.com/downloads/. |
Steps to Reproduce
<AndroidEnableProfiledAot>true</AndroidEnableProfiledAot>
Unhandled exception: System.ArgumentException: Illegal characters in path. at System.IO.Path.CheckInvalidPathChars(String path, Boolean checkAdditional) at System.IO.Path.GetFileName(String path) at Xamarin.Android.Tasks.Aot.<>c__DisplayClass77_0.<RunParallelAotCompiler>b__0(Config config) at Xamarin.Android.Tasks.AsyncTaskExtensions.<>c__DisplayClass0_0
1.b__0(TSource s)`
I think this is happening because I have some spaces in project path. If I remove spaces from the path it works.
Expected Behavior
Profile AOT should be possible even if there are spaces in the project path.
Actual Behavior
AOT is not possible because of an error
Version Information
Log File
Microsoft Visual Studio Professional 2019
Version 16.2.0
VisualStudio.16.Release/16.2.0+29123.88
Microsoft .NET Framework
Version 4.7.03056
Installed Version: Professional
Application Insights Tools for Visual Studio Package 9.1.00611.1
Application Insights Tools for Visual Studio
ASP.NET and Web Tools 2019 16.2.290.48383
ASP.NET and Web Tools 2019
ASP.NET Web Frameworks and Tools 2019 16.2.290.48383
For additional information, visit https://www.asp.net/
Azure App Service Tools v3.0.0 16.2.290.48383
Azure App Service Tools v3.0.0
Azure Functions and Web Jobs Tools 16.2.290.48383
Azure Functions and Web Jobs Tools
C# Tools 3.2.0-beta4-19359-03+15b43b33901c88f68ef43f8314b5a2457716780d
C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.
Common Azure Tools 1.10
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.
Extensibility Message Bus 1.2.0 (d16-2@8b56e20)
Provides common messaging-based MEF services for loosely coupled Visual Studio extension components communication and integration.
IntelliCode Extension 1.0
IntelliCode Visual Studio Extension Detailed Info
Microsoft Azure Tools 2.9
Microsoft Azure Tools for Microsoft Visual Studio 0x10 - v2.9.20626.2
Microsoft Continuous Delivery Tools for Visual Studio 0.4
Simplifying the configuration of Azure DevOps pipelines from within the Visual Studio IDE.
Microsoft JVM Debugger 1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines
Microsoft Library Manager 1.0
Install client-side libraries easily to any web project
Microsoft MI-Based Debugger 1.0
Provides support for connecting Visual Studio to MI compatible debuggers
Microsoft Visual Studio Tools for Containers 1.1
Develop, run, validate your ASP.NET Core applications in the target environment. F5 your application directly into a container with debugging, or CTRL + F5 to edit & refresh your app without having to rebuild the container.
Mono Debugging for Visual Studio 16.2.6 (4cfc7c3)
Support for debugging Mono processes with Visual Studio.
NuGet Package Manager 5.2.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/
ProjectServicesPackage Extension 1.0
ProjectServicesPackage Visual Studio Extension Detailed Info
ResourcePackage Extension 1.0
ResourcePackage Visual Studio Extension Detailed Info
ResourcePackage Extension 1.0
ResourcePackage Visual Studio Extension Detailed Info
SQL Server Data Tools 16.0.61906.28070
Microsoft SQL Server Data Tools
Syntax Visualizer 1.0
An extension for visualizing Roslyn SyntaxTrees.
TypeScript Tools 16.0.10627.2001
TypeScript Tools for Microsoft Visual Studio
Visual Basic Tools 3.2.0-beta4-19359-03+15b43b33901c88f68ef43f8314b5a2457716780d
Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.
Visual F# Tools 10.4 for F# 4.6 16.2.0-beta.19321.1+a24d94ecf97d0d69d4fbe6b8b10cd1f97737fff4
Microsoft Visual F# Tools 10.4 for F# 4.6
Visual Studio Code Debug Adapter Host Package 1.0
Interop layer for hosting Visual Studio Code debug adapters in Visual Studio
Visual Studio Tools for Containers 1.0
Visual Studio Tools for Containers
VisualStudio.Mac 1.0
Mac Extension for Visual Studio
Xamarin 16.2.0.90 (d16-2@ba267630e)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.
Xamarin Designer 16.2.0.325 (remotes/origin/d16-2@f10cfbf83)
Visual Studio extension to enable Xamarin Designer tools in Visual Studio.
Xamarin Templates 16.3.117 (59a59e8)
Templates for building iOS, Android, and Windows apps with Xamarin and Xamarin.Forms.
Xamarin.Android SDK 9.4.0.51 (d16-2/9fa7775)
Xamarin.Android Reference Assemblies and MSBuild support.
Mono: mono/mono@e6f5369c2d2
Java.Interop: xamarin/java.interop/d16-2@d64ada5
LibZipSharp: grendello/LibZipSharp/d16-2@caa0c74
LibZip: nih-at/libzip@b95cf3f
ProGuard: xamarin/proguard@905836d
SQLite: xamarin/sqlite@8212a2d
Xamarin.Android Tools: xamarin/xamarin-android-tools/d16-2@6f6c969
Xamarin.iOS and Xamarin.Mac SDK 12.14.0.110 (a8bcecc)
Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.
The text was updated successfully, but these errors were encountered: