dotnet publish fails on macOS on ARM

[aeons]

Hi,

I tried to publish an image from a fresh web app, but hit an ObjectDisposedException:

❯ dotnet new web -n test-container    
The template "ASP.NET Core Empty" was created successfully.

Processing post-creation actions...
Restoring /Users/dkBjMaMa/test-container/test-container.csproj:
  Determining projects to restore...
  Restored /Users/dkBjMaMa/test-container/test-container.csproj (in 30 ms).
Restore succeeded.

❯ cd test-container

❯ dotnet add package Microsoft.NET.Build.Containers             
...

❯ dotnet publish --os linux --arch x64 -c Release -p:PublishProfile=DefaultContainer
MSBuild version 17.4.0+18d5aef85 for .NET
  Determining projects to restore...
  Restored /Users/User/test-container/test-container.csproj (in 83 ms).
  test-container -> /Users/User/test-container/bin/Release/net7.0/linux-x64/test-container.dll
  test-container -> /Users/User/test-container/bin/Release/net7.0/linux-x64/publish/
  Building image 'test-container' with tags 1.0.0 on top of base image mcr.microsoft.com/dotnet/aspnet:7.0
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: The "CreateNewImage" task failed unexpectedly. [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: System.ObjectDisposedException: Cannot access a disposed object. [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: Object name: 'System.Security.Cryptography.SHA256+Implementation'. [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at System.Security.Cryptography.HashAlgorithm.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at System.Security.Cryptography.CryptoStream.FlushFinalBlockAsync(Boolean useAsync, CancellationToken cancellationToken) [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at System.Security.Cryptography.CryptoStream.FlushFinalBlock() [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing) [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at System.IO.Stream.Close() [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at Microsoft.NET.Build.Containers.Layer.HashDigestGZipStream.Dispose(Boolean disposing) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Layer.cs:line 143 [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at Microsoft.NET.Build.Containers.Layer.FromFiles(IEnumerable`1 fileList) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Layer.cs:line 59 [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at Microsoft.NET.Build.Containers.Layer.FromDirectory(String directory, String containerPath) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Layer.cs:line 24 [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at Microsoft.NET.Build.Containers.Tasks.CreateNewImage.Execute() in D:\a\_work\1\s\Microsoft.NET.Build.Containers\CreateNewImage.cs:line 212 [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/Users/User/test-container/test-container.csproj]

My system:

System Version: macOS 13.0 (22A380)
Model Identifier: MacBookPro18,1
Chip: Apple M1 Pro

❯ dotnet --info                                                                     
.NET SDK:
 Version:   7.0.100
 Commit:    e12b7af219

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  13.0
 OS Platform: Darwin
 RID:         osx.13-arm64
 Base Path:   /usr/local/share/dotnet/sdk/7.0.100/

Host:
  Version:      7.0.0
  Architecture: arm64
  Commit:       d099f075e4

.NET SDKs installed:
  6.0.402 [/usr/local/share/dotnet/sdk]
  7.0.100 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.10 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.10 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

[baronfel]

This is really interesting - The code looks like what I'd expect to see here (no direct invocation of a disposed object):

    {
        long fileSize;
        Span<byte> hash = stackalloc byte[SHA256.HashSizeInBytes];
        byte[] uncompressedHash;

        string tempTarballPath = ContentStore.GetTempFile();
        using (FileStream fs = File.Create(tempTarballPath))
        {
            using (HashDigestGZipStream gz = new(fs, leaveOpen: true))
            {
                using (TarWriter writer = new(gz, TarEntryFormat.Gnu, leaveOpen: true))
                {
                    foreach (var item in fileList)
                    {
                        // Docker treats a COPY instruction that copies to a path like `/app` by
                        // including `app/` as a directory, with no leading slash. Emulate that here.
                        string containerPath = item.containerPath.TrimStart(PathSeparators);

                        writer.WriteEntry(item.path, containerPath);
                    }
                } // Dispose of the TarWriter before getting the hash so the final data get written to the tar stream

                uncompressedHash = gz.GetHash();
            }

            fileSize = fs.Length;

            fs.Position = 0;

            SHA256.HashData(fs, hash);
        }

and we're explicitly not closing any of the streams until the outer filestream exits the scope. @rainersigwald any thoughts on this one? Wondering if tere might be some kind of ARM64 interaction on hashing on macOS?

[rainersigwald]

Yeah, that tracks quite well with what I've figured out too. @eerhardt any hope this immediately rings a bell for you?

[eerhardt]

Hmm, I tried to repro this on my macOS Ventura M1, but it didn't repro.

@aeons - does it repro for you every time? Can you pass /bl to the command and attach a binlog?

[eerhardt]

I wonder if an exception is happening inside

sdk-container-builds/Microsoft.NET.Build.Containers/Layer.cs

Lines 44 to 58 in 4ed9915

	using (HashDigestGZipStream gz = new(fs, leaveOpen: true))
	{
	using (TarWriter writer = new(gz, TarEntryFormat.Gnu, leaveOpen: true))
	{
	foreach (var item in fileList)
	{
	// Docker treats a COPY instruction that copies to a path like `/app` by
	// including `app/` as a directory, with no leading slash. Emulate that here.
	string containerPath = item.containerPath.TrimStart(PathSeparators);
	writer.WriteEntry(item.path, containerPath);
	}
	} // Dispose of the TarWriter before getting the hash so the final data get written to the tar stream
	uncompressedHash = gz.GetHash();

but before gz.GetHash() is being called. GetHash() will call

sdk-container-builds/Microsoft.NET.Build.Containers/Layer.cs

Line 134 in 4ed9915

sha256Stream.FlushFinalBlock();

Which will set the _finalBlockTransformed flag on CryptoStream. If GetHash() is never called (because an exception occurred), Dispose() will still be called, which tries to call FlushFinalBlock() here:

https://github.com/dotnet/runtime/blob/264d7391ec9f6e698051db0621c5e090d0ae4710/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CryptoStream.cs#L750-L753

But in our Dispose method, we are disposing the hashAlgorithm first, and then calling sha256Stream.Dispose():

sdk-container-builds/Microsoft.NET.Build.Containers/Layer.cs

Lines 138 to 143 in 4ed9915

	protected override void Dispose(bool disposing)
	{
	try
	{
	hashAlgorithm.Dispose();
	sha256Stream.Dispose();

Since the hashAlgorithm is already disposed, it is throwing ODE.

I think the fix here is to flip around the ordering of calling Dispose

sdk-container-builds/Microsoft.NET.Build.Containers/Layer.cs

Lines 138 to 143 in 4ed9915

	protected override void Dispose(bool disposing)
	{
	try
	{
	hashAlgorithm.Dispose();
	sha256Stream.Dispose();

Dispose the hash256Stream first, then dispose the hashAlgorithem.

cc @bartonjs - to see check to make sure what I'm saying makes sense. CryptoStream takes a ICryptoTransform, but doesn't dispose of it. It is up to the caller to dispose of it, but it must dispose of the ICryptoTransform after it disposes the CryptoStream instance.

[bartonjs]

I think a better fix would be to drop CryptoStream altogether. And SHA256 while you're at it. (IncrementalHash.CreateForHash(HashAlgorithmName.SHA256) is designed for this scenario).

[aeons]

@aeons - does it repro for you every time? Can you pass /bl to the command and attach a binlog?

I encounted the same issue on a real project I tried to use container builds in, and so far it happens every time.

I have a binlog. From my looking at it, it doesn't contain any more information on the exception. I can email it to you if you are interested, but I would prefer not to share it here.

[baronfel]

@aeons I just merged the PR that would fix this - if you wouldn't mind, could you try the auto-generated package from GitHub Packages (instructions here) and see if that resolves your issue?

[aeons]

It still fails, but now it seems we get the real exception:

/Users/User/.nuget/packages/microsoft.net.build.containers/0.3.0-alpha.12/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: The "CreateNewImage" task failed unexpectedly. [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.3.0-alpha.12/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: System.IO.IOException: Undefined error: 0 [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.3.0-alpha.12/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at Interop.Sys.GetGroupName(UInt32 gid) [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.3.0-alpha.12/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at System.Formats.Tar.TarWriter.ConstructEntryForWriting(String fullPath, String entryName, FileOptions fileOptions) [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.3.0-alpha.12/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at Microsoft.NET.Build.Containers.Layer.FromFiles(IEnumerable`1 fileList) in /home/runner/work/sdk-container-builds/sdk-container-builds/Microsoft.NET.Build.Containers/Layer.cs:line 54 [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.3.0-alpha.12/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at Microsoft.NET.Build.Containers.Layer.FromDirectory(String directory, String containerPath) in /home/runner/work/sdk-container-builds/sdk-container-builds/Microsoft.NET.Build.Containers/Layer.cs:line 24 [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.3.0-alpha.12/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at Microsoft.NET.Build.Containers.Tasks.CreateNewImage.Execute() in /home/runner/work/sdk-container-builds/sdk-container-builds/Microsoft.NET.Build.Containers/CreateNewImage.cs:line 100 [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.3.0-alpha.12/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/Users/User/test-container/test-container.csproj]
/Users/User/.nuget/packages/microsoft.net.build.containers/0.3.0-alpha.12/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/Users/User/test-container/test-container.csproj]

[baronfel]

Ok, this I expect is because we're using the 'convenience' wrappers for constructing Tar Entry objects, and those wrappers do user/group lookups. We've got an item logged to move away from this and do more explicit creation of the file and directory entries (partially because of this, partially to support explicit user/group mapping for rootless container support), but we haven't done that work quite yet. See #160 for the discussion around that.

[aeons]

Interesting. I wonder why that fails on macOS.

[eerhardt]

FYI - @carlossanlop

Should we re-open this issue? Or open a new one? (Or move it to dotnet/runtime since the exception is coming from the TarWriter?)

[baronfel]

Makes sense to reopen - @aeons' still blocked from the end result, even if you did address a blocker they found.

[mikoskinen]

Any updates on this? Also seeing errors when trying to use this with macOS on ARM.

[jacobcarpenter]

I am also experiencing this issue on a Mac (but this one's Intel)

error MSB4018: The "CreateNewImage" task failed unexpectedly.
error MSB4018: System.IO.IOException: No such file or directory
error MSB4018:    at Interop.Sys.GetGroupName(UInt32 gid)
error MSB4018:    at System.Formats.Tar.TarWriter.ConstructEntryForWriting(String fullPath, String entryName, FileOptions fileOptions)
error MSB4018:    at Microsoft.NET.Build.Containers.Layer.FromFiles(IEnumerable`1 fileList) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Layer.cs:line 55
error MSB4018:    at Microsoft.NET.Build.Containers.Layer.FromDirectory(String directory, String containerPath) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Layer.cs:line 25
error MSB4018:    at Microsoft.NET.Build.Containers.Tasks.CreateNewImage.Execute() in D:\a\_work\1\s\Microsoft.NET.Build.Containers\CreateNewImage.cs:line 106
error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask)

[baronfel]

Sorry folks, no update on this one. The root cause is this line - instead of letting the TAR APIs construct a TarEntry for us, we'd need to manually create the group/user - it seems the metadata on the existing files isn't correct somehow on macOS. Neither Rainer nor I have easy access to a macOS device, though, so it will likely be slow going.

[eerhardt]

@carlossanlop - why doesn't TarWriter work "correctly" in this case?

[jacobcarpenter]

@baronfel Thanks for pointing out the crashing line! I can repro in a simple example and created an issue (which automatically got linked above) with the report.

Additionally, it does look like explicitly creating TarEntry instances and calling the overload of TarWriter.WriteEntry that takes a TarEntry works for me. But as I noted in the bug report, there's a lot of convenience code in the crashing overload, so depending on the types of entries you need to create for the container archive, this might be a pain.

[baronfel]

Luckily we should only need to make Directory and RegularFile entries (at least the last time I took a look at this). I think this works on Windows (as you noted) because on Windows the group/user data is mostly elided. Thanks for writing up the issue on Runtime - you wouldn't by chance be interested in trying to fix it here would you? 🎣

[carlossanlop]

The problem is reported in the callstack to be coming from this line specifically: https://github.com/dotnet/runtime/blob/0ca364767715fee7bc9d0273eec4f30a3c6cc8f0/src/libraries/System.Formats.Tar/src/System/Formats/Tar/TarWriter.Unix.cs#L85

It seems we are getting an ERRNO 0. According to the docs, it is a possible value for a failure to retrieve a groupname or a groupid: https://linux.die.net/man/3/getgrgid_r

I assume we're using getgrgid_r, the thread-safe version of that method. It is supported by MacOSX: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/getgrgid_r.3.html

Since the method that captures the ERRNO is returning null (L85 above), we are throwing here: https://github.com/dotnet/runtime/blob/9d44b9bd4eddd5e523a44056c722bcc93bccca0e/src/libraries/Common/src/Interop/Unix/System.Native/Interop.GetGroupName.cs#L20

So my question is for @baronfel: what groupname/groupid is expected here? It seems it does not exist. Either we get the group created before executing all this tar code, or as you suggested, manually construct the entry.

[baronfel]

The User and group information should come from the User metadata of the base image being build upon - this may be empty! When the container is run, the container runtime is responsible for mounting the layer contents as the user specified by the --user option if one is specified, otherwise the metadata from the image takes precedence.

I wonder if we're been wrong all along in our tooling - perhaps there should almost never be an inferred user or group assignment!

[eerhardt]

manually construct the entry.

I'd really like to avoid this plan to address this issue, if we can. I think it may just be masking a bug in the TarWriter itself. Anyone else using TarWriter.WriteEntry on macOS will hit the same problem.

[jacobcarpenter]

you wouldn't by chance be interested in trying to fix it here would you?

Heh; I locally made the requisite changes to Layer.cs and LocalDocker.cs (plus some additional changes for the DOTNET_ROOT environment variable, which isn't set when dotnet is installed to the default location), but dotnet test reports 5 failures in Test.Microsoft.NET.Build.Containers.Filesystem.dll.

The output is a little hard to read; it seems like there's some interleaving of Docker output with test output... but there are two main types of errors:

1. Unexpected return codes from Docker—I don't really know Docker and haven't done much more than install it, so perhaps I've missed some expected system configuration?
2. The same tar-creation crash happening in what appears to be a cached nuget version of the Build.Containers code... 🫤

Here's a gist with the full test output, if it's helpful:

https://gist.github.com/jacobcarpenter/0b105dc62030a6f3af8bd88b1258d6b2

[jacobcarpenter]

Anyone else using TarWriter.WriteEntry on macOS will hit the same problem.

Yes; correct. Should this Issue title be edited to remove the "on ARM" qualifier?

[carlossanlop]

manually construct the entry.

I'd really like to avoid this plan to address this issue, if we can. I think it may just be masking a bug in the TarWriter itself. Anyone else using TarWriter.WriteEntry on macOS will hit the same problem.

I can submit a fix for WriteEntry to not fail to create the entry if the group isn't found and the return value is 0, and I can request backporting it to .NET 7 too. Sounds good?

[baronfel]

I think that solution would be ok - from what I can see the name is really what matters. If the group or user name is present in the entry, then tar will attempt to use the matching user/group from the destination system during extraction. Only if it can't find the name does it then defer to the numeric identifier, with a final fallback to the currently-executing group/user if neither is found.

[eerhardt]

I can submit a fix for WriteEntry to not fail to create the entry if the group isn't found and the return value is 0, and I can request backporting it to .NET 7 too.

Fixing the underlying issue so others don't run into it sounds good to me.

I'm not sure if it needs to be backported to .NET 7 or not. But if you think so, I wouldn't object.

[carlossanlop]

Thanks both.

Only if it can't find the name does it then defer to the numeric identifier

Is this necessary? The tar entry header has a standard field gid. That field would already have the numeric value stored. I am of the opinion that we should just leave the gname field empty if it wasn't possible to find a matching group name for that gid.

with a final fallback to the currently-executing group/user if neither is found.

Do we want this to happen? This WriteEntry method is writing an entry from an existing file in the filesystem, which already has a user owner and a group owner. I am of the opinion that we should not tamper with the information that we tried to retrieve from the filesystem. Instead, we should just leave empty what we couldn't fully collect.

I'm not sure if it needs to be backported to .NET 7 or not

@baronfel I think you can answer the question since sdk-container-builds is the customer - Would this repo benefit from backporting such fix to .NET 7, or should it just stay in main for .NET 8?

[baronfel]

defaults for naming

I agree that leaving it unmodified is the best course of action for the .NET API - I was just trying to relate the behavior of tar during extraction as some context that I thought was interesting. Sorry to muddy the water :)

need for backport

We would benefit a ton from a fix here - it seems like many (all?) of our users on macOS are hitting this wall and only able to succeed using Windows or Linux CI machines. For 7.0.2xx this feature is easily lit up for WebSDK projects, and for 7.0.3xx we're targeting full, automatic support for all project types via shipping in the .NET SDK, so I'd hope that usage of the tool (and therefore errors from macOS users) would also go up.

[baronfel]

This fix is in the runtime and has been merged and backported, so I'll close this issue for now. Thank you all for the report, discussion, and thank you @carlossanlop for the fix!