[macOS-arm64] CoreCLR System.Security.Cryptography.OpenSsl.Tests fail on M1 Helix queues #49083

sdmaclea · 2021-03-03T21:21:01Z

From https://dev.azure.com/dnceng/public/_build/results?buildId=1019012&view=results

No usable version of libssl was found

----- start Tue Mar 2 10:20:41 PST 2021 =============== To repro directly: =====================================================
pushd .
/tmp/helix/working/A43F08FC/p/dotnet exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json xunit.console.dll System.Security.Cryptography.OpenSsl.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing 
popd
===========================================================================================================
/private/tmp/helix/working/A43F08FC/w/C8540A7F/e /private/tmp/helix/working/A43F08FC/w/C8540A7F/e
  Discovering: System.Security.Cryptography.OpenSsl.Tests (method display = ClassAndMethod, method display options = None)
No usable version of libssl was found
./RunTests.sh: line 162: 59272 Abort trap: 6           (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json xunit.console.dll System.Security.Cryptography.OpenSsl.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/private/tmp/helix/working/A43F08FC/w/C8540A7F/e
----- end Tue Mar 2 10:20:43 PST 2021 ----- exit code 134 ----------------------------------------------------------

----- start Wed Mar 3 17:37:29 PST 2021 =============== To repro directly: =====================================================
pushd .
/tmp/helix/working/C24F0A5F/p/dotnet exec --runtimeconfig System.Security.Cryptography.Algorithms.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.Algorithms.Tests.deps.json xunit.console.dll System.Security.Cryptography.Algorithms.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=failing 
popd
===========================================================================================================
/private/tmp/helix/working/C24F0A5F/w/B80F0986/e /private/tmp/helix/working/C24F0A5F/w/B80F0986/e
  Discovering: System.Security.Cryptography.Algorithms.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.Algorithms.Tests (found 1595 of 1601 test cases)
  Starting:    System.Security.Cryptography.Algorithms.Tests (parallel test collections = on, max threads = 8)
    System.Security.Cryptography.Encryption.TripleDes.Tests.TripleDESContractTests.Windows7DoesNotSupportCFB64 [SKIP]
      Condition(s) not met: "IsWindows7"
    System.Security.Cryptography.Dsa.Tests.DSAImportExport.Import_2048 [SKIP]
      Condition(s) not met: "SupportsFips186_3"
    System.Security.Cryptography.Dsa.Tests.DSAImportExport.ExportAutoKey [SKIP]
      Condition(s) not met: "SupportsKeyGeneration"
    System.Security.Cryptography.Encryption.Des.Tests.DesContractTests.Windows7DoesNotSupportCFB [SKIP]
      Condition(s) not met: "IsWindows7"
    System.Security.Cryptography.Encryption.Aes.Tests.AesModeTests.Windows7DoesNotSupportCFB128 [SKIP]
      Condition(s) not met: "IsWindows7"
No usable version of libssl was found
./RunTests.sh: line 162: 37821 Abort trap: 6           (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.Algorithms.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.Algorithms.Tests.deps.json xunit.console.dll System.Security.Cryptography.Algorithms.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/private/tmp/helix/working/C24F0A5F/w/B80F0986/e
----- end Wed Mar 3 17:37:37 PST 2021 ----- exit code 134 ----------------------------------------------------------

/cc @directhex

The text was updated successfully, but these errors were encountered:

ghost · 2021-03-03T21:21:04Z

Tagging subscribers to this area: @safern, @ViktorHofer, @Anipik
See info in area-owners.md if you want to be subscribed.

Issue Details

From https://dev.azure.com/dnceng/public/_build/results?buildId=1019012&view=results

No usable version of libssl was found

----- start Tue Mar 2 10:20:41 PST 2021 =============== To repro directly: =====================================================
pushd .
/tmp/helix/working/A43F08FC/p/dotnet exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json xunit.console.dll System.Security.Cryptography.OpenSsl.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing 
popd
===========================================================================================================
/private/tmp/helix/working/A43F08FC/w/C8540A7F/e /private/tmp/helix/working/A43F08FC/w/C8540A7F/e
  Discovering: System.Security.Cryptography.OpenSsl.Tests (method display = ClassAndMethod, method display options = None)
No usable version of libssl was found
./RunTests.sh: line 162: 59272 Abort trap: 6           (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json xunit.console.dll System.Security.Cryptography.OpenSsl.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/private/tmp/helix/working/A43F08FC/w/C8540A7F/e
----- end Tue Mar 2 10:20:43 PST 2021 ----- exit code 134 ----------------------------------------------------------

/cc @directhex

Author:	sdmaclea
Assignees:	-
Labels:	`arch-arm64`, `area-Infrastructure-libraries`, `os-mac-os-x-big-sur`
Milestone:	6.0.0

safern · 2021-03-04T01:53:37Z

cc: @bartonjs

@ilyas1974 does the M1 machines have libssl installed?

sdmaclea · 2021-03-04T02:13:44Z

More specifically arm64 libssl.

sdmaclea · 2021-03-12T20:55:37Z

I am seeing the same error on my local M1. These libraries are present...

% pkg-config --list-all | grep ssl
libssl                              OpenSSL-libssl - Secure Sockets Layer and cryptography libraries
openssl                             OpenSSL - Secure Sockets Layer and cryptography libraries and tools
libevent_openssl                    libevent_openssl - libevent_openssl adds openssl-based TLS support to libevent

elahmo · 2021-05-30T08:11:54Z

Is there any workaround for this at the moment? I am getting this error as well and [email protected] is installed via Homebrew. on non-M1 machines I was able to fix it via symlinking, but on M1 it doesn't seem to work.

ViktorHofer · 2021-08-06T16:37:23Z

@sdmaclea do you know what should be done here? It's unclear to me with the current amount of information if this in an infrastructure issue that we should own.

bartonjs · 2021-08-06T19:16:37Z

@elahmo I think you're having the problem from #54597 (and our CI might, too).

Basically, Homebrew is giving you the arm64 OpenSSL, but you're running x64 .NET. You need to get the arm64 .NET (assuming it's ready to be used) or the x64 OpenSSL.

elahmo · 2021-08-08T07:48:26Z

@bartonjs yeah, this was the case. Unfortunately, for the project I am using, there is no arm64 dotnet version (3.1), but at least I am able to run the stack locally now.

ViktorHofer · 2021-08-16T15:16:59Z

@bartonjs is there anything left to do as part of this issue?

bartonjs · 2021-08-16T15:51:22Z

@ViktorHofer I haven't seen reports of tests failing, other than this issue... so... maybe everything's good? But I don't know authoritatively, and don't really know how to see if the tests have been running.

ViktorHofer · 2021-08-20T13:58:29Z

and don't really know how to see if the tests have been running.

Just checked our platform matrix and rolling builds, we don't run OSX arm64 libraries tests in CI. Assuming that is intentional, moving to 7.0.

danmoseley · 2021-09-09T16:39:45Z

@ViktorHofer per my onenote table, we do test macOS 11 w/Apple Silicon on both PR and rolling. Can you double check? And confirm that we're not running on Rosetta? We fully support M1's.

macOS 11.00 (ARM64)		PR + rolling	Nightly

ghost · 2021-09-09T16:40:04Z

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq, @GrabYourPitchforks
See info in area-owners.md if you want to be subscribed.

Issue Details

From https://dev.azure.com/dnceng/public/_build/results?buildId=1019012&view=results

No usable version of libssl was found

----- start Tue Mar 2 10:20:41 PST 2021 =============== To repro directly: =====================================================
pushd .
/tmp/helix/working/A43F08FC/p/dotnet exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json xunit.console.dll System.Security.Cryptography.OpenSsl.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing 
popd
===========================================================================================================
/private/tmp/helix/working/A43F08FC/w/C8540A7F/e /private/tmp/helix/working/A43F08FC/w/C8540A7F/e
  Discovering: System.Security.Cryptography.OpenSsl.Tests (method display = ClassAndMethod, method display options = None)
No usable version of libssl was found
./RunTests.sh: line 162: 59272 Abort trap: 6           (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json xunit.console.dll System.Security.Cryptography.OpenSsl.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/private/tmp/helix/working/A43F08FC/w/C8540A7F/e
----- end Tue Mar 2 10:20:43 PST 2021 ----- exit code 134 ----------------------------------------------------------

----- start Wed Mar 3 17:37:29 PST 2021 =============== To repro directly: =====================================================
pushd .
/tmp/helix/working/C24F0A5F/p/dotnet exec --runtimeconfig System.Security.Cryptography.Algorithms.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.Algorithms.Tests.deps.json xunit.console.dll System.Security.Cryptography.Algorithms.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=failing 
popd
===========================================================================================================
/private/tmp/helix/working/C24F0A5F/w/B80F0986/e /private/tmp/helix/working/C24F0A5F/w/B80F0986/e
  Discovering: System.Security.Cryptography.Algorithms.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.Algorithms.Tests (found 1595 of 1601 test cases)
  Starting:    System.Security.Cryptography.Algorithms.Tests (parallel test collections = on, max threads = 8)
    System.Security.Cryptography.Encryption.TripleDes.Tests.TripleDESContractTests.Windows7DoesNotSupportCFB64 [SKIP]
      Condition(s) not met: "IsWindows7"
    System.Security.Cryptography.Dsa.Tests.DSAImportExport.Import_2048 [SKIP]
      Condition(s) not met: "SupportsFips186_3"
    System.Security.Cryptography.Dsa.Tests.DSAImportExport.ExportAutoKey [SKIP]
      Condition(s) not met: "SupportsKeyGeneration"
    System.Security.Cryptography.Encryption.Des.Tests.DesContractTests.Windows7DoesNotSupportCFB [SKIP]
      Condition(s) not met: "IsWindows7"
    System.Security.Cryptography.Encryption.Aes.Tests.AesModeTests.Windows7DoesNotSupportCFB128 [SKIP]
      Condition(s) not met: "IsWindows7"
No usable version of libssl was found
./RunTests.sh: line 162: 37821 Abort trap: 6           (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.Algorithms.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.Algorithms.Tests.deps.json xunit.console.dll System.Security.Cryptography.Algorithms.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/private/tmp/helix/working/C24F0A5F/w/B80F0986/e
----- end Wed Mar 3 17:37:37 PST 2021 ----- exit code 134 ----------------------------------------------------------

/cc @directhex

Author:	sdmaclea
Assignees:	-
Labels:	`arch-arm64`, `area-System.Security`, `disabled-test`, `os-mac-os-x-big-sur`
Milestone:	6.0.0

danmoseley · 2021-09-09T19:14:13Z

Running this query

WorkItems 
| where FriendlyName == "System.Security.Cryptography.Algorithms.Tests"
| join kind= inner (
   Jobs  | project  QueueName , JobId, Build, Type, Source,
    Branch,
  Pipeline = tostring(parse_json(Properties).DefinitionName),
  Pipeline_Configuration = tostring(parse_json(Properties).configuration),
  OS = QueueName,
  Arch = tostring(parse_json(Properties).architecture)
  | where QueueName contains "arm" and QueueName  contains "osx"
) on JobId
| where Queued > ago (7d)
| summarize count() by ExitCode, Pipeline, Pipeline_Configuration, Arch,QueueName, FriendlyName, PassCount, FailCount, ConsoleUri  //  bin(Finished, 1d), 
| order by ExitCode asc

I seem to find Catalyst etc tests only.

ViktorHofer · 2021-09-10T13:52:20Z

Only the runtime tests currently test on arm64:

runtime/eng/pipelines/runtime.yml

Line 969 in b683032

- OSX_arm64

which leverages the OSX.1100.ARM64.Open queue:

runtime/eng/pipelines/coreclr/templates/helix-queues-setup.yml

Line 110 in b683032

- OSX.1100.ARM64.Open

I don't think we yet run libraries tests on OSX 11 arm64.

danmoseley · 2021-09-10T14:58:59Z

We should..

ViktorHofer · 2021-09-10T15:15:15Z

Who usually tracks adding these configurations? That person might have some more knowledge on why it isn't there yet. Would that be @wfurt?

wfurt · 2021-09-10T17:01:24Z

I can take a look. I have ARM macBook so I can easily do local test runs before we jumping to CI.

wfurt · 2021-09-10T17:24:10Z

BTW this may be separate issue but since we don't use openssl for SSL/TLS is there specific reason why we cannot use the libressl that comes with OS? It would be nice IMHO to not have brew dependency and/or make it optional.

bartonjs · 2021-09-10T17:50:33Z

is there specific reason why we cannot use the libressl that comes with OS?

The types that have OpenSSL in their names shouldn't use libressl, since they're not pointer-compatible.

We could possibly use libressl for CCM/GCM/ChaCha-Poly, but that's orthogonal to the tests that are for types specifically using OpenSSL.

danmoseley · 2021-09-11T01:32:31Z

I can take a look. I have ARM macBook so I can easily do local test runs before we jumping to CI.

That would help for sure!

danmoseley · 2022-03-11T16:46:50Z

@bartonjs @wfurt -- I noticed these two test libraries are entirely disabled on M1's - which seems a bit of a precarious place to be. Is this is a significant hole - we apparently shipped this way?

bartonjs · 2022-03-11T19:20:50Z

It's certainly not a pleasing state to be in, but I'm not expecting a lot of major issues.

Our use of OpenSSL on macOS is limited to RSAOpenSsl, DSAOpenSsl, ECDsaOpenSsl, ECDiffieHellmanOpenSsl, AesGcm, and AesCcm. The first four are pretty obvious that OpenSSL is required, and no platform code will instantiate them on macOS. The latter two aren't instantiated by platform code, and use the static IsSupported pattern to indicate that they're not always available.

The problem at the time was that the libssl/libcrypto that the test machines get from Homebrew weren't the same architecture as the dotnet process. Assuming that we can solve the architecture mismatch on our test rigs (presumably there are architecture-specific library load paths, and we need to a) download two copies of OpenSSL via homebrew and b) link them in with the arch paths instead of the common path) we can turn the tests back on.

If it's the case that Homebrew doesn't offer OpenSSL arm64 then the lightup features that depend on OpenSSL are just not available on that platform.

wfurt · 2022-07-15T21:31:22Z

This should be workable now IMHO

furt@Tomass-MacBook-Pro ~ % arch --arm64 brew install  [email protected]
....
....
==> Installing [email protected]
==> Pouring [email protected]_big_sur.bottle.tar.gz

and

furt@Tomass-MacBook-Pro ~ % file /opt/homebrew/opt/[email protected]/lib/*
/opt/homebrew/opt/[email protected]/lib/engines-1.1:         directory
/opt/homebrew/opt/[email protected]/lib/libcrypto.1.1.dylib: Mach-O 64-bit dynamically linked shared library arm64
/opt/homebrew/opt/[email protected]/lib/libcrypto.a:         current ar archive random library
/opt/homebrew/opt/[email protected]/lib/libcrypto.dylib:     Mach-O 64-bit dynamically linked shared library arm64
/opt/homebrew/opt/[email protected]/lib/libssl.1.1.dylib:    Mach-O 64-bit dynamically linked shared library arm64
/opt/homebrew/opt/[email protected]/lib/libssl.a:            current ar archive random library
/opt/homebrew/opt/[email protected]/lib/libssl.dylib:        Mach-O 64-bit dynamically linked shared library arm64
/opt/homebrew/opt/[email protected]/lib/pkgconfig:           directory

wfurt · 2022-07-15T22:19:50Z

at least, all the tests seems to pass locally.

 /Users/furt/github/wfurt-runtime/artifacts/bin/testhost/net7.0-OSX-Debug-arm64/dotnet exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json xunit.console.dll System.Security.Cryptography.OpenSsl.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
  ~/github/wfurt-runtime/artifacts/bin/System.Security.Cryptography.OpenSsl.Tests/Debug/net7.0-unix ~/github/wfurt-runtime/src/libraries/System.Security.Cryptography.OpenSsl/tests
    Discovering: System.Security.Cryptography.OpenSsl.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Security.Cryptography.OpenSsl.Tests (found 823 of 829 test cases)
    Starting:    System.Security.Cryptography.OpenSsl.Tests (parallel test collections = on, max threads = 8)
      System.Security.Cryptography.Rsa.Tests.KeyGeneration.GenerateMaxKey [SKIP]
        Condition(s) not met: "IsStressModeEnabled"
    Finished:    System.Security.Cryptography.OpenSsl.Tests
  === TEST EXECUTION SUMMARY ===
     System.Security.Cryptography.OpenSsl.Tests  Total: 1187, Errors: 0, Failed: 0, Skipped: 1, Time: 10.631s

sdmaclea added arch-arm64 area-Infrastructure-libraries os-macos-bigsur (macOS11) labels Mar 3, 2021

sdmaclea added this to the 6.0.0 milestone Mar 3, 2021

dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Mar 3, 2021

ViktorHofer removed the untriaged New issue has not been triaged by the area owner label Mar 24, 2021

sdmaclea added the disabled-test The test is disabled in source code against the issue label Apr 8, 2021

ViktorHofer modified the milestones: 6.0.0, 7.0.0 Aug 20, 2021

danmoseley modified the milestones: 7.0.0, 6.0.0 Sep 9, 2021

danmoseley added area-System.Security and removed area-Infrastructure-libraries labels Sep 9, 2021

jeffhandley assigned wfurt Sep 15, 2021

jeffhandley modified the milestones: 6.0.0, 7.0.0 Sep 22, 2021

jeffhandley mentioned this issue Jan 29, 2022

System.Security work planned for .NET 7 #64488

Closed

37 tasks

wfurt mentioned this issue Jul 15, 2022

enable OpenSSL tests on arm64 mac #72294

Merged

ghost added the in-pr There is an active PR which will close this issue when it is merged label Jul 15, 2022

wfurt mentioned this issue Jul 25, 2022

Add usable OpenSSL package to ARM64 Mac queues dotnet/arcade#10139

Closed

wfurt closed this as completed in #72294 Aug 3, 2022

ghost removed the in-pr There is an active PR which will close this issue when it is merged label Aug 3, 2022

ghost locked as resolved and limited conversation to collaborators Sep 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[macOS-arm64] CoreCLR System.Security.Cryptography.OpenSsl.Tests fail on M1 Helix queues #49083

[macOS-arm64] CoreCLR System.Security.Cryptography.OpenSsl.Tests fail on M1 Helix queues #49083

sdmaclea commented Mar 3, 2021 •

edited

Loading

ghost commented Mar 3, 2021

safern commented Mar 4, 2021

sdmaclea commented Mar 4, 2021

sdmaclea commented Mar 12, 2021

elahmo commented May 30, 2021

ViktorHofer commented Aug 6, 2021

bartonjs commented Aug 6, 2021

elahmo commented Aug 8, 2021

ViktorHofer commented Aug 16, 2021

bartonjs commented Aug 16, 2021

ViktorHofer commented Aug 20, 2021 •

edited

Loading

danmoseley commented Sep 9, 2021

ghost commented Sep 9, 2021

danmoseley commented Sep 9, 2021

ViktorHofer commented Sep 10, 2021 •

edited

Loading

danmoseley commented Sep 10, 2021

ViktorHofer commented Sep 10, 2021 •

edited

Loading

wfurt commented Sep 10, 2021

wfurt commented Sep 10, 2021

bartonjs commented Sep 10, 2021

danmoseley commented Sep 11, 2021

danmoseley commented Mar 11, 2022

bartonjs commented Mar 11, 2022

wfurt commented Jul 15, 2022

wfurt commented Jul 15, 2022

[macOS-arm64] CoreCLR System.Security.Cryptography.OpenSsl.Tests fail on M1 Helix queues #49083

[macOS-arm64] CoreCLR System.Security.Cryptography.OpenSsl.Tests fail on M1 Helix queues #49083

Comments

sdmaclea commented Mar 3, 2021 • edited Loading

ghost commented Mar 3, 2021

safern commented Mar 4, 2021

sdmaclea commented Mar 4, 2021

sdmaclea commented Mar 12, 2021

elahmo commented May 30, 2021

ViktorHofer commented Aug 6, 2021

bartonjs commented Aug 6, 2021

elahmo commented Aug 8, 2021

ViktorHofer commented Aug 16, 2021

bartonjs commented Aug 16, 2021

ViktorHofer commented Aug 20, 2021 • edited Loading

danmoseley commented Sep 9, 2021

ghost commented Sep 9, 2021

danmoseley commented Sep 9, 2021

ViktorHofer commented Sep 10, 2021 • edited Loading

danmoseley commented Sep 10, 2021

ViktorHofer commented Sep 10, 2021 • edited Loading

wfurt commented Sep 10, 2021

wfurt commented Sep 10, 2021

bartonjs commented Sep 10, 2021

danmoseley commented Sep 11, 2021

danmoseley commented Mar 11, 2022

bartonjs commented Mar 11, 2022

wfurt commented Jul 15, 2022

wfurt commented Jul 15, 2022

sdmaclea commented Mar 3, 2021 •

edited

Loading

ViktorHofer commented Aug 20, 2021 •

edited

Loading

ViktorHofer commented Sep 10, 2021 •

edited

Loading

ViktorHofer commented Sep 10, 2021 •

edited

Loading