Support Kubernetes Go binaries in Nanoserver Images #72

jeremyje · 2020-11-03T22:25:38Z

Go is a statically linked language but for OS specific logic there's the need to occasionally use CGO. Kubernetes binaries tends to have a dependency on netapi32.dll specifically for lookup_windows.go in os.User function. There's logic to query Active Directory user information. For more context see golang/go#21867. It appears that the DLL was removed in Windows Server 1709.

It'd be nice if Windows Nanoserver images had the necessary support to run most Kubernetes binaries without doing the netapi32.dll copy trick. There are probably other dependencies that Kubernetes for Windows needs but this appears to be the most significant.

I think it would also be useful to understand what is and is not included in Windows Nanoserver images. Some of this information used to be documented at https://docs.microsoft.com/en-us/previous-versions/windows/desktop/legacy/mt588480(v=vs.85).

A nondisruptive option might be to introduce a Nanoserver variant that includes the DLLs necessary to run Go binaries with http://golang.org/x/sys/windows linked in and offer it here, https://hub.docker.com/publishers/microsoftowner.

jeremyje · 2020-11-03T22:26:26Z

CC: @craiglpeters @pjh

jsturtevant · 2020-11-04T16:04:02Z

@weijuans-msft for what is/isn't included in nano

ghost · 2021-01-02T16:00:26Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

ghost · 2021-02-02T16:00:23Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

ghost · 2021-03-04T16:01:10Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

jeremyje · 2021-03-04T19:14:05Z

https://techcommunity.microsoft.com/t5/containers/what-s-new-for-windows-containers-on-windows-server-2022/ba-p/2167036 came out. Is netapi32.dll going to be a part of that?

weijuans-msft · 2021-03-31T17:38:39Z

@jeremyje No, we didn't make any changes in the Nanoserver for Windows Server 2022 release.

What are you use cases? If you are comfortable to share publicly, please email us at [email protected]. It'll be helpful to understand the use case so we can prioritize.

Is this the file all you need to be added to Nanoserver container to enable the scenario here?

/Windows/System32/netapi32.dll

weijuans-msft · 2021-04-16T19:44:17Z

For Microsoft folks, internally I have created this scenario in Azure DevOps to track: 32512068.

weijuans-msft · 2021-04-27T04:51:10Z

And this Microsoft internal tracking deliverable in ADO #32788516.

johnSchnake · 2021-05-14T14:26:32Z

FWIW I just ran into this as well; Sonobuoy is working on supporting Windows nodes in K8s and I initially tried nanoserver and ran into the same missing DLL. That's the only one I know causes a problem but I haven't messed with trying to add that DLL back into the image or anything. Easiest fix for us is just to use servercore instead, not sure if you have another recommendation.

ghost · 2021-06-13T16:01:04Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

ghost · 2021-07-14T16:01:08Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

weijuans-msft · 2021-07-14T16:19:21Z

We have it in our internal backlog and will evaluate with other similar related asks and prioritize.

ghost · 2021-08-14T16:01:04Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

To fix the following failure: ``` panic: Failed to load netapi32.dll: The specified module could not be found. goroutine 1 [running]: syscall.(*LazyProc).mustFind(...) c:/go/src/syscall/dll_windows.go:320 syscall.(*LazyProc).Addr(...) c:/go/src/syscall/dll_windows.go:327 syscall.NetGetJoinInformation(0x0, 0xc00061fa48, 0xc00061fa44, 0x16, 0xc0000c96e0) c:/go/src/syscall/zsyscall_windows.go:1804 +0x107 os/user.isDomainJoined(0xc00061faa0, 0xf3624e, 0x0) c:/go/src/os/user/lookup_windows.go:18 +0x5a os/user.lookupFullName(0xc000063b30, 0xc, 0xc000259bc0, 0x16, 0xc0000c96e0, 0x23, 0x16, 0xc0000c96e0, 0x23, 0x1) c:/go/src/os/user/lookup_windows.go:51 +0x2d os/user.newUser(0xc000063ad0, 0xc, 0xc000063b10, 0xc, 0xc0000c9680, 0x1f, 0xc000259bc0, 0x16, 0xc000063b30, 0xc, ...) c:/go/src/os/user/lookup_windows.go:181 +0xd4 os/user.current(0x0, 0x0, 0x0) c:/go/src/os/user/lookup_windows.go:225 +0x2d8 os/user.Current.func1() c:/go/src/os/user/lookup.go:15 +0x29 sync.(*Once).doSlow(0x593ff00, 0x40b75b0) c:/go/src/sync/once.go:66 +0xf7 sync.(*Once).Do(...) c:/go/src/sync/once.go:57 os/user.Current(0xc0002599c0, 0x1b, 0x3f56005) c:/go/src/os/user/lookup.go:15 +0x105 github.com/golang/glog.init.1() C:/gomodcache/github.com/golang/[email protected]/gomodcache/github.com/golang/[email protected]/glog_file.go:63 +0x4b ``` This ugly hack is needed because the [bump of github.com/dgraph-io/ristretto from 0.0.3 to 0.1.0][1] [introduced a new dependency to `glog`][2] which is triggering the following issue with the Windows nano docker image: [3] Once [3] is solved, this hack can be reverted. [1] #8500 [2] hypermodeinc/ristretto#263 [3] microsoft/Windows-Containers#72

…#9006) To fix the following failure: ``` panic: Failed to load netapi32.dll: The specified module could not be found. goroutine 1 [running]: syscall.(*LazyProc).mustFind(...) c:/go/src/syscall/dll_windows.go:320 syscall.(*LazyProc).Addr(...) c:/go/src/syscall/dll_windows.go:327 syscall.NetGetJoinInformation(0x0, 0xc00061fa48, 0xc00061fa44, 0x16, 0xc0000c96e0) c:/go/src/syscall/zsyscall_windows.go:1804 +0x107 os/user.isDomainJoined(0xc00061faa0, 0xf3624e, 0x0) c:/go/src/os/user/lookup_windows.go:18 +0x5a os/user.lookupFullName(0xc000063b30, 0xc, 0xc000259bc0, 0x16, 0xc0000c96e0, 0x23, 0x16, 0xc0000c96e0, 0x23, 0x1) c:/go/src/os/user/lookup_windows.go:51 +0x2d os/user.newUser(0xc000063ad0, 0xc, 0xc000063b10, 0xc, 0xc0000c9680, 0x1f, 0xc000259bc0, 0x16, 0xc000063b30, 0xc, ...) c:/go/src/os/user/lookup_windows.go:181 +0xd4 os/user.current(0x0, 0x0, 0x0) c:/go/src/os/user/lookup_windows.go:225 +0x2d8 os/user.Current.func1() c:/go/src/os/user/lookup.go:15 +0x29 sync.(*Once).doSlow(0x593ff00, 0x40b75b0) c:/go/src/sync/once.go:66 +0xf7 sync.(*Once).Do(...) c:/go/src/sync/once.go:57 os/user.Current(0xc0002599c0, 0x1b, 0x3f56005) c:/go/src/os/user/lookup.go:15 +0x105 github.com/golang/glog.init.1() C:/gomodcache/github.com/golang/[email protected]/gomodcache/github.com/golang/[email protected]/glog_file.go:63 +0x4b ``` This ugly hack is needed because the [bump of github.com/dgraph-io/ristretto from 0.0.3 to 0.1.0][1] [introduced a new dependency to `glog`][2] which is triggering the following issue with the Windows nano docker image: [3] Once [3] is solved, this hack can be reverted. [1] #8500 [2] hypermodeinc/ristretto#263 [3] microsoft/Windows-Containers#72

ghost · 2021-09-14T16:01:04Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

ghost · 2022-05-06T16:01:10Z

This issue has been open for 30 days with no updates.
@msjingli, please provide an update or close this issue.

mloskot · 2022-05-26T10:44:13Z

@weijuans-msft

If there is data on (...) use cases/scenarios, please do share.

There is quite a lot of GUI-less software servers that require the Network Management API, for example licencing servers like Gemalto's, and it would be very helpful if such serves run on nanoserver. Those run on servercore containers fine, but it is hard to justify the extra 1 GB of container size.
The nanoserver offers all features necessary and just the handful of DLLs is missing.

m-deepakraja · 2022-06-24T21:41:37Z

Any update on this issue?

weijuans-msft · 2022-07-20T04:44:50Z

Our engineer is getting close to finish the code change. Next need to test, and also figure out a shipping vehicle. Curious what OS are folks using? Windows Server 2019? Windows Server 2022? Our current engineering process is we will first make the change in the active branch which is Windows Server vNext (general Insider builds are out; we are working on getting the container Insider builds out too). Will folks to be interested in testing with Windows Server vNext? @jterry75 @mloskot @jeremyje

mloskot · 2022-07-20T09:32:35Z

Curious what OS are folks using? Windows Server 2019? Windows Server 2022?

2019, but if Windows Server 2022 release delivers NanoServer improvements, then switching to 2022 is not a problem at all.

Will folks to be interested in testing with Windows Server vNext?

Yes, I am interested to test any new container images of NanoServer to see if they can run my software, if I can switch over from ServerCore.

TBBle · 2022-07-20T10:59:35Z

From the ABI stability support announced with Server 2022, as I recall, for process isolation a Server vNext container image will need a Server vNext host, but a Server 2022 container image will work on both 2022 and vNext hosts (and Windows 11). Is that accurate?

weijuans-msft · 2022-07-20T16:46:26Z

@mloskot Windows Server 2022 GA'ed last Sept hence any changes to that will be released with the shipping vehicle of the monthly patches (usually the 2nd Tue of the month). In another word, currently there is no major improvement in Nano server of the Server 2022 release. But we are and will be doing more on Windows Server vNext, and if those work out, we'll back port to Server 2022. Yeah, that takes a bit time. If I had a magic wand, I would love to make it faster.

Great to see the interest in Windows Server vNext Insider builds. Curious how you usually use it? Basically I could use more adoption data of the Insider builds to convince my team to prioritize this :).

@TBBle Good observation! You are right, Windows Server 2022 container images will be the first that can be run on a host OS at the same version or higher version as a process isolated container. That is the host will need to be Server 2022 or higher.

jterry75 · 2022-07-22T17:08:52Z

@weijuans-msft - We are also on WS2019 and WS2022 for prod (no vNext), but we could certainly test on vNext to support the changes and provide feedback.

ghost · 2022-08-22T16:03:38Z

This issue has been open for 30 days with no updates.
@msjingli, please provide an update or close this issue.

ghost · 2022-09-21T16:03:59Z

This issue has been open for 30 days with no updates.
@msjingli, please provide an update or close this issue.

mloskot · 2022-09-21T16:15:52Z

(Don't close this yet.)

ghost · 2022-10-23T16:02:47Z

This issue has been open for 30 days with no updates.
@akarshm, please provide an update or close this issue.

…DataDog#9006) To fix the following failure: ``` panic: Failed to load netapi32.dll: The specified module could not be found. goroutine 1 [running]: syscall.(*LazyProc).mustFind(...) c:/go/src/syscall/dll_windows.go:320 syscall.(*LazyProc).Addr(...) c:/go/src/syscall/dll_windows.go:327 syscall.NetGetJoinInformation(0x0, 0xc00061fa48, 0xc00061fa44, 0x16, 0xc0000c96e0) c:/go/src/syscall/zsyscall_windows.go:1804 +0x107 os/user.isDomainJoined(0xc00061faa0, 0xf3624e, 0x0) c:/go/src/os/user/lookup_windows.go:18 +0x5a os/user.lookupFullName(0xc000063b30, 0xc, 0xc000259bc0, 0x16, 0xc0000c96e0, 0x23, 0x16, 0xc0000c96e0, 0x23, 0x1) c:/go/src/os/user/lookup_windows.go:51 +0x2d os/user.newUser(0xc000063ad0, 0xc, 0xc000063b10, 0xc, 0xc0000c9680, 0x1f, 0xc000259bc0, 0x16, 0xc000063b30, 0xc, ...) c:/go/src/os/user/lookup_windows.go:181 +0xd4 os/user.current(0x0, 0x0, 0x0) c:/go/src/os/user/lookup_windows.go:225 +0x2d8 os/user.Current.func1() c:/go/src/os/user/lookup.go:15 +0x29 sync.(*Once).doSlow(0x593ff00, 0x40b75b0) c:/go/src/sync/once.go:66 +0xf7 sync.(*Once).Do(...) c:/go/src/sync/once.go:57 os/user.Current(0xc0002599c0, 0x1b, 0x3f56005) c:/go/src/os/user/lookup.go:15 +0x105 github.com/golang/glog.init.1() C:/gomodcache/github.com/golang/[email protected]/gomodcache/github.com/golang/[email protected]/glog_file.go:63 +0x4b ``` This ugly hack is needed because the [bump of github.com/dgraph-io/ristretto from 0.0.3 to 0.1.0][1] [introduced a new dependency to `glog`][2] which is triggering the following issue with the Windows nano docker image: [3] Once [3] is solved, this hack can be reverted. [1] DataDog#8500 [2] hypermodeinc/ristretto#263 [3] microsoft/Windows-Containers#72

ghost · 2022-11-22T16:03:10Z

This issue has been open for 30 days with no updates.
@akarshm, please provide an update or close this issue.

mloskot · 2022-11-22T16:10:56Z

/cc @fady-azmy-msft @akarshm

ghost · 2022-12-23T16:02:45Z

This issue has been open for 30 days with no updates.
@akarshm, please provide an update or close this issue.

mloskot · 2022-12-23T21:50:45Z

Bump

microsoft-github-policy-service · 2023-04-21T17:30:45Z

This issue has been open for 30 days with no updates.
@akarshm, please provide an update or close this issue.

microsoft-github-policy-service · 2023-04-21T17:30:53Z

This issue has been open for 30 days with no updates.
@akarshm, please provide an update or close this issue.

mloskot · 2023-04-21T18:49:15Z

Please, don't close and instead, please, make NanoServer go for Go!

akarshm · 2023-05-30T23:12:46Z

This is in our backlog and will be considered a feature item for the next major release for Windows Server Containers. Unfortunately, I don't have a timeline to share at this time.

TBBle · 2023-05-31T12:52:09Z

Is it reasonable to equate "next major release for Windows Server Containers" with "next Windows Server LTSC"? I recognise that this isn't committed for that next major release, just wondering if there's a distinct release cadence for Windows Server Containers vs Windows Server.

akarshm · 2023-05-31T22:30:06Z

You're right. Windows Server Container releases are aligned to the Windows Server release cadence.

ghost added the triage New and needs attention label Nov 3, 2020

vrapolinario assigned weijuans-msft Dec 2, 2020

vrapolinario added Image composition and removed triage New and needs attention labels Dec 2, 2020

micahyoung mentioned this issue Dec 14, 2020

Apps using os/user won't build on Nanoserver due to missing NetApi32.dll docker-library/golang#348

Closed

johnSchnake mentioned this issue May 14, 2021

Nanoserver is missing DLLs required vmware-tanzu/sonobuoy#1263

Closed

L3n41c mentioned this issue Aug 27, 2021

Copy netapi32.dll from servercore image in nano-based Windows image DataDog/datadog-agent#9006

Merged

5 tasks

L3n41c mentioned this issue Sep 2, 2021

Backport #9006 to 7.31.x: Copy netapi32.dll from servercore image in nano-based Windows image DataDog/datadog-agent#9045

Closed

5 tasks

fady-azmy-msft assigned akarshm and unassigned msjingli Sep 22, 2022

fady-azmy-msft added the enhancement New feature or request label May 16, 2023

paulcacheux mentioned this issue May 27, 2023

fix windows nanoserver crash on glog v1.1.x DataDog/datadog-agent#17340

Merged

10 tasks

akarshm closed this as completed May 30, 2023

avin3sh mentioned this issue Dec 27, 2023

Using gMSA on multiple containers simultaneously causes Domain Trust Relationship to fail #405

Closed

jsturtevant mentioned this issue Feb 13, 2025

How to troubleshoot why nanoserver is failing #564

Open

Support Kubernetes Go binaries in Nanoserver Images #72

Support Kubernetes Go binaries in Nanoserver Images #72

Comments

jeremyje commented Nov 3, 2020

jeremyje commented Nov 3, 2020 • edited Loading

jsturtevant commented Nov 4, 2020

ghost commented Jan 2, 2021

ghost commented Feb 2, 2021

ghost commented Mar 4, 2021

jeremyje commented Mar 4, 2021

weijuans-msft commented Mar 31, 2021 • edited Loading

weijuans-msft commented Apr 16, 2021

weijuans-msft commented Apr 27, 2021

johnSchnake commented May 14, 2021

ghost commented Jun 13, 2021

ghost commented Jul 14, 2021

weijuans-msft commented Jul 14, 2021

ghost commented Aug 14, 2021

ghost commented Sep 14, 2021

ghost commented May 6, 2022

mloskot commented May 26, 2022 • edited Loading

m-deepakraja commented Jun 24, 2022

weijuans-msft commented Jul 20, 2022

mloskot commented Jul 20, 2022

TBBle commented Jul 20, 2022

weijuans-msft commented Jul 20, 2022

jterry75 commented Jul 22, 2022

ghost commented Aug 22, 2022

ghost commented Sep 21, 2022

mloskot commented Sep 21, 2022

ghost commented Oct 23, 2022

ghost commented Nov 22, 2022

mloskot commented Nov 22, 2022

ghost commented Dec 23, 2022

mloskot commented Dec 23, 2022

microsoft-github-policy-service bot commented Apr 21, 2023

microsoft-github-policy-service bot commented Apr 21, 2023

mloskot commented Apr 21, 2023

akarshm commented May 30, 2023

TBBle commented May 31, 2023

akarshm commented May 31, 2023

jeremyje commented Nov 3, 2020 •

edited

Loading

weijuans-msft commented Mar 31, 2021 •

edited

Loading

mloskot commented May 26, 2022 •

edited

Loading