Add Windows MinGW Static Library #564

neptoess · 2020-11-05T15:50:16Z

Re-opening #555
Resolves #128

ghost · 2020-11-05T15:50:18Z

@confluentinc It looks like @neptoess just signed our Contributor License Agreement. 👍

Always at your service,

clabot

neptoess · 2020-11-05T15:51:10Z

@edenhill ,
I will add the Travis CI worker in this PR

neptoess · 2020-11-05T22:55:02Z

@edenhill
Seeing this in the Travis log

=== RUN   TestProducerOAuthBearerConfig
    TestProducerOAuthBearerConfig: producer_test.go:283: NewProducer failed: No provider for SASL mechanism OAUTHBEARER: recompile librdkafka with openssl support. Current build options: PLAIN WindowsSSPI(GSSAPI)
--- FAIL: TestProducerOAuthBearerConfig (0.00s)

Since it seems intentional that PLAIN and GSSAPI are the only SASL options on Windows, should this test be skipped on Windows builds?

neptoess · 2020-11-06T16:14:37Z

build_windows.go and librdkafka_windows.a came from a local librdkafka v1.5.2 build, but once confluentinc/librdkafka#3130 is merged, Travis will automatically build the static Windows bundle.

Restore OATHBEARER testing on Windows

…GINS and WITH_ZLIB

…fka-go into import_v1.5.2

neptoess · 2021-04-07T13:14:39Z

@edenhill
Now that Travis is building static librdkafka bundles for Windows, what else do we need to get this PR merged?

edenhill

Don't do the actual import here, we'll do that for the upcoming 1.7.0 release, just update the tooling as needed. Thanks

.travis.yml

kafka/librdkafka_vendor/bundle-import.sh

… allow them to persist once the script returns

neptoess · 2021-04-07T17:03:42Z

@edenhill

Don't do the actual import here, we'll do that for the upcoming 1.7.0 release, just update the tooling as needed. Thanks

This is done, but note that this will make the Travis Windows jobs fail (since build_windows.go and librdkafka_windows.a are missing)

neptoess · 2021-04-19T15:36:51Z

@edenhill
I think this should be ready to merge. Need anything else from me?

edenhill · 2021-04-20T10:10:24Z

Thanks alot for this!

edenhill · 2021-04-29T13:56:13Z

@neptoess Hey, I just pushed an RC branch with your new Windows builds, can you try it out on Windows? (this RC branch will not be merged, to keep the repo size down).
https://github.com/confluentinc/confluent-kafka-go/tree/import_v1.7.0-RC6

neptoess · 2021-04-29T19:26:06Z

@neptoess Hey, I just pushed an RC branch with your new Windows builds, can you try it out on Windows? (this RC branch will not be merged, to keep the repo size down).
https://github.com/confluentinc/confluent-kafka-go/tree/import_v1.7.0-RC6

@edenhill
Tested locally (both producer and consumer) and it looks to be working well. I see the Travis build (https://travis-ci.org/github/confluentinc/confluent-kafka-go/builds/768834426) worked as well. Good news all around!

edenhill · 2021-04-30T06:28:39Z

Superb! Thank you for all your hard work on getting this done!

leonheess · 2021-05-28T14:20:50Z

This PR sounds like Windows is now supported but the README still says "confluent-kafka-go is not supported on Windows." - can you clarify if and how I can get it to run on Windows?

neptoess · 2021-05-28T14:24:41Z

This PR sounds like Windows is now supported but the README still says "confluent-kafka-go is not supported on Windows." - can you clarify if and how I can get it to run on Windows?

It’s not officially supported, but, as of v1.7.0, a static bundle is being shipped. If you have a working cgo toolchain (the bundle is built with MSYS2 MinGW 64-bit, and is what I’d recommend), it should “just work”

leonheess · 2021-05-28T14:48:37Z

It’s not officially supported, but, as of v1.7.0, a static bundle is being shipped. If you have a working cgo toolchain (the bundle is built with MSYS2 MinGW 64-bit, and is what I’d recommend), it should “just work”

Thanks for the quick reply. I'm getting this error:

Can you point me in any direction?

neptoess · 2021-05-28T15:37:37Z

It’s not officially supported, but, as of v1.7.0, a static bundle is being shipped. If you have a working cgo toolchain (the bundle is built with MSYS2 MinGW 64-bit, and is what I’d recommend), it should “just work”

Thanks for the quick reply. I'm getting this error:

Can you point me in any direction?

Very bizarre. The Travis build and tests worked perfectly fine with this bundle https://travis-ci.org/github/confluentinc/confluent-kafka-go/jobs/770559600

__imp_wassert is defined in the C runtime library. What version of Windows are you running on? What gcc version?

leonheess · 2021-05-28T15:44:00Z

Very bizarre. The Travis build and tests worked perfectly fine with this bundle https://travis-ci.org/github/confluentinc/confluent-kafka-go/jobs/770559600

I saw that that is why I'm kind of confused

__imp_wassert is defined in the C runtime library. What version of Windows are you running on? What gcc version?

neptoess · 2021-05-28T15:50:07Z

Very bizarre. The Travis build and tests worked perfectly fine with this bundle https://travis-ci.org/github/confluentinc/confluent-kafka-go/jobs/770559600

I saw that that is why I'm kind of confused

__imp_wassert is defined in the C runtime library. What version of Windows are you running on? What gcc version?

Hmm. Gcc v8.1.0 is a little outdated. Per https://travis-ci.org/github/edenhill/librdkafka/jobs/770444016#L2714, the v1.7.0 bundle was built with v10.3. Maybe try upgrading?

leonheess · 2021-05-28T16:23:05Z

Hmm. Gcc v8.1.0 is a little outdated. Per https://travis-ci.org/github/edenhill/librdkafka/jobs/770444016#L2714, the v1.7.0 bundle was built with v10.3. Maybe try upgrading?

I upgraded to v11.1 and it's working! ty so much ❤

Co-authored-by: neptoess <[email protected]>

dassio · 2022-06-25T02:15:39Z

for anyone that trying to build without too much background in C++, here is what i did :
install mingw-w64: pacman -S --needed mingw-w64-x86_64-toolchain
compile with command: env GOOS=windows GOARCH=amd64 CGO_ENABLED=1 CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ CGO_CXXFLAGS="-IC:\msys64\mingw64\include" go build

neptoess · 2022-06-25T02:24:51Z

for anyone that trying to build without too much background in C++, here is what i did :
install mingw-w64: pacman -S --needed mingw-w64-x86_64-toolchain
compile with command: env GOOS=windows GOARCH=amd64 CGO_ENABLED=1 CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ CGO_CXXFLAGS="-IC:\msys64\mingw64\include" go build

If you have C:\msys64\mingw64\bin in your PATH, you shouldn’t need to define all these environment variables. I just do plain old go build

behoyh · 2023-04-11T20:58:56Z

I'm getting error

# command-line-arguments
C:\Program Files\Go\pkg\tool\windows_amd64\link.exe: running gcc failed: exit status 1
C:/Users/bhanna/scoop/apps/mingw/current/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:\Users\bhanna\AppData\Local\Temp\go-link-3107212506\000026.o: in function `_cgo_preinit_init':
\\_\_\runtime\cgo/gcc_libinit_windows.c:40: undefined reference to `__imp___iob_func'
C:/Users/bhanna/scoop/apps/mingw/current/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:\Users\bhanna\AppData\Local\Temp\go-link-3107212506\000026.o: in function `x_cgo_notify_runtime_init_done':
\\_\_\runtime\cgo/gcc_libinit_windows.c:105: undefined reference to `__imp___iob_func'
C:/Users/bhanna/scoop/apps/mingw/current/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:\Users\bhanna\AppData\Local\Temp\go-link-3107212506\000026.o: in function `_cgo_beginthread':
\\_\_\runtime\cgo/gcc_libinit_windows.c:149: undefined reference to `__imp___iob_func'
C:/Users/bhanna/scoop/apps/mingw/current/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:\Users\bhanna\AppData\Local\Temp\go-link-3107212506\000027.o: in function `x_cgo_thread_start':
\\_\_\runtime\cgo/gcc_util.c:18: undefined reference to `__imp___iob_func'
C:/Users/bhanna/scoop/apps/mingw/current/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Development/tracer/vendor/github.com/confluentinc/confluent-kafka-go/v2/kafka/librdkafka_vendor/librdkafka_windows.a(rdkafka_error.c.obj):(.text+0x8e): undefined reference to `__imp__vsnprintf_s'
C:/Users/bhanna/scoop/apps/mingw/current/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Development/tracer/vendor/github.com/confluentinc/confluent-kafka-go/v2/kafka/librdkafka_vendor/librdkafka_windows.a(rdkafka_mock.c.obj):(.text+0x8e): undefined reference to `__imp__vsnprintf_s'
C:/Users/bhanna/scoop/apps/mingw/current/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Development/tracer/vendor/github.com/confluentinc/confluent-kafka-go/v2/kafka/librdkafka_vendor/librdkafka_windows.a(rdkafka_txnmgr.c.obj):(.text+0xca): undefined reference to `__imp__vsnprintf_s'
C:/Users/bhanna/scoop/apps/mingw/current/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Development/tracer/vendor/github.com/confluentinc/confluent-kafka-go/v2/kafka/librdkafka_vendor/librdkafka_windows.a(rdkafka_idempotence.c.obj):(.text+0x8e): undefined reference to `__imp__vsnprintf_s'
C:/Users/bhanna/scoop/apps/mingw/current/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Development/tracer/vendor/github.com/confluentinc/confluent-kafka-go/v2/kafka/librdkafka_vendor/librdkafka_windows.a(rdkafka_aux.c.obj):(.text+0x8e): undefined reference to `__imp__vsnprintf_s'
C:/Users/bhanna/scoop/apps/mingw/current/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Development/tracer/vendor/github.com/confluentinc/confluent-kafka-go/v2/kafka/librdkafka_vendor/librdkafka_windows.a(rdkafka_admin.c.obj):(.text+0x8e): more undefined references to `__imp__vsnprintf_s' follow
C:/Users/bhanna/scoop/apps/mingw/current/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Development/tracer/vendor/github.com/confluentinc/confluent-kafka-go/v2/kafka/librdkafka_vendor/librdkafka_windows.a(regexp.c.obj):(.text+0x29fa): undefined reference to `_setjmp'
collect2.exe: error: ld returned 1 exit status

when trying to install with mingw v12.2.0-rt_v10-rev2 installed via Scoop: scoop install mingw

is this supported or is there another way to install?

neptoess · 2023-04-11T22:35:03Z

Not sure about Scoop. The MinGW install I was using was via MSYS2

behoyh · 2023-04-11T23:18:56Z

Thanks for the response, I believe the versions should be the same according to gcc --version:

gcc.exe (x86_64-posix-seh-rev2, Built by MinGW-W64 project) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

behoyh · 2023-04-12T22:54:43Z

So as you say, what got this to work was installing MSYS2 version of gcc, via pacman:

update pacman

pacman -Syu

uninstall 32 target gcc if installed

pacman -R gcc

install gcc on x64 targeting x64

pacman -S mingw-w64-x86_64-gcc

Add to PATH:

C:\msys64\mingw64\bin
C:\msys64\usr\bin

Update bundle-import.sh to handle Windows MinGW static lib

b72e241

neptoess marked this pull request as draft November 5, 2020 15:51

neptoess added 4 commits November 5, 2020 11:07

Add Windows static bundle and build_windows.go

5218d2d

Add Windows worker

d7d8efb

Replace static bundle with one built from edenhill/librdkafka:v1.5.2

c658936

Use MinGW GCC from MSYS2

6bd2666

neptoess changed the base branch from import_v1.5.2 to master November 5, 2020 20:18

Set go_import_path

9389d46

Disable OAuthBearer tests on Windows

1b5d134

neptoess marked this pull request as ready for review November 6, 2020 15:09

neptoess added 5 commits November 10, 2020 15:02

Build librdkafka_windows.a with OATHBEARER support

816f9ac

Restore OATHBEARER testing on Windows

Add libdl and libz to bundled Windows static lib, to support WITH_PLU…

4688340

…GINS and WITH_ZLIB

Remove libdl from static Windows bundle

8e5b5c2

Handle Libs.private in bundle-import.sh

3a7191e

Merge branch 'master' of https://github.com/confluentinc/confluent-ka…

a04306f

…fka-go into import_v1.5.2

neptoess mentioned this pull request Apr 5, 2021

How to use Golang client on Windows #128

Closed

neptoess added 3 commits April 7, 2021 07:47

Merge branch 'master' of https://github.com/confluentinc/confluent-ka…

8e616bd

…fka-go into import_v1.5.2

Update bundled librdkafka_windows.a to v1.6.1

e55431f

Add Go 1.16 Travis worker

13f482f

edenhill suggested changes Apr 7, 2021

View reviewed changes

.travis.yml Show resolved Hide resolved

.travis.yml Outdated Show resolved Hide resolved

.travis.yml Show resolved Hide resolved

kafka/librdkafka_vendor/bundle-import.sh Outdated Show resolved Hide resolved

neptoess added 3 commits April 7, 2021 12:40

Addressing PR comments

be842e4

Remove static bundle (so this PR can just be updated tooling)

e71f0b9

Add exports back to .travis.yml, since running them in a script won't…

2f4ebcf

… allow them to persist once the script returns

Remove Libs.private processing in bundle-import.sh

ff1f153

neptoess mentioned this pull request Apr 16, 2021

CMake: Remove Libs.private entries from rdkafka-static.pc and rdkafka++-static.pc confluentinc/librdkafka#3354

Merged

edenhill approved these changes Apr 20, 2021

View reviewed changes

edenhill merged commit 80c58f8 into confluentinc:master Apr 20, 2021

eran-levy pushed a commit to eran-levy/confluent-kafka-go that referenced this pull request Apr 9, 2022

Add Windows MinGW Static Library (@neptoess, confluentinc#564)

c098e1b

Co-authored-by: neptoess <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Windows MinGW Static Library #564

Add Windows MinGW Static Library #564

neptoess commented Nov 5, 2020 •

edited

Loading

ghost commented Nov 5, 2020

neptoess commented Nov 5, 2020

neptoess commented Nov 5, 2020

neptoess commented Nov 6, 2020

neptoess commented Apr 7, 2021

edenhill left a comment

neptoess commented Apr 7, 2021 •

edited

Loading

neptoess commented Apr 19, 2021

edenhill commented Apr 20, 2021

edenhill commented Apr 29, 2021

neptoess commented Apr 29, 2021

edenhill commented Apr 30, 2021

leonheess commented May 28, 2021

neptoess commented May 28, 2021

leonheess commented May 28, 2021

neptoess commented May 28, 2021

leonheess commented May 28, 2021 •

edited

Loading

neptoess commented May 28, 2021

leonheess commented May 28, 2021 •

edited

Loading

dassio commented Jun 25, 2022

neptoess commented Jun 25, 2022

behoyh commented Apr 11, 2023 •

edited

Loading

neptoess commented Apr 11, 2023

behoyh commented Apr 11, 2023

behoyh commented Apr 12, 2023 •

edited

Loading

Add Windows MinGW Static Library #564

Add Windows MinGW Static Library #564

Conversation

neptoess commented Nov 5, 2020 • edited Loading

ghost commented Nov 5, 2020

neptoess commented Nov 5, 2020

neptoess commented Nov 5, 2020

neptoess commented Nov 6, 2020

neptoess commented Apr 7, 2021

edenhill left a comment

Choose a reason for hiding this comment

neptoess commented Apr 7, 2021 • edited Loading

neptoess commented Apr 19, 2021

edenhill commented Apr 20, 2021

edenhill commented Apr 29, 2021

neptoess commented Apr 29, 2021

edenhill commented Apr 30, 2021

leonheess commented May 28, 2021

neptoess commented May 28, 2021

leonheess commented May 28, 2021

neptoess commented May 28, 2021

leonheess commented May 28, 2021 • edited Loading

neptoess commented May 28, 2021

leonheess commented May 28, 2021 • edited Loading

dassio commented Jun 25, 2022

neptoess commented Jun 25, 2022

behoyh commented Apr 11, 2023 • edited Loading

neptoess commented Apr 11, 2023

behoyh commented Apr 11, 2023

behoyh commented Apr 12, 2023 • edited Loading

update pacman

uninstall 32 target gcc if installed

install gcc on x64 targeting x64

Add to PATH:

neptoess commented Nov 5, 2020 •

edited

Loading

neptoess commented Apr 7, 2021 •

edited

Loading

leonheess commented May 28, 2021 •

edited

Loading

leonheess commented May 28, 2021 •

edited

Loading

behoyh commented Apr 11, 2023 •

edited

Loading

behoyh commented Apr 12, 2023 •

edited

Loading