Skip to content

Commit

Permalink
Appveyor: build with VS 2015 and msvcr 140, use vcpkg for deps
Browse files Browse the repository at this point in the history
  • Loading branch information
@edenhill
edenhill committed Aug 12, 2021
1 parent 38edf86 commit f57074c
Show file tree
Hide file tree
Showing 9 changed files with 125 additions and 105 deletions.
38 changes: 24 additions & 14 deletions .appveyor.yml
View file
Original file line number Diff line number Diff line change
@@ -1,24 +1,33 @@
version: 1.0.0-R-post{build}
version: 1.8.0-R-post{build}
pull_requests:
do_not_increment_build_number: true
image: Visual Studio 2013
image: Visual Studio 2015
configuration: Release
environment:
runtime: v140
matrix:
- platform: x64
arch: x64
- platform: win32
arch: x86
install:
- ps: "& .\\win32\\install-openssl.ps1"
- ps: "& .\\win32\\install-coapp.ps1"
# Update vcpkg (is outdated on the VS 2015 image)
- cmd: |
cd "C:\Tools\vcpkg"
git pull
.\bootstrap-vcpkg.bat
cd %appveyor_build_folder%
cache:
- c:\OpenSSL-Win32 -> win32\install-openssl.ps1
- c:\OpenSSL-Win64 -> win32\install-openssl.ps1
- c:\tools\vcpkg\installed
- C:\Users\appveyor\AppData\Local\vcpkg\archives
- C:\Users\appveyor\AppData\Local\vcpkg\installed
nuget:
account_feed: true
project_feed: true
disable_publish_on_pr: true
before_build:
- cmd: nuget restore win32/librdkafka.sln
- cmd: vcpkg install zstd zlib openssl --triplet %arch%-windows
build:
project: win32/librdkafka.sln
publish_nuget: true
Expand All @@ -27,7 +36,7 @@ build:
parallel: true
verbosity: normal
test_script:
- cmd: cd tests && ..\win32\outdir\v120\%PLATFORM%\%CONFIGURATION%\tests.exe -l -Q -p1 && cd ..
- cmd: cd tests && ..\win32\outdir\%runtime%\%PLATFORM%\%CONFIGURATION%\tests.exe -l -Q -p1 && cd ..
artifacts:
- path: test_report*.json
name: Test report
Expand All @@ -41,7 +50,8 @@ artifacts:
name: Libraries
- path: '**\*.exe'
name: Executables
before_deploy:
#before_deploy:
after_test:
- ps: >-
# FIXME: Add to Deployment condition above:
Expand All @@ -53,13 +63,13 @@ before_deploy:
$autopkgFile = "win32/librdkafka.autopkg"
pwd
pwd
ls $autopkgFile
ls $autopkgFile
# Get the ".autopkg.template" file, replace "@version" with the Appveyor version number, then save to the ".autopkg" file.
cat ($autopkgFile + ".template") | % { $_ -replace "@version", $env:appveyor_build_version } > $autopkgFile
Expand All @@ -76,9 +86,9 @@ before_deploy:
deploy:
- provider: S3
access_key_id:
secure: iBK0xb23FMYOrOsOb8cw3YGyU+6vvPX5BF+PXuMub8M=
secure: iBK0xb23FMYOrOsOb8cw3YGyU+6vvPX5BF+PXuMub8M=
secret_access_key:
secure: jJsj373UiOtuXf/u0LLL0Q8XQMyu4s/ucx0+vH4GpKbAfZJUwYB4dEO1//mQDNuC
secure: jJsj373UiOtuXf/u0LLL0Q8XQMyu4s/ucx0+vH4GpKbAfZJUwYB4dEO1//mQDNuC
region: us-west-1
bucket: librdkafka-ci-packages
folder: librdkafka/p-librdkafka__bld-appveyor__plat-windows__arch-$(platform)__bldtype-$(configuration)__tag-$(APPVEYOR_REPO_TAG_NAME)__sha-$(APPVEYOR_REPO_COMMIT)__bid-$(APPVEYOR_BUILD_ID)
Expand Down
13 changes: 13 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,19 @@ librdkafka v1.7.0 is feature release:
* The binary librdkafka artifacts for Alpine are now using Alpine 3.12.
OpenSSL 1.1.1k.
* Improved static librdkafka Windows builds using MinGW (@neptoess, #3130).
* The `librdkafka.redist` NuGet package now has updated zlib, zstd and
OpenSSL versions (from vcpkg).


## Security considerations

* The zlib version bundled with the `librdkafka.redist` NuGet package has now been upgraded
from zlib 1.2.8 to 1.2.11, fixing the following CVEs:
* CVE-2016-9840: undefined behaviour (compiler dependent) in inflate (decompression) code: this is used by the librdkafka consumer. Risk of successfully exploitation through consumed messages is eastimated very low.
* CVE-2016-9841: undefined behaviour (compiler dependent) in inflate code: this is used by the librdkafka consumer. Risk of successfully exploitation through consumed messages is eastimated very low.
* CVE-2016-9842: undefined behaviour in inflateMark(): this API is not used by librdkafka.
* CVE-2016-9843: issue in crc32_big() which is called from crc32_z(): this API is not used by librdkafka.

## Upgrade considerations

* The C++ `oauthbearer_token_refresh_cb()` was missing a `Handle *`
Expand All @@ -88,6 +99,8 @@ librdkafka v1.7.0 is feature release:
This is more correct than the previous `consumer_lag` which was using
either `committed_offset` or `app_offset` (last message passed
to application).
* The `librdkafka.redist` NuGet package is now built with MSVC runtime v140
(VS 2015). Previous versions were built with MSVC runtime v120 (VS 2013).


## Fixes
Expand Down
125 changes: 68 additions & 57 deletions packaging/nuget/packaging.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -327,19 +327,19 @@ def build (self, buildtype):
a.info.get('arch'),
a.info.get('bldtype'))
if 'toolset' not in a.info:
a.info['toolset'] = 'v120'
a.info['toolset'] = 'v140'

mappings = [
[{'arch': 'x64', 'plat': 'linux', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './include/librdkafka/rdkafka.h', 'build/native/include/librdkafka/rdkafka.h'],
[{'arch': 'x64', 'plat': 'linux', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './include/librdkafka/rdkafkacpp.h', 'build/native/include/librdkafka/rdkafkacpp.h'],
[{'arch': 'x64', 'plat': 'linux', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './include/librdkafka/rdkafka_mock.h', 'build/native/include/librdkafka/rdkafka_mock.h'],
[{'arch': 'x64', 'plat': 'linux', 'lnk': 'std', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './include/librdkafka/rdkafka.h', 'build/native/include/librdkafka/rdkafka.h'],
[{'arch': 'x64', 'plat': 'linux', 'lnk': 'std', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './include/librdkafka/rdkafkacpp.h', 'build/native/include/librdkafka/rdkafkacpp.h'],
[{'arch': 'x64', 'plat': 'linux', 'lnk': 'std', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './include/librdkafka/rdkafka_mock.h', 'build/native/include/librdkafka/rdkafka_mock.h'],

# Travis OSX build
[{'arch': 'x64', 'plat': 'osx', 'fname_glob': 'librdkafka-clang.tar.gz'}, './lib/librdkafka.dylib', 'runtimes/osx-x64/native/librdkafka.dylib'],
# Travis Manylinux build
[{'arch': 'x64', 'plat': 'linux', 'fname_glob': 'librdkafka-manylinux*x86_64.tgz'}, './lib/librdkafka.so.1', 'runtimes/linux-x64/native/centos6-librdkafka.so'],
# Travis Ubuntu 14.04 build
[{'arch': 'x64', 'plat': 'linux', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './lib/librdkafka.so.1', 'runtimes/linux-x64/native/librdkafka.so'],
[{'arch': 'x64', 'plat': 'linux', 'lnk': 'std', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './lib/librdkafka.so.1', 'runtimes/linux-x64/native/librdkafka.so'],
# Travis CentOS 7 RPM build
[{'arch': 'x64', 'plat': 'linux', 'fname_glob': 'librdkafka1*el7.x86_64.rpm'}, './usr/lib64/librdkafka.so.1', 'runtimes/linux-x64/native/centos7-librdkafka.so'],
# Travis Alpine build
Expand All @@ -348,32 +348,37 @@ def build (self, buildtype):
[{'arch': 'arm64', 'plat': 'linux', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './lib/librdkafka.so.1', 'runtimes/linux-arm64/native/librdkafka.so'],

# Common Win runtime
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'msvcr120.zip'}, 'msvcr120.dll', 'runtimes/win-x64/native/msvcr120.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'msvcr120.zip'}, 'msvcp120.dll', 'runtimes/win-x64/native/msvcp120.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'msvcr140.zip'}, 'vcruntime140.dll', 'runtimes/win-x64/native/vcruntime140.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'msvcr140.zip'}, 'msvcp140.dll', 'runtimes/win-x64/native/msvcp140.dll'],
# matches librdkafka.redist.{VER}.nupkg
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/x64/Release/librdkafka.dll', 'runtimes/win-x64/native/librdkafka.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/x64/Release/librdkafkacpp.dll', 'runtimes/win-x64/native/librdkafkacpp.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/x64/Release/zlib.dll', 'runtimes/win-x64/native/zlib.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/x64/Release/libzstd.dll', 'runtimes/win-x64/native/libzstd.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/x64/Release/librdkafka.dll', 'runtimes/win-x64/native/librdkafka.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/x64/Release/librdkafkacpp.dll', 'runtimes/win-x64/native/librdkafkacpp.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/x64/Release/libcrypto-1_1-x64.dll', 'runtimes/win-x64/native/libcrypto-1_1-x64.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/x64/Release/libssl-1_1-x64.dll', 'runtimes/win-x64/native/libssl-1_1-x64.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/x64/Release/zlib1.dll', 'runtimes/win-x64/native/zlib1.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/x64/Release/zstd.dll', 'runtimes/win-x64/native/zstd.dll'],
# matches librdkafka.{VER}.nupkg
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka*', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v120/x64/Release/librdkafka.lib', 'build/native/lib/win/x64/win-x64-Release/v120/librdkafka.lib'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka*', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v120/x64/Release/librdkafkacpp.lib', 'build/native/lib/win/x64/win-x64-Release/v120/librdkafkacpp.lib'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka*.nupkg', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v140/x64/Release/librdkafka.lib', 'build/native/lib/win/x64/win-x64-Release/v140/librdkafka.lib'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka*.nupkg', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v140/x64/Release/librdkafkacpp.lib', 'build/native/lib/win/x64/win-x64-Release/v140/librdkafkacpp.lib'],

[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'msvcr120.zip'}, 'msvcr120.dll', 'runtimes/win-x86/native/msvcr120.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'msvcr120.zip'}, 'msvcp120.dll', 'runtimes/win-x86/native/msvcp120.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'msvcr140.zip'}, 'vcruntime140.dll', 'runtimes/win-x86/native/vcruntime140.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'msvcr140.zip'}, 'msvcp140.dll', 'runtimes/win-x86/native/msvcp140.dll'],
# matches librdkafka.redist.{VER}.nupkg
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/Win32/Release/librdkafka.dll', 'runtimes/win-x86/native/librdkafka.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/Win32/Release/librdkafkacpp.dll', 'runtimes/win-x86/native/librdkafkacpp.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/Win32/Release/zlib.dll', 'runtimes/win-x86/native/zlib.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/Win32/Release/libzstd.dll', 'runtimes/win-x86/native/libzstd.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/Win32/Release/librdkafka.dll', 'runtimes/win-x86/native/librdkafka.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/Win32/Release/librdkafkacpp.dll', 'runtimes/win-x86/native/librdkafkacpp.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/Win32/Release/libcrypto-1_1.dll', 'runtimes/win-x86/native/libcrypto-1_1.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/Win32/Release/libssl-1_1.dll', 'runtimes/win-x86/native/libssl-1_1.dll'],

[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/Win32/Release/zlib1.dll', 'runtimes/win-x86/native/zlib1.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/Win32/Release/zstd.dll', 'runtimes/win-x86/native/zstd.dll'],

# matches librdkafka.{VER}.nupkg
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka*', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v120/Win32/Release/librdkafka.lib', 'build/native/lib/win/x86/win-x86-Release/v120/librdkafka.lib'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka*', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v120/Win32/Release/librdkafkacpp.lib', 'build/native/lib/win/x86/win-x86-Release/v120/librdkafkacpp.lib']
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka*.nupkg', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v140/Win32/Release/librdkafka.lib', 'build/native/lib/win/x86/win-x86-Release/v140/librdkafka.lib'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka*.nupkg', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v140/Win32/Release/librdkafkacpp.lib', 'build/native/lib/win/x86/win-x86-Release/v140/librdkafkacpp.lib']
]

for m in mappings:
Expand All @@ -385,36 +390,36 @@ def build (self, buildtype):
fname_excludes = attributes['fname_excludes']
del attributes['fname_excludes']

artifact = None
for a in self.arts.artifacts:
found = True
outf = os.path.join(self.stpath, m[2])
member = m[1]

found = False
# Try all matching artifacts until we find the wanted file (member)
for a in self.arts.artifacts:
for attr in attributes:
if a.info[attr] != attributes[attr]:
found = False
break
if a.info.get(attr, None) != attributes[attr]:
continue

if not fnmatch(a.fname, fname_glob):
found = False
continue

for exclude in fname_excludes:
if exclude in a.fname:
found = False
break
continue

if found:
artifact = a
break
try:
zfile.ZFile.extract(a.lpath, member, outf)
except KeyError as e:
continue
except Exception as e:
raise Exception('file not found in archive %s: %s. Files in archive are: %s' % (a.lpath, e, zfile.ZFile(a.lpath).getnames()))

if artifact is None:
raise MissingArtifactError('unable to find artifact with tags %s matching "%s"' % (str(attributes), fname_glob))
found = True
break

if not found:
raise MissingArtifactError('unable to find artifact with tags %s matching "%s" for file "%s"' % (str(attributes), fname_glob, member))

outf = os.path.join(self.stpath, m[2])
member = m[1]
try:
zfile.ZFile.extract(artifact.lpath, member, outf)
except KeyError as e:
raise Exception('file not found in archive %s: %s. Files in archive are: %s' % (artifact.lpath, e, zfile.ZFile(artifact.lpath).getnames()))

print('Tree extracted to %s' % self.stpath)

Expand All @@ -437,28 +442,34 @@ def verify (self, path):
"build/native/include/librdkafka/rdkafka.h",
"build/native/include/librdkafka/rdkafkacpp.h",
"build/native/include/librdkafka/rdkafka_mock.h",
"build/native/lib/win/x64/win-x64-Release/v120/librdkafka.lib",
"build/native/lib/win/x64/win-x64-Release/v120/librdkafkacpp.lib",
"build/native/lib/win/x86/win-x86-Release/v120/librdkafka.lib",
"build/native/lib/win/x86/win-x86-Release/v120/librdkafkacpp.lib",
"build/native/lib/win/x64/win-x64-Release/v140/librdkafka.lib",
"build/native/lib/win/x64/win-x64-Release/v140/librdkafkacpp.lib",
"build/native/lib/win/x86/win-x86-Release/v140/librdkafka.lib",
"build/native/lib/win/x86/win-x86-Release/v140/librdkafkacpp.lib",
"runtimes/linux-x64/native/centos7-librdkafka.so",
"runtimes/linux-x64/native/centos6-librdkafka.so",
"runtimes/linux-x64/native/alpine-librdkafka.so",
"runtimes/linux-x64/native/librdkafka.so",
"runtimes/linux-arm64/native/librdkafka.so",
"runtimes/osx-x64/native/librdkafka.dylib",
# win x64
"runtimes/win-x64/native/librdkafka.dll",
"runtimes/win-x64/native/librdkafkacpp.dll",
"runtimes/win-x64/native/msvcr120.dll",
"runtimes/win-x64/native/msvcp120.dll",
"runtimes/win-x64/native/zlib.dll",
"runtimes/win-x64/native/libzstd.dll",
"runtimes/win-x64/native/vcruntime140.dll",
"runtimes/win-x64/native/msvcp140.dll",
"runtimes/win-x64/native/libcrypto-1_1-x64.dll",
"runtimes/win-x64/native/libssl-1_1-x64.dll",
"runtimes/win-x64/native/zlib1.dll",
"runtimes/win-x64/native/zstd.dll",
# win x86
"runtimes/win-x86/native/librdkafka.dll",
"runtimes/win-x86/native/librdkafkacpp.dll",
"runtimes/win-x86/native/msvcr120.dll",
"runtimes/win-x86/native/msvcp120.dll",
"runtimes/win-x86/native/zlib.dll",
"runtimes/win-x86/native/libzstd.dll"]
"runtimes/win-x86/native/vcruntime140.dll",
"runtimes/win-x86/native/msvcp140.dll",
"runtimes/win-x86/native/libcrypto-1_1.dll",
"runtimes/win-x86/native/libssl-1_1.dll",
"runtimes/win-x86/native/zlib1.dll",
"runtimes/win-x86/native/zstd.dll"]

missing = list()
with zfile.ZFile(path, 'r') as zf:
Expand Down
Loading

0 comments on commit f57074c

Please sign in to comment.