Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically use CMake for Windows builds #672

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,9 @@ jobs:
include:
- os: macos-11.0
- os: windows-2019
features: cmake-build,libz-static,curl-static
rdkafka-sys-features: cmake-build,libz-static,curl-static
# The Windows build should automatically use CMake
features: libz-static,curl-static
rdkafka-sys-features: libz-static,curl-static
- os: ubuntu-20.04
features: tracing
- os: ubuntu-20.04
Expand Down
579 changes: 290 additions & 289 deletions README.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ See also the [rdkafka-sys changelog](rdkafka-sys/changelog.md).
* Update MSRV to 1.70
* Remove testign for old Kafka versions (before 3.0). Add tests for 3.7.
* Fix test dependency on docker compose.
* Automatically use CMake for Windows targets

## 0.36.2 (2024-01-16)

Expand Down
6 changes: 6 additions & 0 deletions rdkafka-sys/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,12 @@ sasl2-sys = { version = "0.1.6", optional = true }
pkg-config = "0.3.9"
cmake = { version = "0.1.0", optional = true }

# Windows only supports the CMake build. As we can't conditionally
# make cmake-build a default feature only on some platforms, we instead
# have to make the cmake dependency required.
[target.'cfg(target_os = "windows")'.build-dependencies]
cmake = { version = "0.1.0", optional = false }

[lib]
name = "rdkafka_sys"
path = "src/lib.rs"
Expand Down
189 changes: 95 additions & 94 deletions rdkafka-sys/README.md
Original file line number Diff line number Diff line change
@@ -1,94 +1,95 @@
# rdkafka-sys

Low level bindings to [librdkafka](https://github.com/edenhill/librdkafka),
a C library for the [Apache Kafka] protocol with producer, consumer, and
admin clients.

For a safe wrapper, see the [rdkafka] crate.

## Version

The rdkafka-sys version number is in the format `X.Y.Z+RX.RY.RZ`, where
`X.Y.Z` is the version of this crate and follows SemVer conventions, while
`RX.RY.RZ` is the version of the bundled librdkafka.

Note that versions before v2.0.0+1.4.2 did not follow this convention, and
instead directly correspond to the bundled librdkafka version.

## Build

### Known issues

* When any of librdkafka's optional dependencies are enabled, like libz or
OpenSSL, if you have multiple versions of that library installed upon your
system, librdkafka's build system may disagree with Cargo about which
version of the library to use! **This can result in subtly broken
builds,** if librdkafka compiles against the headers for one version but
Cargo links against a different version. For complete confidence when
building release binaries, use an environment like a Docker container or a
chroot jail where you can guarantee that only one version of each
dependency is present. The current design of Cargo unfortunately makes
this nearly impossible to fix.

* Windows is only supported when using the CMake build system via the
`cmake-build` Cargo feature.

### Features

By default a submodule with the librdkafka sources will be used to compile
and statically link the library.

The **`dynamic-linking`** feature can be used to link rdkafka to a locally
installed version of librdkafka: if the feature is enabled, the build script
will use `pkg-config` to check the version of the library installed in the
system, and it will configure the compiler to dynamically link against it.
The system version of librdkafka must exactly match the version of
librdkafka bundled with this crate.

The **`cmake-build`** feature builds librdkafka with its [CMake] build
system, rather than its default [mklove]-based build system. This feature
requires that CMake is installed on the build machine.

The following features directly correspond to librdkafka features (i.e.,
flags you would pass to `configure` if you were compiling manually).

* The **`ssl`** feature enables SSL support. By default, the system's
OpenSSL library is dynamically linked, but static linking of the version
bundled with the [openssl-sys] crate can be requested with the
`ssl-vendored` feature.
* The **`gssapi`** feature enables SASL GSSAPI support with Cyrus
libsasl2. By default the system's libsasl2 is dynamically linked, but
static linking of the version bundled with the [sasl2-sys] crate can be
requested with the `gssapi-vendored` feature.
* The **`libz`** feature enables support for zlib compression. This
feature is enabled by default. By default, the system's libz is
dynamically linked, but static linking of the version bundled with the
[libz-sys] crate can be requested with the `libz-static` feature.
* The **`curl`** feature enables the HTTP client via curl. By default, the
system's curl is dynamically linked, but static linking of the version
bundled with the [curl-sys] create can be requested with the
`curl-static` feature.
* The **`zstd`** feature enables support for ZSTD compression. By default,
this builds and statically links the version bundled with the [zstd-sys]
crate, but dynamic linking of the system's version can be requested with
the `zstd-pkg-config` feature.
* The **`external-lz4`** feature statically links against the copy of
liblz4 bundled with the [lz4-sys] crate. By default, librdkafka
statically links against its own bundled version of liblz4. Due to
limitations with lz4-sys, it is not yet possible to dynamically link
against the system's version of liblz4.

All features are disabled by default unless noted otherwise above. The build
process is defined in [`build.rs`].

[`build.rs`]: https://github.com/fede1024/rust-rdkafka/tree/master/rdkafka-sys/build.rs
[Apache Kafka]: https://kafka.apache.org
[CMake]: https://cmake.org
[libz-sys]: https://crates.io/crates/libz-sys
[curl-sys]: https://crates.io/crates/curl-sys
[lz4-sys]: https://crates.io/crates/lz4-sys
[mklove]: https://github.com/edenhill/mklove
[openssl-sys]: https://crates.io/crates/openssl-sys
[rdkafka]: https://docs.rs/rdkafka
[sasl2-sys]: https://docs.rs/sasl2-sys
[zstd-sys]: https://crates.io/crates/zstd-sys
# rdkafka-sys

Low level bindings to [librdkafka](https://github.com/edenhill/librdkafka),
a C library for the [Apache Kafka] protocol with producer, consumer, and
admin clients.

For a safe wrapper, see the [rdkafka] crate.

## Version

The rdkafka-sys version number is in the format `X.Y.Z+RX.RY.RZ`, where
`X.Y.Z` is the version of this crate and follows SemVer conventions, while
`RX.RY.RZ` is the version of the bundled librdkafka.

Note that versions before v2.0.0+1.4.2 did not follow this convention, and
instead directly correspond to the bundled librdkafka version.

## Build

### Known issues

* When any of librdkafka's optional dependencies are enabled, like libz or
OpenSSL, if you have multiple versions of that library installed upon your
system, librdkafka's build system may disagree with Cargo about which
version of the library to use! **This can result in subtly broken
builds,** if librdkafka compiles against the headers for one version but
Cargo links against a different version. For complete confidence when
building release binaries, use an environment like a Docker container or a
chroot jail where you can guarantee that only one version of each
dependency is present. The current design of Cargo unfortunately makes
this nearly impossible to fix.

* Windows builds always use the CMake build system, regardless of whether
the `cmake-build` Cargo feature is enabled or not. This is because Windows
is only supported with CMake as the build system.

### Features

By default a submodule with the librdkafka sources will be used to compile
and statically link the library.

The **`dynamic-linking`** feature can be used to link rdkafka to a locally
installed version of librdkafka: if the feature is enabled, the build script
will use `pkg-config` to check the version of the library installed in the
system, and it will configure the compiler to dynamically link against it.
The system version of librdkafka must exactly match the version of
librdkafka bundled with this crate.

The **`cmake-build`** feature builds librdkafka with its [CMake] build
system, rather than its default [mklove]-based build system. This feature
requires that CMake is installed on the build machine.

The following features directly correspond to librdkafka features (i.e.,
flags you would pass to `configure` if you were compiling manually).

* The **`ssl`** feature enables SSL support. By default, the system's
OpenSSL library is dynamically linked, but static linking of the version
bundled with the [openssl-sys] crate can be requested with the
`ssl-vendored` feature.
* The **`gssapi`** feature enables SASL GSSAPI support with Cyrus
libsasl2. By default the system's libsasl2 is dynamically linked, but
static linking of the version bundled with the [sasl2-sys] crate can be
requested with the `gssapi-vendored` feature.
* The **`libz`** feature enables support for zlib compression. This
feature is enabled by default. By default, the system's libz is
dynamically linked, but static linking of the version bundled with the
[libz-sys] crate can be requested with the `libz-static` feature.
* The **`curl`** feature enables the HTTP client via curl. By default, the
system's curl is dynamically linked, but static linking of the version
bundled with the [curl-sys] create can be requested with the
`curl-static` feature.
* The **`zstd`** feature enables support for ZSTD compression. By default,
this builds and statically links the version bundled with the [zstd-sys]
crate, but dynamic linking of the system's version can be requested with
the `zstd-pkg-config` feature.
* The **`external-lz4`** feature statically links against the copy of
liblz4 bundled with the [lz4-sys] crate. By default, librdkafka
statically links against its own bundled version of liblz4. Due to
limitations with lz4-sys, it is not yet possible to dynamically link
against the system's version of liblz4.

All features are disabled by default unless noted otherwise above. The build
process is defined in [`build.rs`].

[`build.rs`]: https://github.com/fede1024/rust-rdkafka/tree/master/rdkafka-sys/build.rs
[Apache Kafka]: https://kafka.apache.org
[CMake]: https://cmake.org
[libz-sys]: https://crates.io/crates/libz-sys
[curl-sys]: https://crates.io/crates/curl-sys
[lz4-sys]: https://crates.io/crates/lz4-sys
[mklove]: https://github.com/edenhill/mklove
[openssl-sys]: https://crates.io/crates/openssl-sys
[rdkafka]: https://docs.rs/rdkafka
[sasl2-sys]: https://docs.rs/sasl2-sys
[zstd-sys]: https://crates.io/crates/zstd-sys
7 changes: 3 additions & 4 deletions rdkafka-sys/build.rs
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
use std::borrow::Borrow;
use std::env;
use std::ffi::OsStr;
#[cfg(feature = "cmake-build")]
use std::fs;
use std::path::{Path, PathBuf};
use std::process::{self, Command};

Expand Down Expand Up @@ -88,7 +86,7 @@ fn main() {
}
}

#[cfg(not(feature = "cmake-build"))]
#[cfg(all(not(feature = "cmake-build"), not(target_os = "windows")))]
fn build_librdkafka() {
let mut configure_flags: Vec<String> = Vec::new();

Expand Down Expand Up @@ -200,8 +198,9 @@ fn build_librdkafka() {
println!("cargo:root={}", out_dir);
}

#[cfg(feature = "cmake-build")]
#[cfg(any(feature = "cmake-build", target_os = "windows"))]
fn build_librdkafka() {
use std::fs;
let mut config = cmake::Config::new("librdkafka");
let mut cmake_library_paths = vec![];

Expand Down
5 changes: 3 additions & 2 deletions rdkafka-sys/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,9 @@
//! dependency is present. The current design of Cargo unfortunately makes
//! this nearly impossible to fix.
//!
//! * Windows is only supported when using the CMake build system via the
//! `cmake-build` Cargo feature.
//! * Windows builds always use the CMake build system, regardless of whether
//! the `cmake-build` Cargo feature is enabled or not. This is because Windows
//! is only supported with CMake as the build system.
//!
//! ### Features
//!
Expand Down
5 changes: 3 additions & 2 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -152,13 +152,14 @@
//! * GNU `make`
//! * `pthreads`
//! * `zlib`: optional, but included by default (feature: `libz`)
//! * `cmake`: optional, *not* included by default (feature: `cmake-build`)
//! * `cmake`: optional¹, *not* included by default (feature: `cmake-build`)
//! * `libssl-dev`: optional, *not* included by default (feature: `ssl`)
//! * `libsasl2-dev`: optional, *not* included by default (feature: `gssapi`)
//! * `libzstd-dev`: optional, *not* included by default (feature: `zstd-pkg-config`)
//!
//! Note that using the CMake build system, via the `cmake-build` feature, is
//! encouraged if you can take the dependency on CMake.
//! encouraged if you can take the dependency on CMake. ¹Windows is only supported
//! via CMake, where it is the default build system.
//!
//! By default a submodule with the librdkafka sources pinned to a specific
//! commit will be used to compile and statically link the library. The
Expand Down