Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify build commands, update readme #583

Merged
merged 1 commit into from
Oct 17, 2017
Merged

Simplify build commands, update readme #583

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ set(DRISHTI_INCLUDE_DIRECTORIES "${CMAKE_CURRENT_LIST_DIR}/src/lib")
option(DRISHTI_BUILD_SHARED_SDK "Build shared SDK" OFF)
option(DRISHTI_BUILD_MERGED_SDK "Merge all static dependencies" OFF)

option(DRISHTI_COPY_3RDPARTY_LICENSES "Install 3rd party licenses" OFF)
option(DRISHTI_COPY_3RDPARTY_LICENSES "Install 3rd party licenses" ON)

# See: drishti-upload/config.cmake
# option(DRISHTI_BUILD_MIN_SIZE "Build minimum size lib (exclude training)" ON)
Expand Down Expand Up @@ -246,7 +246,7 @@ option(DRISHTI_SERIALIZE_WITH_CVMATIO "Perform serialization with cvmatio" ON)
# ${DRISHTISDK}/src/lib/drishti/<module>/ut/
# ${DRISHTISDK}/src/tests

# DRISHIT_BUILD_INTEGRATION_TESTS:
# DRISHTI_BUILD_INTEGRATION_TESTS:
# ${DRISHTISDK}/src/examples/integration/

option(DRISHTI_BUILD_BENCHMARKS "Build internal benchmarks (size and speed)." OFF)
Expand Down Expand Up @@ -384,13 +384,12 @@ option(DRISHTI_BUILD_ASAN_TEST "Intentional ASAN test" OFF)
option(DRISHTI_BUILD_ACF "Drishti ACF lib" ON)
option(DRISHTI_BUILD_FACE "Drishti face lib" ON)
option(DRISHTI_BUILD_HCI "Drishti video and HCI lib" ON)
option(DRISHTI_BUILD_REGRESSION_SIMD "Build multivariate gradient boosting using SIMD" ON)
option(DRISHTI_BUILD_REGRESSION_FIXED_POINT "Build multivariate gradient boosting using fixed point" ON)
drishti_option(DRISHTI_BUILD_REGRESSION_SIMD "Build multivariate gradient boosting using SIMD" ON IF(${DRISHTI_MOBILE}))
drishti_option(DRISHTI_BUILD_REGRESSION_FIXED_POINT "Build multivariate gradient boosting using fixed point" ON IF(${DRISHTI_MOBILE}))

# 3rd party libraries
option(DRISHTI_BUILD_DEST "Build dest lib" OFF)
option(DRISHTI_BUILD_EOS "EOS 2D-3D fitting" OFF) # duplicate symbols
option(DRISHTI_BUILD_CV_ML "Build OpenCV ML classes" OFF)

##################################################
### Installation/packaging paths and variables ###
Expand Down
42 changes: 29 additions & 13 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,13 @@ Goal: SDK size <= 1 MB and combined resources (object detection + regression mod

## Quick Start (i.e., How do I make this library work?)

Drishti is a [CMake](https://github.com/kitware/CMake) based project that uses the [Hunter](https://github.com/ruslo/hunter) package manager to download and build project dependencies from source as needed. Hunter contains [detailed documentation](https://docs.hunter.sh/en/latest), but a few high level notes are provided here to help orient first time users. In practice, some working knowledge of CMake may also be required. Hunter itself is written in CMake, and is installed as part of the build process from a single `HunterGate()` macro at the top of the root `CMakeLists.txt` file (typically `cmake/Hunter/HunterGate.cmake`) (you don't have to build or install it). Each CMake dependency's `find_package(FOO)` call that is paired with a `hunter_add_package(FOO CONFIG REQUIRED)` will be managed by Hunter. Additional configuration (i.e., custom version and/or CMake arguments) can be performed for each package with an optional `hunter_config(FOO VERSION 1.0.0 CMAKE_ARGS ${FOO_CMAKE_ARGS}))` call in an optional local config file that is specified in the `HunterGate()` call. In most cases, the only system requirement for building a Hunter project is a recent CMake with CURL support and a working compiler correpsonding to the operative toolchain.
Drishti is a [CMake](https://github.com/kitware/CMake) based project that uses the [Hunter](https://github.com/ruslo/hunter) package manager to download and build project dependencies from source as needed. Hunter contains [detailed documentation](https://docs.hunter.sh/en/latest), but a few high level notes are provided here to help orient first time users. In practice, some working knowledge of CMake may also be required. Hunter itself is written in CMake, and is installed as part of the build process from a single `HunterGate()` macro at the top of the root `CMakeLists.txt` file (typically `cmake/Hunter/HunterGate.cmake`) (you don't have to build or install it). Each CMake dependency's `find_package(FOO)` call that is paired with a `hunter_add_package(FOO CONFIG REQUIRED)` will be managed by Hunter. Additional configuration (i.e., custom version and/or CMake arguments) can be performed for each package with an optional `hunter_config(FOO VERSION 1.0.0 CMAKE_ARGS ${FOO_CMAKE_ARGS}))` call in an optional local config file that is specified in the `HunterGate()` call. In most cases, the only system requirement for building a Hunter project is a recent CMake with CURL support and a working compiler correpsonding to the operative toolchain. Hunter will maintain all dependencies in a [versioned](https://docs.hunter.sh/en/latest/overview/customization.html) local [cache](https://docs.hunter.sh/en/latest/overview/shareable.html) by default (typically `${HOME}/.hunter`) where they can be reused in subsequent builds and shared between different projects. They can also be stored in a server side [binary cache](https://docs.hunter.sh/en/latest/overview/binaries.html).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional configuration (i.e., custom version and/or CMake arguments) can be performed for each package with an optional hunter_config(FOO VERSION 1.0.0 CMAKE_ARGS ${FOO_CMAKE_ARGS})) call in an optional local config file that is specified in the HunterGate() call

Not sure why user may need to know this.

In most cases, the only system requirement for building a Hunter project is a recent CMake with CURL support and a working compiler correpsonding to the operative toolchain

Info is here:

Probably should be moved to "Quick start: Tools".

They can also be stored in a server side binary cache.

Not very informative. How it can be helpful for the first time user?

Copy link
Collaborator Author

@headupinclouds headupinclouds Oct 16, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why user may need to know this.

I think that makes sense. That is probably more critical for creating a new project. I'll cut this (in line w/ preference for bare bones description that doesn't duplicate Hunter docs too much).

Info is here: https://docs.hunter.sh/en/latest/contributing.html#reporting-bugs

I'll link this. I think it is important to keep it in the "setup" notes. I believe the default Ubuntu 16.04 doesn't have CURL support, for example. (I'll incorporate feedback and update via direct README.md commit after the merge.)

Not very informative. How it can be helpful for the first time user?

In the toolchain discussion, the CI builds + cache are mentioned. I think it is important to introduce the idea of a binary cache so the user understands the pre-built packages are downloaded from a server. Maybe a little more information could help.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the default Ubuntu 16.04 doesn't have CURL support, for example

I'm pretty sure it has. The CURL support issue (OpenSSL support to be precise) usually about manually compiled CMake.

In the toolchain discussion, the CI builds + cache are mentioned

Okay, I see. However not all of them are "stable", e.g. xcode-hid-sections.


The [Travis](https://github.com/elucideye/drishti/blob/master/.travis.yml) (Linux/OSX/iOS/Android) and [Appveyor](https://github.com/elucideye/drishti/blob/master/appveyor.yml) (Windows) CI scripts in the project's root directory can serve as a reference for basic setup when building from source. To support cross platform builds and testing, the CI scripts make use of [Polly](https://github.com/ruslo/polly): a set of common CMake toolchains paired with a simple `polly.py` CMake build script. Polly is used here for convenience to generate `CMake` command line invocations -- it is not required for building Hunter projects.

To reproduce the CI builds on a local host, you must install `CMake` and `Polly` and add the paths for the `cmake` and `polly.py` executables to your system's `PATH` variable. The `bin/hunter_env.{sh,cmd}` scripts (used in the CI builds) can be used to install these tools for you in your local terminal. You may want to add the `PATH` variables permanently to your `.bashrc` file (or equivalent) for future sessions.

Linux/OS X:
Linux/OSX:
```
source bin/hunter_env.sh
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now it can be:

  • Install CMake
  • Install Python (for Polly)
  • Clone Polly and add <polly>/bin to PATH

?

By the way "Install CMake" includes not only CMake but the real build tools and compiler. So probably this page can be helpful: http://cgold.readthedocs.io/en/latest/first-step.html

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the bullet list as a high level description.

So probably this page can be helpful: http://cgold.readthedocs.io/en/latest/first-step.html

Perfect. I'll add a link appropriately in the text.

```
Expand All @@ -51,22 +51,38 @@ Windows:
bin\hunter_env.cmd
```

After the environment is configured, you can build for any supported `Polly` toolchain (see `polly.py --help`) using the `bin/drishti_build.{sh,cmd}` scripts.
After the environment is configured, you can build for any supported `Polly` toolchain (see `polly.py --help`) with a command like this:

Linux/OS X:
```
TOOLCHAIN=osx-10-11-hid-sections-lto
CONFIG=Release
INSTALL=--install
bin/drishti_build.sh "${TOOLCHAIN}" "${CONFIG}" "${INSTALL}"
polly.py --toolchain ${TOOLCHAIN} --config ${CONFIG} --fwd HUNTER_CONFIGURATION_TYPES=${CONFIG} --install --verbose
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess first time user is not interested in --install.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think --install is important. My sense is that the majority of users are looking for a working lib they can stick into their project. An "Integration" section discussing Hunter vs Non-Hunter (shared lib) solutions is also needed in a follow up.

```

You can override the default CMake options in the project by adding KEY=VALUE pairs to a forwarded list like this `--fwd DRISHTI_SOME_OPTION1=YES DRISHTI_SOME_OPTION2=YES`. The `--fwd HUNTER_CONFIGURATION_TYPES=${CONFIG}` option is specified above to limit the Hunter package builds to a single configuration type (typically Release) when using multi-configuration generators (Visual Studio, Xcode, etc). It is not required for standard `clang` or `gcc` builds.

The configurations listed below have all been tested. In general, most C++11 toolchains should work with minimal effort. A CI comment indicates that the configuration is part of the Travis or Appveyor CI tests, so all Hunter packages will be available in the server side binary cache.

Linux (Ubunty Trusty 14.04):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we can save some space by omitting TOOLCHAIN=. Just say that "Next arguments for --toolchain tested and can be used...".

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could go either way on this. The CONFIG does impact the cache. If a user wants a quick build they will need to know which TOOLCHAIN + CONFIG pair to use.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CONFIG does impact the cache. If a user wants a quick build they will need to know which TOOLCHAIN + CONFIG pair to use

Okay, I see it now.

* TOOLCHAIN=gcc-5-pic-hid-sections-lto CONFIG=Release # CI
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess CONFIG=... not needed (?)

* TOOLCHAIN=libcxx CONFIG=Release # w/ clang 3.8

OSX:
* TOOLCHAIN=osx-10-11-hid-sections-lto CONFIG=Release # CI
* TOOLCHAIN=osx-10-12-sanitize-address-hid-sections CONFIG=Release # CI
* TOOLCHAIN=xcode-hid-sections CONFIG=Release # generic

iOS:
* TOOLCHAIN=ios-nocodesign-10-1-arm64-dep-9-0-device-libcxx-hid-sections-lto CONFIG=MinSizeRel # CI
* TOOLCHAIN=ios-10-1-arm64-dep-8-0-hid-sections CONFIG=Release

Android (from OSX):
* TOOLCHAIN=android-ndk-r10e-api-19-armeabi-v7a-neon-hid-sections CONFIG=MinSizeRel # CI
* TOOLCHAIN=android-ndk-r10e-api-19-armeabi-v7a-neon-hid-sections-lto CONFIG=MinSizeRel

Windows:
```
set TOOLCHAIN vs-14-2015-win64-sdk-8-1
set CONFIG Release
bin\drishti_build.cmd %TOOLCHAIN% %CONFIG%
```
* TOOLCHAIN=vs-14-2015-sdk-8-1 CONFIG=Release # CI
* TOOLCHAIN=vs-14-2015-sdk-8-1 CONFIG=Debug # CI
* TOOLCHAIN=vs-14-2015-win64-sdk-8-1 CONFIG=Release # CI
* TOOLCHAIN=vs-14-2015-win64-sdk-8-1 CONFIG=Debug # CI

The polly out of source build trees are located in `_builds/${TOOLCHAIN}`, the final build products (the stuff you want) are installed in `_install/${TOOLCHAIN}`, and the build logs are dumped in `_logs/${TOOLCHAIN}`. The iOS frameworks are installed in `_frameworks/${TOOLCHAIN}`.

Expand Down
2 changes: 1 addition & 1 deletion bin/drishti_build.cmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ python %POLLY_ROOT%\bin\polly.py ^
--config "%1%" ^
--toolchain "%2%" ^
--test ^
--fwd HUNTER_USE_CACHE_SERVERS=NO ^
--fwd HUNTER_USE_CACHE_SERVERS=YES ^
HUNTER_DISABLE_BUILDS=NO ^
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUNTER_USE_CACHE_SERVERS=ONLY HUNTER_DISABLE_BUILDS=YES?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script was added based on issue discussions to provide an easy option for local host builds. Once cmake defaults + documentation is cleaned up I think it can be removed (along with some others). It isn't actually used right now in appveyor.yml. The appveyor build-appveyor.cmd script used in CI still forces cache behavior:

- cmd: bin\build-appveyor.cmd "%CONFIG%" "%TOOLCHAIN%"

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

DRISHTI_COPY_3RDPARTY_LICENSES=ON ^
DRISHTI_BUILD_TESTS=ON ^
Expand Down
2 changes: 0 additions & 2 deletions bin/drishti_build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,6 @@ ARGS=(
--verbose
--ios-multiarch --ios-combined
--fwd
DRISHTI_BUILD_REGRESSION_SIMD=NO
DRISHTI_BUILD_REGRESSION_FIXED_POINT=NO
DRISHTI_BUILD_TESTS=YES
DRISHTI_BUILD_EXAMPLES=YES
DRISHTI_COPY_3RDPARTY_LICENSES=ON
Expand Down