UCX Loadable Modules

Some parts of the code, especially those with external lib dependencies (e.g cuda, verbs, xpmem..) are built as separate libraries which are dl-opened at runtime. These are the main points of how it's done:

General

• UCX starting v1.6 is using loadable modules
• Modules are installed to <libdir>/ucx/
• Module code should depend only on "parent" components and not on "sibling" / "child" modules

Autoconf/Automake

• The source directory for the module contains Makefile.am and configure.m4
  • The parent directories (up all way to top_srcdir) must also contain Makefile.am and configure.m4 files
  • Makefile.am is added from parent dir by SUBDIRS = ...
  • configure.m4 is added from parent dir by m4_include([.../configure.m4]). note: must specify full path here
• configure.m4:
  • AC_ARG_WITH/AC_ARG_ENABLE
  • Autoconf checks for module dependencies (e.g pkg-config)
  • Setting xx_LDFLAGS/xx_CPPFLAGS/xx_LIBS
  • Adding the module to the list of parent component, so it would be loaded. e.g: uct_modules+=":xpmem"
  • AC_CONFIG_FILES([path/to/Makefile]) - to process Makefile.am
• Makefile.am:
  • There should be a top-level if HAVE_xx / endif wrapping the whole Makefile.am
  • module_LTLIBRARIES += ... - to declare module object as installed to moduledir
  • include $(top_srcdir)/config/module.am - for custom build rules which enable using the module from build directory by creating local symlinks
  • Add -version-info $(SOVERSION) to xx_LDFLAGS

Packaging (RPM)

• Spec file:
  • Module are built as RPM sub-package (eg ucx-xpmem), so they should have own description, files, etc.
  • Modules are build using RPM conditional builds
  • Spec file should have bcond_with/bcond_without declaration for each module
    • If the dependencies are satisfied in Fedora - it should be ON by default - bcond_without
    • If the dependencies are NOT satisfied in Fedora - it should be OFF by default - bcond_with
  • Build requirements should be added inside a condition the the module is built
  • All modules should be enabled/disabled explicitly for ./configure, to not be dependent on auto-detection, but on rpmbuild configuration only.
• Build process:
  • Source RPM - no with/without arguments are used.
  • Binary RPM in Fedora - the default with/without settings are used - since it's built as-is from src RPM.
    The following sub-packages would be built: (TBD)
  • Binary RPM for MLNX_OFED - need to use the following arguments: --with knem
    The following sub-packages would be built: (TBD)
  • The contrib/buildrpm.sh script, when run in "binary rpm build" mode, constructs rpmbuild command line according to what the local "./configure" script auto-detected. This is done by replacing the list of build modules in contrib/rpmdef.sh.in, and processing the resulting file in the script. The reason is that currently this step is done as part of regression testing (jenkins) only.
  • A module should be added explicitly to contrib/buildrpm.sh to add it as needed to rpmbuild command

Checklist for new module

• configure.m4
  • AC_ARG_WITH
  • Add to xx_modules
• Makefile.am
  • define module_LTLIBRARIES
  • so version
  • include $(top_srcdir)/config/module.am
• ucx.spec.in
  • bcond_with/bcond_without
  • conditional BuildRequires
  • add --with_xx/--without_xx to ./configure command
  • sub-package for module
• contrib/buildrpm.sh
  • pass --with/--without to rpmbuild