[RELEASE] MRC v24.10.00 #507

dagardner-nv · 2024-10-21T18:08:49Z

❄️ Code freeze for `branch-24.10` and `v24.10` release

What does this mean?

Only critical/hotfix level issues should be merged into branch-24.10 until release (merging of this PR).

All other development PRs should be retargeted towards the next release branch: branch-25.02.

What is the purpose of this PR?

Update documentation
Allow testing for the new release
Enable a means to merge branch-24.10 into main for the release

Forward-merge branch-24.06 into branch-24.10

* Invoke `morpheus_utils_initialize_install_prefix` function after calling `project` * Related to nv-morpheus/Morpheus#1776 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #485

* When a Python generator source yields a value, and the subscriber is no longer subscribed, stop the source. * Fix out of date docstring comment. This is a partial fix for nv-morpheus/Morpheus#1838 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: #493

* Allows a Python generator source to check if the subscriber is still subscribed. * Define a class `SubscriberFuncWrapper` for Python sources rather than just a lambda. The reason is that python objects captured by the lambda need to be destroyed while the gil is held, which causes a problem if the lambda is destroyed unexpectedly. * Update `conftest.py` to set the loglevel to `DEBUG` if the `GLOG_v` environment variable is defined. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #496

* Since CPU-only mode will become a supported feature we want to avoid unnecessary warnings. Relates to nv-morpheus/Morpheus#1851 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #497

…iber` (#499) * Remove the `make_source_subscriber` method in favor of inspecting the Python function signature. * Since the `make_source_subscriber` method was never part of a release I think this can still be considered a non-breaking change. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #499

Closes #478 Requires nv-morpheus/utilities#75 Authors: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) - David Gardner (https://github.com/dagardner-nv) URL: #494

If the NUMA node check fails for a GPU, the entire pipeline cannot start. This allows setting `MRC_SKIP_NUMA_CHECK=1` to bypass the assertion if the check fails. Closes #504 Authors: - Michael Demoret (https://github.com/mdemoret-nv) - David Gardner (https://github.com/dagardner-nv) Approvers: - Devin Robison (https://github.com/drobison00) - David Gardner (https://github.com/dagardner-nv) URL: #505

* Include MRC libs with quotes not <> * Add MRC version string to coro module (all the other Python modules have this) * Fix spelling mistake in a comment in `python/mrc/core/executor.cpp` Closes [#492](#492) Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #503

`rapidsai/shared-action-workflows/get-pr-info` was deleted in favor of `nv-gha-runners/get-pr-info` rapidsai/shared-actions#11 Authors: - Christopher Harris (https://github.com/cwharris) - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #500

This PR is replacing the `VAULT_HOST` variable with `AWS_ROLE_ARN`. This is required to use the new token service to get AWS credentials Authors: - Jordan Jacobelli (https://github.com/jjacobelli) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #506

Overview of new classes added in this PR: - `DynamicRouterBase` - Sink - `WritableProvider`, `ReadableAcceptor`, `SinkChannelOwner` - Source - `MultiWritableAcceptor`, `MultiReadableProvider`, `MultiSourceChannelOwner` - Runnable with ability to change number of outputs at runtime - Cannot be added to a Segment - Good backpressure support. Can still process requests if downstream blocks - `LambdaDynamicRouter` - Provide key function via lambda - `DynamicRouterComponentBase` - Sink - `ForwardingWritableProvider` - Source - `MultiWritableAcceptor`, `MultiReadableProvider`, `MultiSourceChannelOwner` - Non-Runnable with ability to change number of outputs at runtime - Limited backpressure support. Will block if downstream blocks - `LambdaDynamicRouterComponent` - Provide key function via lambda - `StaticRouterRunnableBase` - Sink - `WritableProvider`, `ReadableAcceptor`, `SinkChannelOwner` - Source - `MultiWritableAcceptor`, `MultiReadableProvider`, `MultiSourceChannelOwner` - Runnable with fixed number of outputs - Can be added to a Segment since it supports child outputs - Good backpressure support. Can still process requests if downstream blocks - `LambdaStaticRouterRunnable` - Provide key function via lambda - `StaticRouterComponentBase` - Sink - `ForwardingWritableProvider` - Source - `MultiWritableAcceptor`, `MultiReadableProvider`, `MultiSourceChannelOwner` - Non-Runnable with fixed number of outputs - Can be added to a Segment since it supports child outputs - Limited backpressure support. Will block if downstream blocks - `LambdaStaticRouterComponent` - Provide key function via lambda Closes #491 Authors: - Michael Demoret (https://github.com/mdemoret-nv) - David Gardner (https://github.com/dagardner-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #502

#511) Fixes a few issues: - Headers were being installed incorrectly with new `HEADER_SET` - Fixed packages for pymrc component - Standardized the component naming Closes #509 Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) - Christopher Harris (https://github.com/cwharris) URL: #511

build interface includes was accidentally using ``` .../build/conda/cuda-12.5/release/autogenerated/include ``` instead of ``` .../build/conda/cuda-12.5/release/cpp/mrc/autogenerated/include ``` this PR fixes that, such when depending on mrc from down-stream libraries, the following error does not occur: ``` CMake Error in python/morpheus_llm/morpheus_llm/_lib/CMakeLists.txt: Imported target "mrc::pymrc" includes non-existent path "/home/coder/mrc/build/conda/cuda-12.5/release/autogenerated/include" in its INTERFACE_INCLUDE_DIRECTORIES. Possible reasons include: * The path was deleted, renamed, or moved to another location. * An install or uninstall procedure did not complete successfully. * The installation package was faulty and references files it does not provide. ``` Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #514

Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: #517

codecov · 2024-11-01T22:08:47Z

Codecov Report

Attention: Patch coverage is 77.05128% with 179 lines in your changes missing coverage. Please review.

Project coverage is 74.15%. Comparing base (cc89f5f) to head (e1f286f).
Report is 44 commits behind head on main.

Files with missing lines	Patch %	Lines
cpp/mrc/include/mrc/segment/object.hpp	80.48%	24 Missing ⚠️
cpp/mrc/include/mrc/node/source_properties.hpp	12.00%	22 Missing ⚠️
python/mrc/core/node.cpp	84.95%	17 Missing ⚠️
cpp/mrc/include/mrc/channel/status.hpp	0.00%	15 Missing ⚠️
cpp/mrc/include/mrc/node/operators/router.hpp	85.71%	14 Missing ⚠️
...pp/mrc/src/internal/segment/builder_definition.cpp	81.03%	11 Missing ⚠️
cpp/mrc/include/mrc/edge/edge_builder.hpp	44.44%	10 Missing ⚠️
cpp/mrc/include/mrc/segment/builder.hpp	38.46%	8 Missing ⚠️
python/mrc/_pymrc/src/segment.cpp	80.00%	8 Missing ⚠️
cpp/mrc/include/mrc/node/operators/zip.hpp	90.14%	7 Missing ⚠️
... and 12 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #507      +/-   ##
==========================================
+ Coverage   74.09%   74.15%   +0.05%     
==========================================
  Files         403      407       +4     
  Lines       14401    15049     +648     
  Branches     1132     1191      +59     
==========================================
+ Hits        10670    11159     +489     
- Misses       3731     3890     +159

Flag	Coverage Δ
cpp	`70.69% <67.55%> (-0.16%)`	⬇️
py	`44.23% <73.59%> (+2.41%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
cpp/mrc/include/mrc/core/fiber_pool.hpp	`100.00% <100.00%> (ø)`
cpp/mrc/include/mrc/core/task_queue.hpp	`94.73% <100.00%> (ø)`
cpp/mrc/include/mrc/core/utils.hpp	`45.16% <ø> (ø)`
cpp/mrc/include/mrc/edge/deferred_edge.hpp	`100.00% <ø> (ø)`
cpp/mrc/include/mrc/edge/edge.hpp	`91.56% <100.00%> (+0.54%)`	⬆️
cpp/mrc/include/mrc/edge/edge_channel.hpp	`85.00% <100.00%> (+0.78%)`	⬆️
cpp/mrc/include/mrc/edge/edge_connector.hpp	`33.33% <ø> (-4.17%)`	⬇️
cpp/mrc/include/mrc/edge/edge_readable.hpp	`84.84% <ø> (ø)`
cpp/mrc/include/mrc/edge/edge_writable.hpp	`90.62% <ø> (ø)`
...mrc/experimental/modules/mirror_tap/mirror_tap.hpp	`100.00% <100.00%> (ø)`
... and 48 more

... and 34 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cc89f5f...e1f286f. Read the comment docs.

dagardner-nv and others added 11 commits July 3, 2024 10:18

Creating branch for v24.10

f754c78

Updating versions for v24.10.00

246ac97

Merge pull request #484 from nv-morpheus/branch-24.06

8377f6a

Forward-merge branch-24.06 into branch-24.10

dagardner-nv added the ! - Release Related to a Major Release label Oct 21, 2024

dagardner-nv requested review from a team as code owners October 21, 2024 18:08

dagardner-nv self-assigned this Oct 21, 2024

cwharris and others added 6 commits October 21, 2024 23:15

Updating CHANGELOG

fff44d6

AnuradhaKaruppiah approved these changes Nov 1, 2024

View reviewed changes

dagardner-nv and others added 2 commits November 1, 2024 21:48

IWYU fixes (#517)

af67e11

Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: #517

Updating CHANGELOG

e1f286f

dagardner-nv merged commit bb7a868 into main Nov 1, 2024
26 of 29 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RELEASE] MRC v24.10.00 #507

[RELEASE] MRC v24.10.00 #507

dagardner-nv commented Oct 21, 2024

codecov bot commented Nov 1, 2024

[RELEASE] MRC v24.10.00 #507

[RELEASE] MRC v24.10.00 #507

Conversation

dagardner-nv commented Oct 21, 2024

❄️ Code freeze for branch-24.10 and v24.10 release

What does this mean?

What is the purpose of this PR?

codecov bot commented Nov 1, 2024

Codecov Report

❄️ Code freeze for `branch-24.10` and `v24.10` release