[Humble backport] rosbag2_storage_mcap: merge into rosbag2 repo (#1163)…

… (#1189) * [backport] rosbag2_storage_mcap: merge into rosbag2 repo (#1163) * rosbag2_storage_mcap: merge into ros2/rosbag2 Signed-off-by: James Smith <[email protected]> mcap_storage: 'none' is a valid storage preset profile (#86) Signed-off-by: James Smith <[email protected]> bloom: add changelog changes 0.6.0 * ci: include rosbag2_storage_mcap Signed-off-by: James Smith <[email protected]> * package.xml: include ROS Tooling WG maintainers Signed-off-by: James Smith <[email protected]> * rosbag2_storage_mcap: update readme after move Signed-off-by: James Smith <[email protected]> Signed-off-by: James Smith <[email protected]> * zstd_vendor: do not remove zstd_errors.h Signed-off-by: James Smith <[email protected]> Signed-off-by: James Smith <[email protected]> (cherry picked from commit 953c8ed) Signed-off-by: James Smith <[email protected]>
ros2 · May 23, 2023 · 14c489d · 14c489d
1 parent 9e001ec
commit 14c489d
Show file tree

Hide file tree

Showing 30 changed files with 2,363 additions and 11 deletions.
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
@@ -82,6 +82,27 @@ jobs:
             rosbag2_transport
             shared_queues_vendor
 
+  ament_lint_clang_format: # Linters applicable to C++ packages formatted with clang-format
+    name: ament_${{ matrix.linter }}
+    runs-on: ubuntu-latest
+    container:
+      image: rostooling/setup-ros-docker:ubuntu-focal-ros-rolling-ros-base-latest
+    strategy:
+      fail-fast: false
+      matrix:
+          linter: [cppcheck, cpplint, clang_format]
+          include:
+          - linter: clang_format
+            arguments: "--config rosbag2_storage_mcap/.clang-format"
+    steps:
+    - uses: actions/checkout@v2
+    - uses: ros-tooling/[email protected]
+      with:
+        linter: ${{ matrix.linter }}
+        arguments: ${{ matrix.arguments }}
+        distribution: rolling
+        package-name: rosbag2_storage_mcap
+
   ament_lint_python: # Linters applicable to Python packages
     name: ament_${{ matrix.linter }}
     runs-on: ubuntu-latest

diff --git a/mcap_vendor/CHANGELOG.rst b/mcap_vendor/CHANGELOG.rst
@@ -0,0 +1,64 @@
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Changelog for package mcap_vendor
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+0.6.0 (2022-11-28)
+------------------
+* Fix Windows build (`#73 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/73>`_)
+  Update mcap version to newest windows-compatible release.
+  Add visibility macros for tests.
+  Add clang-format preprocessor indentation for visibility_control to be readable.
+* Contributors: Emerson Knapp
+
+0.5.0 (2022-11-02)
+------------------
+* mcap_vendor: update to v0.6.0 (`#69 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/69>`_)
+* Cleanup in `mcap_vendor` package (`#62 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/62>`_)
+* Switch to using the vendored zstd library. (`#59 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/59>`_)
+* Contributors: Chris Lalancette, Michael Orlov, James Smith
+
+0.4.0 (2022-10-06)
+------------------
+
+0.2.0 (2022-09-08)
+------------------
+* Support timestamp-ordered playback (`#50 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/50>`_)
+* Support regex topic filtering
+* Contributors: James Smith
+
+0.1.7 (2022-08-15)
+------------------
+* Add all lz4 sources to fix undefined symbols at runtime (`#46 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/46>`_)
+* Contributors: Emerson Knapp
+
+0.1.6 (2022-07-22)
+------------------
+* Upgrade mcap to fix LZ4 error and segfault (`#42 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/42>`_)
+  Incorporates fixes from https://github.com/foxglove/mcap/pull/478 and https://github.com/foxglove/mcap/pull/482
+* Add missing buildtool_depend on git (`#37 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/37>`_)
+  This vendor package uses git to fetch sources for other packages. It should declare a dependency on that build tool.
+  This should address the current cause of RPM build failures for RHEL: https://build.ros2.org/view/Rbin_rhel_el864/job/Rbin_rhel_el864__mcap_vendor__rhel_8_x86_64__binary/
+* Contributors: Jacob Bandes-Storch, Scott K Logan
+
+0.1.5 (2022-04-25)
+------------------
+* Test Foxy & Galactic in CI, fix missing test_depends in mcap_vendor/package.xml (`#33 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/33>`_)
+* Contributors: Jacob Bandes-Storch
+
+0.1.4 (2022-04-21)
+------------------
+* fix: minor issues (`#31 <https://github.com/wep21/rosbag2_storage_mcap/issues/31>`_)
+  * remove unnecessary block
+  * use target_link_libraries instead of ament_target_dependencies
+  * remove ros environment
+  * add prefix to compile definition
+* Update email address for Foxglove maintainers (`#32 <https://github.com/wep21/rosbag2_storage_mcap/issues/32>`_)
+* Contributors: Daisuke Nishimatsu, Jacob Bandes-Storch
+
+0.1.3 (2022-04-20)
+------------------
+
+0.1.2 (2022-04-20)
+------------------
+* Added mcap_vendor package. Updated CMakeLists.txt to fetch dependencies with FetchContent rather than Conan.
+* Contributors: Jacob Bandes-Storch
diff --git a/mcap_vendor/CMakeLists.txt b/mcap_vendor/CMakeLists.txt
@@ -0,0 +1,82 @@
+cmake_minimum_required(VERSION 3.5)
+project(mcap_vendor LANGUAGES C CXX ASM)
+
+## Dependencies
+find_package(ament_cmake REQUIRED)
+find_package(zstd_vendor REQUIRED)
+find_package(zstd REQUIRED)
+
+## Compile options
+if(NOT CMAKE_CXX_STANDARD)
+  set(CMAKE_CXX_STANDARD 17)
+  set(CMAKE_CXX_STANDARD_REQUIRED ON)
+  set(CMAKE_CXX_EXTENSIONS OFF)
+endif()
+if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
+  add_compile_options(-Wall -Wextra -Wpedantic -Werror)
+endif()
+if (MSVC)
+  add_compile_options(/W4 /WX
+    /wd4251 # suppress warning about having template instances (such as std::string) as public class members
+  )
+  # suppress error disallowing `fopen` in favor of `fopen_s`
+  add_compile_definitions(_CRT_SECURE_NO_WARNINGS)
+endif()
+
+## Define vendor macro
+macro(build_mcap_vendor)
+  include(FetchContent)
+  fetchcontent_declare(mcap
+    GIT_REPOSITORY https://github.com/foxglove/mcap.git
+    GIT_TAG dc6561d9ba867901709e36526dcf7f7359861e9c # releases/cpp/v0.7.0
+  )
+  fetchcontent_makeavailable(mcap)
+
+  fetchcontent_declare(lz4
+    GIT_REPOSITORY https://github.com/lz4/lz4.git
+    GIT_TAG d44371841a2f1728a3f36839fd4b7e872d0927d3 # v1.9.3
+  )
+  fetchcontent_makeavailable(lz4)
+
+  file(GLOB _lz4_srcs
+    ${lz4_SOURCE_DIR}/lib/*.c)
+
+  add_library(mcap SHARED
+    src/main.cpp
+    ${_lz4_srcs}
+  )
+
+  set(_mcap_include_dir ${mcap_SOURCE_DIR}/cpp/mcap/include)
+
+  target_include_directories(mcap PRIVATE
+    ${lz4_SOURCE_DIR}/lib
+  )
+  target_include_directories(mcap PUBLIC
+    "$<BUILD_INTERFACE:${_mcap_include_dir}>"
+    "$<INSTALL_INTERFACE:include/${PROJECT_NAME}>"
+  )
+  ament_target_dependencies(mcap zstd)
+
+  install(
+    DIRECTORY ${_mcap_include_dir}/mcap
+    DESTINATION include/${PROJECT_NAME}
+  )
+
+  install(
+    TARGETS mcap
+    EXPORT mcap
+    ARCHIVE DESTINATION lib
+    LIBRARY DESTINATION lib
+    RUNTIME DESTINATION bin
+  )
+endmacro()
+
+## Call vendor macro
+build_mcap_vendor()
+
+ament_export_include_directories(include/${PROJECT_NAME})
+ament_export_targets(mcap HAS_LIBRARY_TARGET)
+ament_export_dependencies(zstd_vendor zstd)
+
+## Package
+ament_package()
diff --git a/mcap_vendor/package.xml b/mcap_vendor/package.xml
@@ -0,0 +1,19 @@
+<?xml version="1.0"?>
+<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
+<package format="3">
+  <name>mcap_vendor</name>
+  <version>0.6.0</version>
+  <description>mcap vendor package</description>
+  <maintainer email="[email protected]">Foxglove</maintainer>
+  <maintainer email="[email protected]">ROS Tooling Working Group</maintainer>
+  <license>Apache License 2.0</license>
+
+  <buildtool_depend>ament_cmake</buildtool_depend>
+  <buildtool_depend>git</buildtool_depend>
+
+  <depend>zstd_vendor</depend>
+
+  <export>
+    <build_type>ament_cmake</build_type>
+  </export>
+</package>
diff --git a/mcap_vendor/src/main.cpp b/mcap_vendor/src/main.cpp
@@ -0,0 +1,16 @@
+// Copyright 2022, Foxglove Technologies. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#define MCAP_IMPLEMENTATION
+#include <mcap/mcap.hpp>
diff --git a/rosbag2_storage_mcap/.clang-format b/rosbag2_storage_mcap/.clang-format
@@ -0,0 +1,37 @@
+---
+Language: Cpp
+Standard: c++17
+BasedOnStyle: Google
+
+AllowShortFunctionsOnASingleLine: Empty
+AllowShortLambdasOnASingleLine: Empty
+AccessModifierOffset: -2
+TabWidth: 2
+ContinuationIndentWidth: 2
+UseTab: Never
+BreakConstructorInitializers: BeforeComma
+BraceWrapping:
+  AfterClass: true
+  AfterFunction: true
+  AfterNamespace: true
+  AfterStruct: true
+  AfterEnum: true
+BreakBeforeBraces: Custom
+ColumnLimit: 100
+ConstructorInitializerAllOnOneLineOrOnePerLine: false
+DerivePointerAlignment: false
+FixNamespaceComments: true
+PointerAlignment: Middle
+ReflowComments: false
+SortIncludes: true
+IndentPPDirectives: BeforeHash
+
+IncludeCategories:
+  - Regex: '^"'
+    Priority: 1
+  - Regex: "^<mcap"
+    Priority: 2
+  - Regex: "^<.*/"
+    Priority: 3
+  - Regex: "^<.*"
+    Priority: 4
diff --git a/rosbag2_storage_mcap/.vscode/settings.json b/rosbag2_storage_mcap/.vscode/settings.json
@@ -0,0 +1,13 @@
+// -*- jsonc -*-
+{
+    "git.alwaysSignOff": true,
+
+    // https://github.com/microsoft/vscode-cpptools/issues/722
+    "C_Cpp.autoAddFileAssociations": false,
+    "C_Cpp.default.cppStandard": "c++17",
+
+    "[cpp]": {
+        "editor.formatOnSave": true
+    },
+    "cmake.sourceDirectory": "${workspaceFolder}",
+}
diff --git a/rosbag2_storage_mcap/CHANGELOG.rst b/rosbag2_storage_mcap/CHANGELOG.rst
@@ -0,0 +1,127 @@
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Changelog for package rosbag2_storage_mcap
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+0.6.0 (2022-11-28)
+------------------
+* mcap_storage: 'none' is a valid storage preset profile (`#86 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/86>`_)
+* mcap_storage: handle update_metadata call (`#83 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/83>`_)
+* Update clang-format rules to fit ROS 2 style guide (`#80 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/80>`_)
+* Revert "read_order: throw exception from set_read_order for unsupported orders"
+  This reverts commit aef9b9a65293f9e5d80a858ef84e485a8655a0c0.
+* read_order: throw exception from set_read_order for unsupported orders
+* Fix compile flags to work on rosbag_storage:0.17.x (`#78 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/78>`_)
+  This fixes the compile flags for rolling, which has two versions -- one that does not support read order (0.17.x) and one that does support read order (0.18.x).
+* Fix Windows build (`#73 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/73>`_)
+  Update mcap version to newest windows-compatible release.
+  Add visibility macros for tests.
+  Add clang-format preprocessor indentation for visibility_control to be readable.
+* Contributors: Andrew Symington, Emerson Knapp, James Smith, james-rms
+
+0.5.0 (2022-11-02)
+------------------
+* set defaults for SQLite plugin parity (`#68 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/68>`_)
+* rosbag2_storage_mcap: add storage preset profiles (`#57 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/57>`_)
+* rename test_fixture_interfaces package to testdata (`#64 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/64>`_)
+* Switch to using the vendored zstd library. (`#59 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/59>`_)
+* Add set_read_order reader API (`#54 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/54>`_)
+* Contributors: Chris Lalancette, Emerson Knapp, James Smith
+
+0.4.0 (2022-10-06)
+------------------
+* Some minor improvements in rosbag2_storage_mcap after review (`#58 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/58>`_)
+  1. Fixed some findings from Clang-Tidy
+  1. Some renames according to the ROS2 coding style
+  1. Add default initializations for member variables
+  1. Moved code responsible for adding schema and channel from write(msg)
+  to create_topic(topic) method to reduce performance burden on first
+  message write and in lieu to preparation for moving schema collection
+  process to upper SequentialWriter layer.
+* Revert "rosbag2_storage_mcap: add storage preset profiles"
+  This reverts commit 38830add3935b978968fe2703d3180b413ccc8c2.
+* rosbag2_storage_mcap: add storage preset profiles
+* Contributors: James Smith, Michael Orlov
+
+0.3.0 (2022-09-09)
+------------------
+* Store IDL message definitions in Schema records when no MSG definition is available
+  (`#43 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/43>`_)
+* Contributors: James Smith
+
+0.2.0 (2022-09-08)
+------------------
+* Support timestamp-ordered playback (`#50 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/50>`_)
+* Support regex topic filtering
+* Contributors: James Smith
+
+0.1.7 (2022-08-15)
+------------------
+* Add all lz4 sources to fix undefined symbols at runtime (`#46 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/46>`_)
+* Contributors: Emerson Knapp
+
+0.1.6 (2022-07-22)
+------------------
+* Upgrade mcap to fix LZ4 error and segfault (`#42 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/42>`_)
+  Incorporates fixes from https://github.com/foxglove/mcap/pull/478 and https://github.com/foxglove/mcap/pull/482
+* Contributors: Jacob Bandes-Storch
+
+0.1.5 (2022-04-25)
+------------------
+* Fix build for Foxy (`#34 <https://github.com/ros-tooling/rosbag2_storage_mcap/issues/34>`_)
+* Contributors: Jacob Bandes-Storch
+
+0.1.4 (2022-04-21)
+------------------
+* fix: minor issues (`#31 <https://github.com/wep21/rosbag2_storage_mcap/issues/31>`_)
+  * remove unnecessary block
+  * use target_link_libraries instead of ament_target_dependencies
+  * remove ros environment
+  * add prefix to compile definition
+* Update email address for Foxglove maintainers (`#32 <https://github.com/wep21/rosbag2_storage_mcap/issues/32>`_)
+* Contributors: Daisuke Nishimatsu, Jacob Bandes-Storch
+
+0.1.3 (2022-04-20)
+------------------
+
+0.1.2 (2022-04-20)
+------------------
+* Added mcap_vendor package. Updated CMakeLists.txt to fetch dependencies with FetchContent rather than Conan.
+* Contributors: Jacob Bandes-Storch
+
+0.1.1 (2022-04-01)
+------------------
+* CMake build script will now execute pip install conan automatically.
+* Contributors: Daisuke Nishimatsu
+
+0.1.0 (2022-03-24)
+------------------
+* [1.0.0] Use Summary section for get_metadata() and seek(), implement remaining methods (`#17 <https://github.com/wep21/rosbag2_storage_mcap/issues/17>`_)
+* feat: add play impl (`#16 <https://github.com/wep21/rosbag2_storage_mcap/issues/16>`_)
+* chore: refine package.xml (`#15 <https://github.com/wep21/rosbag2_storage_mcap/issues/15>`_)
+* Don't throw when READ_WRITE mode is used; add .mcap file extension to recorded files (`#14 <https://github.com/wep21/rosbag2_storage_mcap/issues/14>`_)
+  I may be missing something, but from a cursory glance at [this code](https://github.com/ros2/rosbag2/blob/342d8ed3c1c4ae0411a4a92b60e79a728b8974b8/rosbag2_storage/src/rosbag2_storage/impl/storage_factory_impl.hpp#L108-L135), it appears that the `APPEND` mode is never used. This means we need to support `READ_WRITE`.
+  This also adds a `.mcap` extension to recorded file names.
+* Add dynamic message definition lookup (`#13 <https://github.com/wep21/rosbag2_storage_mcap/issues/13>`_)
+  Currently, an exception will be thrown if lookup fails.
+* Switch C++ formatter to clang-format (`#12 <https://github.com/wep21/rosbag2_storage_mcap/issues/12>`_)
+  Remove uncrustify linter in favor of clang-format, which is easier to configure for use in VS Code format-on-save.
+* Merge pull request `#7 <https://github.com/wep21/rosbag2_storage_mcap/issues/7>`_ from ros-tooling/jhurliman/reader-writer
+  Reader and writer implementation
+* uninitialized struct
+* lint
+* lint
+* lint
+* Reader and writer implementation
+* Merge pull request `#6 <https://github.com/wep21/rosbag2_storage_mcap/issues/6>`_ from wep21/add-metadata-impl
+  feat: add metadata impl
+* feat: add metadata impl
+* Merge pull request `#5 <https://github.com/wep21/rosbag2_storage_mcap/issues/5>`_ from wep21/mcap-storage-impl
+  feat: mcap storage impl
+* chore: update cmake minimum version
+* chore: install mcap header
+* chore: include mcap header
+* fix: move fetch content into rosbag2 storage mcap
+* Merge pull request `#3 <https://github.com/wep21/rosbag2_storage_mcap/issues/3>`_ from ros-tooling/emersonknapp/mcap_plugin_skeleton
+  Add mcap storage plugin skeleton and CI
+* Add rosbag2_storage_mcap skeleton
+* Contributors: Daisuke Nishimatsu, Emerson Knapp, Jacob Bandes-Storch, John Hurliman, wep21