diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 000000000..6297226e8 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,59 @@ +# Ignore everything + + +# Allow files and directories +# Example : +# !**/ArmoniK.Api.Common + +# Ignore unnecessary files inside allowed directories +# This should go after the allowed directories +**/*~ +**/*.log +**/*.obj +**/obj +**/bin +**/*.o +**/*.a +**/*.so +**/*.lib +**/build* +**/.build* +**/gen +**/install* +**/.git +**/.github +**/.vs +**/.vscode +**/Api +**/.docs +**/out +**/grpc +**/win64 +**/tools/tools +**/.classpath +**/.dockerignore +**/Dockerfile +**/.env +**/.gitignore +**/.project +**/.settings +**/.toolstarget +**/.idea +**/*.*proj.user +**/*.dbmdl +**/*.dump +**/*.jfm +**/azds.yaml +**/charts +**/docker-compose* +docker-compose +**/node_modules +**/secrets.dev.yaml +**/values.dev.yaml +LICENSE +README.md +**/publish +terraform +tmp + + diff --git a/.docs/content/1.guide/5.cpp.md b/.docs/content/1.guide/5.cpp.md new file mode 100644 index 000000000..75c644539 --- /dev/null +++ b/.docs/content/1.guide/5.cpp.md @@ -0,0 +1,121 @@ +# Compilation steps for cpp API + +## Compilation of ArmoniK.Api.cpp Client and Server on Linux using Docker + +In order to compile the client and server on Linux, we use a Docker image to set up the necessary environment and dependencies. This ensures a consistent and clean environment for compilation. + +### Prerequisites Linux + +1. Install Docker on your Linux system. Follow the instructions on the [official Docker documentation](https://docs.docker.com/engine/install/). +2. Clone the repository containing the source code and the necessary scripts. + +### Compilation Steps for Linux + +1. Open a terminal in the root directory of the cloned repository. +2. Run the `compile.sh` script: +This script compile the cpp project on linux systems. + +```bash [bash] +cd packages/cpp/tools +./compile.sh +``` + +The `compile.sh` script does the following: + +- Sets the image tag for the Docker image. +- Determines the absolute paths of the necessary directories (working, proto, build, and install directories). +- Checks if the Docker image exists. If not, it builds the Docker image using the Dockerfile.ubuntu file. +- Compiles the project source using the Docker image. +Once the compilation is complete, the compiled binaries will be located in the install directory. + +Now you have successfully compiled the client and server on Linux using Docker. + +### Compiling the Client and Server on Windows + +This guide explains how to compile the Armonik API client and server on Windows + +### Prerequisites Windows + +Before getting started, make sure you have the following tools and packages installed on your machine: + +- PowerShell +- Visual Studio 2022 +- Git + +Before getting started, you will need PowerShell and be inform that the script will install localy in the folder tools/win64 all prerequisites excepting Visual Studio 2022 and CMake plugins : + +- Chocolatey package manager +- Grpc 1.54.0 built from source +- CMake +- NASM + +### Compilation Steps for windows + +Follow these steps to compile the Armonik API client and server: + +From a PowerShell, go to the folder package/cpp/tools + +```powershell [PowerShell] +cd packages\cpp\tools +``` + +This will install the required dependencies and compile the Armonik API client and server. + +Wait for the script to complete. This may take some time, depending on the speed of your machine and the size of the project. + +Once the script has completed, you should see the compiled output in the install directory. From the root folder of repository ArmoniK.API + +```powershell [PowerShell] +cd packages\cpp\tools\win64 +``` + +### Troubleshooting + +If you encounter any issues during the compilation process, try the following troubleshooting steps: + +- Make sure you have all the prerequisites installed correctly. +- Check that you are running PowerShell as an administrator. + +### Conclusion + +Compiling the Armonik API client and server on Windows can be a complex process. + +By following the steps outlined in this guide, you should be able to compile the project successfully and start using the Armonik API on Windows. + +## Compilation of the Worker ArmoniK.Api.cpp Image for Deployment in Armonik Infrastructure + +The worker image is a Docker image that is built specifically to be deployed in the Armonik infrastructure. This image contains the necessary dependencies and configurations for the worker to function correctly. + +### Prerequisites + +1. Install Docker on your Linux system. Follow the instructions on the [official Docker documentation](https://docs.docker.com/engine/install/). +2. Clone the repository containing the source code and the necessary scripts. + +### Compilation Steps + +1. Open a terminal in the root directory of the cloned repository. +2. Run the `build-worker.sh` script: + + ```bash + cd packages/cpp/tools + ./build-worker.sh + ``` + + The build-worker.sh script does the following: + + - Sets the image tag for the Docker image. + - Determines the absolute paths of the necessary directories (script, working, and root directories). + - Changes to the root directory where the Protos are located. + - Builds the worker Docker image using the Dockerfile.worker file. + + Now you should have the final image + +3. Once the worker image has been built, you can use the following command to list all the Docker images available on your system: + + ```bash + docker images | grep armonik-api-cpp + ``` + +The worker image should be listed with the specified image tag (e.g., armonik-api-cpp:v0.1). + +Now you have successfully compiled the worker image for deployment in the Armonik infrastructure. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e37db9f92..e467b028b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -76,6 +76,50 @@ jobs: exit 1 fi + format-cpp: + name: Format C++ + runs-on: ubuntu-latest + timeout-minutes: 15 + + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + ref: ${{ github.ref }} + fetch-depth: 0 + + - name: Run clang-format + run: | + sudo apt-get update + sudo apt-get install -y clang-format + + - name: Check Diff + id: check-diff + run: | + cd packages/cpp + git ls-files *.{c,h,hpp,cpp,cc} | xargs clang-format -style=file:.clang-format -i + DIFF="$(git diff --name-only)" + + if [ -z "$DIFF" ]; then + echo "OK: Format is clean" + else + echo "Error: Format was not clean" + echo "List of files:" + echo "$DIFF" + git diff + exit 1 + fi + + - name: Generate patch + if: ${{ failure() && steps.check-diff.conclusion == 'failure' }} + run: | + git diff > patch-cpp.diff + - uses: actions/upload-artifact@v3 + if: ${{ failure() && steps.check-diff.conclusion == 'failure' }} + with: + name: patch-cpp + path: ./patch-cpp.diff + format-protobuf: name: Format Protobuf runs-on: ubuntu-latest @@ -189,4 +233,19 @@ jobs: uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3 with: name: code-coverage-report-xml - path: packages/python/coverage.xml \ No newline at end of file + path: packages/python/coverage.xml + + build-cpp-packages: + name: Build C++ Packages + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + ref: ${{ github.ref }} + + - name: Build the package + run: | + cd packages/cpp/tools/ + ./compile.sh diff --git a/.gitignore b/.gitignore index 04311842e..0deaa71df 100644 --- a/.gitignore +++ b/.gitignore @@ -377,6 +377,7 @@ dist !packages/common !packages/csharp +!packages/cpp !packages/python !packages/angular !packages/web diff --git a/packages/cpp/.clang-format b/packages/cpp/.clang-format new file mode 100644 index 000000000..067b805b5 --- /dev/null +++ b/packages/cpp/.clang-format @@ -0,0 +1,192 @@ +--- +Language: Cpp +# BasedOnStyle: LLVM +AccessModifierOffset: -2 +AlignAfterOpenBracket: Align +AlignArrayOfStructures: None +AlignConsecutiveMacros: None +AlignConsecutiveAssignments: None +AlignConsecutiveBitFields: None +AlignConsecutiveDeclarations: None +AlignEscapedNewlines: Right +AlignOperands: Align +AlignTrailingComments: true +AllowAllArgumentsOnNextLine: true +AllowAllParametersOfDeclarationOnNextLine: true +AllowShortEnumsOnASingleLine: true +AllowShortBlocksOnASingleLine: Never +AllowShortCaseLabelsOnASingleLine: false +AllowShortFunctionsOnASingleLine: All +AllowShortLambdasOnASingleLine: All +AllowShortIfStatementsOnASingleLine: Never +AllowShortLoopsOnASingleLine: false +AlwaysBreakAfterDefinitionReturnType: None +AlwaysBreakAfterReturnType: None +AlwaysBreakBeforeMultilineStrings: false +AlwaysBreakTemplateDeclarations: MultiLine +AttributeMacros: + - __capability +BinPackArguments: true +BinPackParameters: true +BraceWrapping: + AfterCaseLabel: false + AfterClass: false + AfterControlStatement: Never + AfterEnum: false + AfterFunction: false + AfterNamespace: false + AfterObjCDeclaration: false + AfterStruct: false + AfterUnion: false + AfterExternBlock: false + BeforeCatch: false + BeforeElse: false + BeforeLambdaBody: false + BeforeWhile: false + IndentBraces: false + SplitEmptyFunction: true + SplitEmptyRecord: true + SplitEmptyNamespace: true +BreakBeforeBinaryOperators: None +BreakBeforeConceptDeclarations: true +BreakBeforeBraces: Attach +BreakBeforeInheritanceComma: false +BreakInheritanceList: BeforeColon +BreakBeforeTernaryOperators: true +BreakConstructorInitializersBeforeComma: false +BreakConstructorInitializers: BeforeColon +BreakAfterJavaFieldAnnotations: false +BreakStringLiterals: true +ColumnLimit: 120 +CommentPragmas: '^ IWYU pragma:' +QualifierAlignment: Leave +CompactNamespaces: false +ConstructorInitializerIndentWidth: 4 +ContinuationIndentWidth: 4 +Cpp11BracedListStyle: true +DeriveLineEnding: true +DerivePointerAlignment: false +DisableFormat: false +EmptyLineAfterAccessModifier: Never +EmptyLineBeforeAccessModifier: LogicalBlock +ExperimentalAutoDetectBinPacking: false +PackConstructorInitializers: BinPack +BasedOnStyle: '' +ConstructorInitializerAllOnOneLineOrOnePerLine: false +AllowAllConstructorInitializersOnNextLine: true +FixNamespaceComments: true +ForEachMacros: + - foreach + - Q_FOREACH + - BOOST_FOREACH +IfMacros: + - KJ_IF_MAYBE +IncludeBlocks: Preserve +IncludeCategories: + - Regex: '^"(llvm|llvm-c|clang|clang-c)/' + Priority: 2 + SortPriority: 0 + CaseSensitive: false + - Regex: '^(<|"(gtest|gmock|isl|json)/)' + Priority: 3 + SortPriority: 0 + CaseSensitive: false + - Regex: '.*' + Priority: 1 + SortPriority: 0 + CaseSensitive: false +IncludeIsMainRegex: '(Test)?$' +IncludeIsMainSourceRegex: '' +IndentAccessModifiers: false +IndentCaseLabels: false +IndentCaseBlocks: false +IndentGotoLabels: true +IndentPPDirectives: None +IndentExternBlock: AfterExternBlock +IndentRequires: false +IndentWidth: 2 +IndentWrappedFunctionNames: false +InsertTrailingCommas: None +JavaScriptQuotes: Leave +JavaScriptWrapImports: true +KeepEmptyLinesAtTheStartOfBlocks: true +LambdaBodyIndentation: Signature +MacroBlockBegin: '' +MacroBlockEnd: '' +MaxEmptyLinesToKeep: 1 +NamespaceIndentation: None +ObjCBinPackProtocolList: Auto +ObjCBlockIndentWidth: 2 +ObjCBreakBeforeNestedBlockParam: true +ObjCSpaceAfterProperty: false +ObjCSpaceBeforeProtocolList: true +PenaltyBreakAssignment: 2 +PenaltyBreakBeforeFirstCallParameter: 19 +PenaltyBreakComment: 300 +PenaltyBreakFirstLessLess: 120 +PenaltyBreakOpenParenthesis: 0 +PenaltyBreakString: 1000 +PenaltyBreakTemplateDeclaration: 10 +PenaltyExcessCharacter: 1000000 +PenaltyReturnTypeOnItsOwnLine: 60 +PenaltyIndentedWhitespace: 0 +PointerAlignment: Right +PPIndentWidth: -1 +ReferenceAlignment: Pointer +ReflowComments: true +RemoveBracesLLVM: false +SeparateDefinitionBlocks: Leave +ShortNamespaceLines: 1 +SortIncludes: CaseSensitive +SortJavaStaticImport: Before +SortUsingDeclarations: true +SpaceAfterCStyleCast: false +SpaceAfterLogicalNot: false +SpaceAfterTemplateKeyword: true +SpaceBeforeAssignmentOperators: true +SpaceBeforeCaseColon: false +SpaceBeforeCpp11BracedList: false +SpaceBeforeCtorInitializerColon: true +SpaceBeforeInheritanceColon: true +SpaceBeforeParens: ControlStatements +SpaceBeforeParensOptions: + AfterControlStatements: true + AfterForeachMacros: true + AfterFunctionDefinitionName: false + AfterFunctionDeclarationName: false + AfterIfMacros: true + AfterOverloadedOperator: false + BeforeNonEmptyParentheses: false +SpaceAroundPointerQualifiers: Default +SpaceBeforeRangeBasedForLoopColon: true +SpaceInEmptyBlock: false +SpaceInEmptyParentheses: false +SpacesBeforeTrailingComments: 1 +SpacesInAngles: Never +SpacesInConditionalStatement: false +SpacesInContainerLiterals: true +SpacesInCStyleCastParentheses: false +SpacesInLineCommentPrefix: + Minimum: 1 + Maximum: -1 +SpacesInParentheses: false +SpacesInSquareBrackets: false +SpaceBeforeSquareBrackets: false +BitFieldColonSpacing: Both +Standard: Latest +StatementAttributeLikeMacros: + - Q_EMIT +StatementMacros: + - Q_UNUSED + - QT_REQUIRE_VERSION +TabWidth: 8 +UseCRLF: false +UseTab: Never +WhitespaceSensitiveMacros: + - STRINGIZE + - PP_STRINGIZE + - BOOST_PP_STRINGIZE + - NS_SWIFT_NAME + - CF_SWIFT_NAME +... + diff --git a/packages/cpp/.gitignore b/packages/cpp/.gitignore new file mode 100644 index 000000000..486622079 --- /dev/null +++ b/packages/cpp/.gitignore @@ -0,0 +1,11 @@ +pkg/ +source/Protos +build/ +*.egg-info +**/*build +**/*.obj +**/*.o +**/install +**/win64 +**/tools/tools +**/tools/grpc diff --git a/packages/cpp/ArmoniK.Api.Client/CMakeLists.txt b/packages/cpp/ArmoniK.Api.Client/CMakeLists.txt new file mode 100644 index 000000000..bdeddbc53 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Client/CMakeLists.txt @@ -0,0 +1,137 @@ +set(PROJECT_NAME ArmoniK.Api.Client) +set(NAMESPACE ArmoniK::Api::Client) +set(version 0.1.0) + +set(PROTO_FILES + "submitter_service.proto" + "applications_service.proto" + "sessions_service.proto" + # "tasks_service.proto" Name conflict with grpc generated files ListTasks and ListTasksRaw (grpc add the suffix RAW to all interfaces) + "results_service.proto" + "auth_service.proto" + "partitions_service.proto" + "events_service.proto" + "versions_service.proto" + "auth_common.proto" + "sessions_common.proto" + "submitter_common.proto" + "tasks_common.proto" + "results_common.proto" + "partitions_common.proto" + "events_common.proto" + "versions_common.proto" + "applications_common.proto") + +set(PROTO_DEPS + "session_status.proto" + "sort_direction.proto" + "objects.proto" + "result_status.proto" + "task_status.proto") + +foreach(file ${PROTO_FILES} ${PROTO_DEPS}) + configure_file("${PROTO_FILES_DIR}/${file}" "${BUILD_DIR}/${PROJECT_NAME}/${file}" COPYONLY) +endforeach() +list(TRANSFORM PROTO_FILES PREPEND ${BUILD_DIR}/${PROJECT_NAME}/) + +find_package(Protobuf REQUIRED) +find_package(gRPC CONFIG REQUIRED) +find_package(Threads) + +SET(SOURCES_FILES_DIR "${CMAKE_CURRENT_SOURCE_DIR}/source") +SET(HEADER_FILES_DIR "${CMAKE_CURRENT_SOURCE_DIR}/header") + +FILE(GLOB_RECURSE SRC_CLIENT_FILES ${SOURCES_FILES_DIR}/*.cpp) +FILE(GLOB_RECURSE HEADER_CLIENT_FILES ${HEADER_FILES_DIR}/*.h) + + +file(MAKE_DIRECTORY ${BUILD_DIR}/${PROJECT_NAME}) + +add_library(${PROJECT_NAME} STATIC ${PROTO_FILES} ${SRC_CLIENT_FILES} ${HEADER_CLIENT_FILES}) + +target_link_libraries(${PROJECT_NAME} PUBLIC protobuf::libprotobuf gRPC::grpc++_unsecure ArmoniK.Api.Common) + +target_include_directories(${PROJECT_NAME} + PUBLIC + "$" + "$" +) + +set(PROTO_BINARY_DIR "${BUILD_DIR}/${PROJECT_NAME}/") +set(PROTO_IMPORT_DIRS "${PROTO_FILES_DIR}") + +protobuf_generate( + TARGET ${PROJECT_NAME} + OUT_VAR PROTO_GENERATED_FILES + IMPORT_DIRS ${BUILD_DIR}/${PROJECT_NAME} + PROTOC_OUT_DIR "${PROTO_BINARY_DIR}") +set_source_files_properties(${PROTO_GENERATED_FILES} PROPERTIES SKIP_UNITY_BUILD_INCLUSION on) + +get_target_property(grpc_cpp_plugin_location gRPC::grpc_cpp_plugin LOCATION) + +protobuf_generate( + TARGET ${PROJECT_NAME} + OUT_VAR PROTO_GENERATED_FILES + LANGUAGE grpc + GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc + PLUGIN "protoc-gen-grpc=${grpc_cpp_plugin_location}" + # PLUGIN_OPTIONS "generate_mock_code=true" + IMPORT_DIRS ${BUILD_DIR}/${PROJECT_NAME} + PROTOC_OUT_DIR "${PROTO_BINARY_DIR}") + +set_source_files_properties(${PROTO_GENERATED_FILES} PROPERTIES SKIP_UNITY_BUILD_INCLUSION on) + +target_include_directories(${PROJECT_NAME} + PUBLIC + "$" + "$") + +target_include_directories(${PROJECT_NAME} + PUBLIC + "$" + "$") + + + + +set_property(TARGET ${PROJECT_NAME} PROPERTY VERSION ${version}) +set_property(TARGET ${PROJECT_NAME} PROPERTY SOVERSION 0) +set_property(TARGET ${PROJECT_NAME} PROPERTY + INTERFACE_${PROJECT_NAME}_MAJOR_VERSION 0) +set_property(TARGET ${PROJECT_NAME} APPEND PROPERTY + COMPATIBLE_INTERFACE_STRING ${PROJECT_NAME}_MAJOR_VERSION +) + +# generate the version file for the config file +write_basic_package_version_file( + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}ConfigVersion.cmake" + VERSION "${version}" + COMPATIBILITY AnyNewerVersion +) + +install(TARGETS ${PROJECT_NAME} + EXPORT ${PROJECT_NAME}Targets + LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} + ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} + RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} + INCLUDES DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} + ) + +FILE(GLOB INCLUDE_FILES ${PROTO_BINARY_DIR}/*.h) +install(FILES ${INCLUDE_FILES} DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) + +install(EXPORT ${PROJECT_NAME}Targets + FILE ${PROJECT_NAME}Targets.cmake + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) + +configure_package_config_file(${CMAKE_CURRENT_SOURCE_DIR}/Config.cmake.in + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}Config.cmake" + INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) + +install(FILES + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}Config.cmake" + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}ConfigVersion.cmake" + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) diff --git a/packages/cpp/ArmoniK.Api.Client/Config.cmake.in b/packages/cpp/ArmoniK.Api.Client/Config.cmake.in new file mode 100644 index 000000000..751f562ff --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Client/Config.cmake.in @@ -0,0 +1,5 @@ +@PACKAGE_INIT@ + +include("${CMAKE_INSTALL_PREFIX}/lib/cmake/Armonik.Api.Client/Armonik.Api.ClientTargets.cmake") + +check_required_components(Armonik.Api.Client) \ No newline at end of file diff --git a/packages/cpp/ArmoniK.Api.Client/header/serializer/ArgsSerializer.h b/packages/cpp/ArmoniK.Api.Client/header/serializer/ArgsSerializer.h new file mode 100644 index 000000000..6f70f09be --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Client/header/serializer/ArgsSerializer.h @@ -0,0 +1 @@ +#pragma once diff --git a/packages/cpp/ArmoniK.Api.Client/header/serializer/Serializer.h b/packages/cpp/ArmoniK.Api.Client/header/serializer/Serializer.h new file mode 100644 index 000000000..6f70f09be --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Client/header/serializer/Serializer.h @@ -0,0 +1 @@ +#pragma once diff --git a/packages/cpp/ArmoniK.Api.Client/header/submitter/SubmitterClient.h b/packages/cpp/ArmoniK.Api.Client/header/submitter/SubmitterClient.h new file mode 100644 index 000000000..2226516ea --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Client/header/submitter/SubmitterClient.h @@ -0,0 +1,100 @@ +/** + * @file submitter_client_ext.h + * @brief This file contains the SubmitterClient class definition. + */ + +#pragma once +#include +#include + +#include "submitter_common.pb.h" +#include "submitter_service.grpc.pb.h" + +/** + * @brief Data structure for task payload + * @param keys The expected output keys + * @param payload The task payload + * @param dependencies The dependencies of the task + * + */ +struct payload_data { + std::string keys; + std::vector payload; + std::vector dependencies; +}; + +/** + * @brief The SubmitterClientExt class provides methods to create and manage task submissions. + */ +class SubmitterClient { +private: + grpc::ClientContext context_; + std::unique_ptr stub_; + +public: + /** + * @brief Construct a new Submitter Client object + * + */ + SubmitterClient(std::unique_ptr stub); + + /** + * @brief Creates a new session with the control plane. + * @param default_task_options The default task options. + * @param partition_ids The partition ids. + */ + std::string create_session(armonik::api::grpc::v1::TaskOptions default_task_options, + const std::vector &partition_ids); + + /** + * @brief Converts task requests into a vector of future large task request objects. + * @param task_requests The vector of task requests. + * @param session_id The session ID. + * @param task_options The task options. + * @param chunk_max_size The maximum chunk size. + * @return A vector of future large task request objects. + */ + static std::vector>> + to_request_stream(const std::vector &task_requests, std::string session_id, + armonik::api::grpc::v1::TaskOptions task_options, size_t chunk_max_size); + + /** + * @brief Creates a large task request object with specified parameters. + * @param task_request The task request. + * @param is_last Indicates if this is the last task request in the stream. + * @param chunk_max_size The maximum chunk size. + * @return A future large task request object. + */ + static std::future> + task_chunk_stream(const armonik::api::grpc::v1::TaskRequest &task_request, bool is_last, size_t chunk_max_size); + + /** + * @brief Creates tasks asynchronously with the specified options and requests. + * @param session_id The session ID. + * @param task_options The task options. + * @param task_requests The vector of task requests. + * @return A future create task reply object. + */ + std::future + create_tasks_async(std::string session_id, armonik::api::grpc::v1::TaskOptions task_options, + const std::vector &task_requests); + + /** + * @brief Submits tasks with dependencies to the session context. + * @param session_id The session id. + * @param task_options The task options. + * @param payloads_with_dependencies A vector of tuples containing the payload, its data, and its dependencies. + * @param max_retries The maximum number of retries for submitting tasks. + * @return A vector of submitted task IDs. + */ + std::tuple, std::vector> + submit_tasks_with_dependencies(std::string session_id, armonik::api::grpc::v1::TaskOptions task_options, + const std::vector &payloads_with_dependencies, int max_retries); + + /** + * @brief Get result without streaming. + * @param result_request The vector of result requests. + * @return A vector containing the data associated to the result + */ + std::future> get_result_async(const armonik::api::grpc::v1::ResultRequest &result_request); +}; diff --git a/packages/cpp/ArmoniK.Api.Client/source/submitter/SubmitterClient.cpp b/packages/cpp/ArmoniK.Api.Client/source/submitter/SubmitterClient.cpp new file mode 100644 index 000000000..2e3f74b4f --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Client/source/submitter/SubmitterClient.cpp @@ -0,0 +1,337 @@ +/** + * @file SubmitterClientExt.h + * @brief Header file for the SubmitterClientExt class. + */ +#include "submitter/SubmitterClient.h" + +#include +#include +#include + +#include "objects.grpc.pb.h" +#include "submitter_common.pb.h" +#include "submitter_service.grpc.pb.h" + +using armonik::api::grpc::v1::ResultRequest; +using armonik::api::grpc::v1::TaskOptions; +using armonik::api::grpc::v1::TaskRequest; +using armonik::api::grpc::v1::submitter::CreateSessionReply; +using armonik::api::grpc::v1::submitter::CreateSessionRequest; +using armonik::api::grpc::v1::submitter::Submitter; +using grpc::Channel; +using grpc::ClientContext; +using grpc::Status; +using namespace armonik::api::grpc::v1::submitter; + +/** + * @brief Construct a new Submitter Client:: Submitter Client object + * + * @param stub the gRPC client stub + */ +SubmitterClient::SubmitterClient(std::unique_ptr stub) { stub_ = std::move(stub); } + +/** + * @brief Create a new session. + * @param partition_ids The partitions ids. + * @param default_task_options The default task options. + */ +std::string SubmitterClient::create_session(TaskOptions default_task_options, + const std::vector &partition_ids = {}) { + CreateSessionRequest request; + *request.mutable_default_task_option() = std::move(default_task_options); + for (const auto &partition_id : partition_ids) { + request.add_partition_ids(partition_id); + } + CreateSessionReply reply; + + Status status = stub_->CreateSession(&context_, request, &reply); + if (!status.ok()) { + std::stringstream message; + message << "Error: " << status.error_code() << ": " << status.error_message() + << ". details : " << status.error_details() << std::endl; + std::cout << "CreateSession rpc failed: " << std::endl; + throw std::runtime_error(message.str().c_str()); + } + return reply.session_id(); +} + +/** + * @brief Convert task_requests to request_stream. + * + * @param task_requests A vector of TaskRequest objects. + * @param session_id The session ID. + * @param task_options The TaskOptions object. + * @param chunk_max_size Maximum chunk size. + * @return A vector of futures containing CreateLargeTaskRequest objects. + */ +std::vector>> +SubmitterClient::to_request_stream(const std::vector &task_requests, std::string session_id, + TaskOptions task_options, const size_t chunk_max_size) { + std::vector>> async_chunk_payload_tasks; + async_chunk_payload_tasks.push_back( + std::async([session_id = std::move(session_id), task_options = std::move(task_options)]() mutable { + CreateLargeTaskRequest_InitRequest create_large_task_request_init; + create_large_task_request_init.set_session_id(std::move(session_id)); + *create_large_task_request_init.mutable_task_options() = std::move(task_options); + + CreateLargeTaskRequest create_large_task_request; + *create_large_task_request.mutable_init_request() = std::move(create_large_task_request_init); + + return std::vector{std::move(create_large_task_request)}; + })); + + for (auto task_request = task_requests.begin(); task_request != task_requests.end(); ++task_request) { + const bool is_last = task_request == task_requests.end() - 1; + + async_chunk_payload_tasks.push_back(task_chunk_stream(*task_request, is_last, chunk_max_size)); + } + + return async_chunk_payload_tasks; +} + +/** + * @brief Create a task_chunk_stream. + * + * @param task_request The TaskRequest object. + * @param is_last A boolean indicating if this is the last request. + * @param chunk_max_size Maximum chunk size. + * @return A future containing a vector of CreateLargeTaskRequest objects. + */ +std::future> +SubmitterClient::task_chunk_stream(const TaskRequest &task_request, bool is_last, size_t chunk_max_size) { + return std::async(std::launch::async, [&task_request, chunk_max_size, is_last]() { + std::vector requests; + armonik::api::grpc::v1::InitTaskRequest header_task_request; + armonik::api::grpc::v1::TaskRequestHeader header; + + header.mutable_data_dependencies()->Add(task_request.data_dependencies().begin(), + task_request.data_dependencies().end()); + header.mutable_expected_output_keys()->Add(task_request.expected_output_keys().begin(), + task_request.expected_output_keys().end()); + *header_task_request.mutable_header() = std::move(header); + + CreateLargeTaskRequest create_init_task_request; + *create_init_task_request.mutable_init_task() = std::move(header_task_request); + + // Add init task request + requests.push_back(std::move(create_init_task_request)); + + if (task_request.payload().empty()) { + CreateLargeTaskRequest empty_task_request; + + armonik::api::grpc::v1::DataChunk task_payload; + *task_payload.mutable_data() = {}; + *empty_task_request.mutable_task_payload() = std::move(task_payload); + requests.push_back(std::move(empty_task_request)); + } + + size_t start = 0; + + while (start < task_request.payload().size()) { + + size_t chunk_size = std::min(chunk_max_size, task_request.payload().size() - start); + + CreateLargeTaskRequest chunk_task_request; + + armonik::api::grpc::v1::DataChunk task_payload; + + *task_payload.mutable_data() = task_request.payload().substr(start, chunk_size); + *chunk_task_request.mutable_task_payload() = std::move(task_payload); + + requests.push_back(std::move(chunk_task_request)); + + start += chunk_size; + } + + CreateLargeTaskRequest complete_task_request; + armonik::api::grpc::v1::DataChunk end_payload; + + end_payload.set_data_complete(true); + *complete_task_request.mutable_task_payload() = std::move(end_payload); + requests.push_back(std::move(complete_task_request)); + + if (is_last) { + CreateLargeTaskRequest last_task_request; + armonik::api::grpc::v1::InitTaskRequest init_task_request; + + init_task_request.set_last_task(true); + *last_task_request.mutable_init_task() = std::move(init_task_request); + + requests.push_back(std::move(last_task_request)); + } + + return requests; + }); +} + +/** + * @brief Asynchronously create tasks. + * + * @param session_id The session ID. + * @param task_options The TaskOptions object. + * @param task_requests A vector of TaskRequest objects. + * @return A future containing a CreateTaskReply object. + */ +std::future SubmitterClient::create_tasks_async(std::string session_id, TaskOptions task_options, + const std::vector &task_requests) { + return std::async(std::launch::async, [this, task_requests, session_id = std::move(session_id), + task_options = std::move(task_options)]() mutable { + armonik::api::grpc::v1::Configuration config_response; + ClientContext context_configuration; + + const auto config_status = + stub_->GetServiceConfiguration(&context_configuration, armonik::api::grpc::v1::Empty(), &config_response); + size_t chunk = 0; + if (config_status.ok()) { + chunk = config_response.data_chunk_max_size(); + } else { + throw std::runtime_error("Fail to get service configuration"); + } + + CreateTaskReply reply{}; + + reply.set_allocated_creation_status_list(new CreateTaskReply_CreationStatusList()); + ClientContext context_client_writer; + std::unique_ptr stream(stub_->CreateLargeTasks(&context_client_writer, &reply)); + + // task_chunk_stream(task_requests, ) + std::vector> async_task_requests; + std::vector> create_large_task_requests; + auto create_task_request_async = + to_request_stream(task_requests, std::move(session_id), std::move(task_options), chunk); + + for (auto &f : create_task_request_async) { + for (auto &create_large_task_request : f.get()) { + stream->Write(create_large_task_request); + } + } + + stream->WritesDone(); + Status status = stream->Finish(); + if (!status.ok()) { + std::stringstream message; + message << "Error: " << status.error_code() << ": " << status.error_message() + << ". details : " << status.error_details() << std::endl; + throw std::runtime_error(message.str().c_str()); + } + + return reply; + }); +} + +/** + * @brief Submit tasks with dependencies. + * + * @param session_id The session ID. + * @param task_options The task options + * @param payloads_with_dependencies A vector of tuples containing task payload + * and its dependencies. + * @param max_retries Maximum number of retries. + * @return A vector of task IDs. + */ +std::tuple, std::vector> +SubmitterClient::submit_tasks_with_dependencies(std::string session_id, TaskOptions task_options, + const std::vector &payloads_with_dependencies, + int max_retries = 5) { + std::vector task_ids; + std::vector failed_task_ids; + std::vector requests; + for (auto &payload : payloads_with_dependencies) { + TaskRequest request; + auto &bytes = payload.payload; + + request.add_expected_output_keys(payload.keys); + + *request.mutable_payload() = std::string(bytes.begin(), bytes.end()); + + *request.mutable_data_dependencies() = {payload.dependencies.begin(), payload.dependencies.end()}; + + requests.push_back(std::move(request)); + } + + auto tasks_async = create_tasks_async(std::move(session_id), std::move(task_options), requests); + + const CreateTaskReply createTaskReply = tasks_async.get(); + + switch (createTaskReply.Response_case()) { + case CreateTaskReply::RESPONSE_NOT_SET: + throw std::runtime_error("Issue with Server !"); + case CreateTaskReply::kCreationStatusList: { + auto task_reply_creation_statuses = createTaskReply.creation_status_list().creation_statuses(); + + for (auto &task_created : task_reply_creation_statuses) { + if (task_created.Status_case() == CreateTaskReply_CreationStatus::kTaskInfo) { + task_ids.push_back(task_created.task_info().task_id()); + } else { + failed_task_ids.push_back(task_created.task_info().task_id()); + } + } + break; + } + + case CreateTaskReply::kError: + std::stringstream message; + message << "Error while creating tasks ! : Error Message : " << createTaskReply.error() << std::endl; + throw std::runtime_error(message.str().c_str()); + } + return std::make_tuple(std::move(task_ids), std::move(failed_task_ids)); +} + +/** + * @brief Asynchronously gets tasks. + * + * @param result_request A vector of ResultRequest objects. + * @return A future containing data result. + */ +std::future> SubmitterClient::get_result_async(const ResultRequest &result_request) { + return std::async(std::launch::async, [this, &result_request]() { + ResultReply result_writer; + ClientContext context_configuration; + ClientContext context_result; + armonik::api::grpc::v1::Configuration config_response; + + const auto config_status = + stub_->GetServiceConfiguration(&context_configuration, armonik::api::grpc::v1::Empty(), &config_response); + + size_t size = 0; + if (config_status.ok()) { + size = config_response.data_chunk_max_size(); + } else { + throw std::runtime_error("Fail to get service configuration"); + } + + auto streamingCall = stub_->TryGetResultStream(&context_result, result_request); + + if (!streamingCall) { + throw std::runtime_error("Fail to get result"); + } + + std::vector result_data; + for (size_t count = 0; count < size; count++) { + streamingCall->WaitForInitialMetadata(); + streamingCall->Read(&result_writer); + std::string dataString; + switch (result_writer.type_case()) { + case ResultReply::kResult: + dataString = result_writer.result().data(); + result_data.resize(dataString.length()); + std::memcpy(result_data.data(), dataString.data(), dataString.size()); + + break; + case ResultReply::kError: + throw std::runtime_error("Error in task "); + + case ResultReply::kNotCompletedTask: + throw std::runtime_error("Task not completed"); + + case ResultReply::TYPE_NOT_SET: + throw std::runtime_error("Issue with the Server"); + + default: + throw std::runtime_error("Unknown return type !"); + } + } + + return result_data; + }); +} diff --git a/packages/cpp/ArmoniK.Api.Common/CMakeLists.txt b/packages/cpp/ArmoniK.Api.Common/CMakeLists.txt new file mode 100644 index 000000000..d7e2f9ed6 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/CMakeLists.txt @@ -0,0 +1,108 @@ +# make cache variables for install destinations +include(GNUInstallDirs) +include(CMakePackageConfigHelpers) + +set(PROJECT_NAME ArmoniK.Api.Common) +set(NAMESPACE ArmoniK::Api::Common) + +set(PROTO_FILES + "objects.proto" + "result_status.proto" + "task_status.proto" + "session_status.proto" + "sort_direction.proto") + +foreach(file ${PROTO_FILES}) + configure_file("${PROTO_FILES_DIR}/${file}" "${BUILD_DIR}/${PROJECT_NAME}/${file}" COPYONLY) +endforeach() +list(TRANSFORM PROTO_FILES PREPEND ${BUILD_DIR}/${PROJECT_NAME}/) + +set(CMAKE_FIND_DEBUG_MODE FALSE) +# Trouver les packages requis +find_package(Protobuf REQUIRED) +find_package(gRPC CONFIG REQUIRED) +find_package(Threads) +set(CMAKE_FIND_DEBUG_MODE FALSE) + +SET(SOURCES_FILES_DIR "${CMAKE_CURRENT_SOURCE_DIR}/source") +SET(HEADER_FILES_DIR "${CMAKE_CURRENT_SOURCE_DIR}/header") + +FILE(GLOB_RECURSE SRC_CLIENT_FILES ${SOURCES_FILES_DIR}/*.cpp) +FILE(GLOB_RECURSE HEADER_CLIENT_FILES ${HEADER_FILES_DIR}/*.h) + +file(MAKE_DIRECTORY ${BUILD_DIR}/${PROJECT_NAME}) + +add_library(${PROJECT_NAME} STATIC ${PROTO_FILES} ${SRC_CLIENT_FILES} ${HEADER_CLIENT_FILES}) + +target_link_libraries(${PROJECT_NAME} PUBLIC protobuf::libprotobuf gRPC::grpc++_unsecure) +set(PROTO_BINARY_DIR "${BUILD_DIR}/${PROJECT_NAME}") +set(PROTO_IMPORT_DIRS "${PROTO_FILES_DIR}") + +protobuf_generate( + TARGET ${PROJECT_NAME} + OUT_VAR PROTO_GENERATED_FILES + IMPORT_DIRS ${BUILD_DIR}/${PROJECT_NAME} + PROTOC_OUT_DIR "${PROTO_BINARY_DIR}") +set_source_files_properties(${PROTO_GENERATED_FILES} PROPERTIES SKIP_UNITY_BUILD_INCLUSION on) + +get_target_property(grpc_cpp_plugin_location gRPC::grpc_cpp_plugin LOCATION) + +protobuf_generate( + TARGET ${PROJECT_NAME} + OUT_VAR PROTO_GENERATED_FILES + LANGUAGE grpc + GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc + PLUGIN "protoc-gen-grpc=${grpc_cpp_plugin_location}" + # PLUGIN_OPTIONS "generate_mock_code=true" + IMPORT_DIRS ${BUILD_DIR}/${PROJECT_NAME} + PROTOC_OUT_DIR "${PROTO_BINARY_DIR}") + +set_source_files_properties(${PROTO_BINARY_DIR} PROPERTIES SKIP_UNITY_BUILD_INCLUSION on) + +target_include_directories(${PROJECT_NAME} + PUBLIC + "$" + "$" + "$") + +set_property(TARGET ${PROJECT_NAME} PROPERTY VERSION ${version}) +set_property(TARGET ${PROJECT_NAME} PROPERTY SOVERSION 0) +set_property(TARGET ${PROJECT_NAME} PROPERTY + INTERFACE_${PROJECT_NAME}_MAJOR_VERSION 0) +set_property(TARGET ${PROJECT_NAME} APPEND PROPERTY + COMPATIBLE_INTERFACE_STRING ${PROJECT_NAME}_MAJOR_VERSION +) + +# generate the version file for the config file +write_basic_package_version_file( + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}ConfigVersion.cmake" + VERSION "${version}" + COMPATIBILITY AnyNewerVersion +) + +install(TARGETS ${PROJECT_NAME} + EXPORT ${PROJECT_NAME}Targets + LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} + ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} + RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} + INCLUDES DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} + ) + +FILE(GLOB INCLUDE_FILES ${PROTO_BINARY_DIR}/*.h) +install(FILES ${INCLUDE_FILES} DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) + +install(EXPORT ${PROJECT_NAME}Targets + FILE ${PROJECT_NAME}Targets.cmake + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) + +configure_package_config_file(${CMAKE_CURRENT_SOURCE_DIR}/Config.cmake.in + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}Config.cmake" + INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) + +install(FILES + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}Config.cmake" + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}ConfigVersion.cmake" + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) diff --git a/packages/cpp/ArmoniK.Api.Common/Config.cmake.in b/packages/cpp/ArmoniK.Api.Common/Config.cmake.in new file mode 100644 index 000000000..04cea6bdb --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/Config.cmake.in @@ -0,0 +1,5 @@ +@PACKAGE_INIT@ + +include("${CMAKE_INSTALL_PREFIX}/lib/cmake/Armonik.Api.Common/Armonik.Api.CommonTargets.cmake") + +check_required_components(Armonik.Api.Common) \ No newline at end of file diff --git a/packages/cpp/ArmoniK.Api.Common/header/options/ComputePlane.h b/packages/cpp/ArmoniK.Api.Common/header/options/ComputePlane.h new file mode 100644 index 000000000..43140e822 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/header/options/ComputePlane.h @@ -0,0 +1,60 @@ +#pragma once +#include + +/** + * @brief The armonik namespace contains classes and functions related to the Armonik API. + */ +namespace armonik::api::common::options { +/** + * @brief The ComputePlane class manages the communication addresses for workers and agents. + */ +class ComputePlane { +public: + /** + * @brief Constructs a ComputePlane object with the given configuration. + * @param configuration The IConfiguration object containing address information. + */ + ComputePlane(const utils::IConfiguration &configuration) { + set_worker_address(configuration.get("ComputePlane__WorkerChannel__Address")); + set_agent_address(configuration.get(std::string("ComputePlane__AgentChannel__Address"))); + } + + /** + * @brief Returns the server address. + * @return A reference to the server address string. + */ + std::string_view get_server_address() const { return worker_address_; } + + /** + * @brief Sets the worker address with the given socket address. + * @param socket_address The socket address to set for the worker. + */ + void set_worker_address(std::string socket_address) { + if (socket_address.find("unix:") != 0) { + socket_address.insert(0, "unix:"); + } + worker_address_ = std::move(socket_address); + } + + /** + * @brief Sets the agent address with the given agent address. + * @param agent_address The agent address to set for the agent. + */ + void set_agent_address(std::string agent_address) { + if (agent_address.find("unix:") != 0) { + agent_address.insert(0, "unix:"); + } + agent_address_ = std::move(agent_address); + } + + /** + * @brief Returns the agent address. + * @return A reference to the agent address string. + */ + std::string_view get_agent_address() const { return agent_address_; } + +private: + std::string worker_address_; ///< The worker address string. + std::string agent_address_; ///< The agent address string. +}; +}; // namespace armonik::api::common::options diff --git a/packages/cpp/ArmoniK.Api.Common/header/options/GrpcSocketType.h b/packages/cpp/ArmoniK.Api.Common/header/options/GrpcSocketType.h new file mode 100644 index 000000000..9542d5318 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/header/options/GrpcSocketType.h @@ -0,0 +1,16 @@ +#pragma once + +/** + * @namespace armonik::api::common::options + * @brief This namespace contains common options for the armonik API. + */ +namespace armonik::api::common::options { +/** + * @enum grpc_socket_type + * @brief Enumerates the types of gRPC sockets supported by armonik API. + */ +enum grpc_socket_type { + tcp = 1, /**< @brief TCP/IP socket type */ + UnixDomainSocket = 2 /**< @brief Unix domain socket type */ +}; +}; // namespace armonik::api::common::options diff --git a/packages/cpp/ArmoniK.Api.Common/header/serilog/SerilogContext.h b/packages/cpp/ArmoniK.Api.Common/header/serilog/SerilogContext.h new file mode 100644 index 000000000..d5bc8b6b4 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/header/serilog/SerilogContext.h @@ -0,0 +1,77 @@ +#pragma once +#include "serilog_typedef.h" + +/** + * @brief This namespace encapsulates the Serilog logging system used in the Armonik API. + */ +namespace armonik::api::common::serilog { +/** + * @brief A struct representing a Serilog context with support for adding and retrieving properties. + */ +struct serilog_context { +public: + /** + * @brief Retrieves a reference to a Serilog properties pair at a given index. + * @param index_ The index of the desired properties pair. + * @return A constant reference to the properties pair at the specified index. + */ + const serilog_properties_pair_t &operator[](size_t index) const { return _properties[index]; } + + /** + * @brief Appends properties from another Serilog properties vector to this context. + * @param other_ The other Serilog properties vector to append. + */ + void append(const serilog_properties_vector_t &other) { + if (other.empty()) + return; + _properties.insert(_properties.end(), other.begin(), other.end()); + } + + /** + * @brief Retrieves a reference to a Serilog properties pair at a given index. + * @param index_ The index of the desired properties pair. + * @return A reference to the properties pair at the specified index. + */ + serilog_properties_pair_t &operator[](size_t index) { return _properties[index]; } + + /** + * @brief Default constructor for the serilog_context struct. + */ + serilog_context(); + + /** + * @brief Checks if the properties vector is empty. + * @return True if the properties vector is empty, otherwise false. + */ + [[nodiscard]] bool empty() const { return _properties.empty(); } + + /** + * @brief Retrieves the number of properties pairs in the context. + * @return The size of the properties vector. + */ + [[nodiscard]] size_t size() const { return _properties.size(); } + + /** + * @brief Constructor for the serilog_context struct with move semantics for parameters. + * @param level The logging level for this context. + * @param parameters A vector of Serilog properties pairs. + * @param logger_name The name of the logger. + */ + serilog_context(const logging_level level, serilog_properties_vector_t parameters, const char *logger_name) + : level(level), logger_name(logger_name), _properties(std::move(parameters)) {} + + /** + * @brief Adds a key-value pair to the Serilog properties vector. + * @param key_ The key of the properties pair. + * @param value_ The value of the properties pair. + */ + void add(std::string key, utils::json_string value) { _properties.emplace_back(std::move(key), std::move(value)); } + + logging_level level; ///< The logging level for this context. + const std::string logger_name; + ///< The name of the logger associated with this context. +private: + serilog_properties_vector_t _properties; + ///< The vector of Serilog properties pairs. +}; +} // namespace armonik::api::common::serilog diff --git a/packages/cpp/ArmoniK.Api.Common/header/serilog/serilog.h b/packages/cpp/ArmoniK.Api.Common/header/serilog/serilog.h new file mode 100644 index 000000000..137ba6608 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/header/serilog/serilog.h @@ -0,0 +1,810 @@ +/** + +@file serilog_log_entry.h +@brief Defines a class that represents a log entry in the Serilog format. +*/ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "serilog/SerilogContext.h" +#include "serilog/serilog_typedef.h" +#include "utils/StringsUtils.h" + +namespace armonik::api::common::serilog +/** + +@brief Increments the logging level. +@param other The logging level to be incremented. +@return The incremented logging level. +*/ +{ +inline logging_level &operator++(logging_level &other) { + other = static_cast(std::min((other + 1), 5)); + return other; +} + +/** + +@brief Increments the logging level. +@param other The logging level to be incremented. +@param dummy A dummy integer parameter to indicate postfix increment. +@return The original logging level value. +*/ +inline logging_level operator++(logging_level &other, int) { + const logging_level r_val = other; + ++other; + return r_val; +} + +/** + +@brief Decrements the logging level. +@param other The logging level to be decremented. +@return The decremented logging level. +*/ +inline logging_level &operator--(logging_level &other) { + other = static_cast(std::max((other - 1), 0)); + return other; +} + +/** + +@brief Decrements the logging level. +@param other The logging level to be decremented. +@param dummy A dummy integer parameter to indicate postfix decrement. +@return The original logging level value. +*/ +inline logging_level operator--(logging_level &other, int) { + const logging_level r_val = other; + --other; + return r_val; +} + +/** + +@brief String representation of the logging levels. +*/ +const std::string logging_level_strings[6] = {"Verbose", "Debug", "Information", "Warning", "Error", "Fatal"}; + +/** + * @brief Short string representation of the logging levels. + * + */ +const std::string logging_level_strings_short[6] = {"VRB", "DBG", "INF", "WRN", "ERR", "FTL"}; + +/** + * @brief Represents a log entry in the Serilog format. + * + */ +class serilog_log_entry; + +/** + * @brief Represents a log entry in the Serilog format. + * + * + */ +class serilog_log_entry { +public: + /* + * @brief Constructs a new serilog_log_entry object. + * @param message The log message. + * @param context The Serilog context of the log entry. + * + */ + serilog_log_entry(std::string message, serilog_context &&context) + : context(std::move(context)), _message(std::move(message)) { + init_time(); + } + + /** + * @brief Returns the log entry as a raw JSON string. + * @return The log entry as a raw JSON string. + * + */ + [[nodiscard]] std::string to_raw_json_entry() const { + std::stringstream string_stream; + string_stream << R"({"@t": ")" << time << R"(", "@mt":")" << utils::string_tools::escape_json(_message) + << R"(", "@l":")" << logging_level_strings[context.level] << std::string(R"(","Logger":")") + << context.logger_name << std::string("\""); + if (context.empty()) { + string_stream << "}"; + return string_stream.str(); + } + const auto parameters_specified = context.size(); + for (size_t i = 0; i < parameters_specified; ++i) { + string_stream << ",\"" << utils::string_tools::escape_json(context[i].first) << "\":\"" + << utils::string_tools::escape_json(context[i].second.str_val) << "\""; + } + string_stream << "}"; + const auto str = string_stream.str(); + const auto cc = str.c_str(); + return cc; + } + + /** + * @brief Returns the log message. + * @return The log message. + * + */ + [[nodiscard]] const std::string &message() const { return _message; } + + const serilog_context context; + char time[24]{}; + +private: + /** + * @brief Initializes the time of the log entry. + */ + void init_time() { + auto str = std::to_string( + std::chrono::duration_cast(std::chrono::system_clock::now().time_since_epoch()) + .count() % + 1000); + + std::stringstream ss; + ss << std::setw(3) << std::setfill('0') << str; + str = ss.str(); + time_t raw_time; + std::time(&raw_time); + const struct tm *time_info = localtime(&raw_time); + strftime(time, 23, "%FT%T.", time_info); + time[19] = '.'; + std::copy(str.data(), &str[3], &time[20]); + time[23] = '\0'; + } + + /** + * @brief The log message. + * + */ + std::string _message; +}; + +/** + @class serilog + @brief A class for logging messages with different levels and properties using Serilog format. +*/ +class serilog { +public: + inline static logging_level base_level; + inline static logging_level base_level_serilog; + + logging_level level = logging_level::debug; + logging_level level_serilog = logging_level::verbose; + + logging_format format_mode = CONSOLE; + + /** + @brief Default constructor. + */ + serilog(logging_format l_format_mode = CONSOLE) : format_mode(l_format_mode) { initialize(name_); } + + /** + @brief Constructor that takes a name and a vector of properties. + @param name The name of the logger. + @param properties A vector of properties. + */ + serilog(const char *name, serilog_properties_vector_t &&properties, logging_format l_format_mode = CONSOLE) + : format_mode(l_format_mode), properties_(std::move(properties)) { + initialize(name); + } + + /** + @brief Constructor that takes a name and a single property pair. + @param name The name of the logger. + @param property_pair A single property pair. + */ + serilog(const char *name, serilog_properties_pair_t &&property_pair, logging_format l_format_mode = CONSOLE) + : format_mode(l_format_mode), properties_({std::move(property_pair)}) { + initialize(name); + } + + /** + @brief Constructor that takes a name. + @param name The name of the logger. + */ + explicit serilog(const char *name, logging_format l_format_mode = CONSOLE) : format_mode(l_format_mode) { + initialize(name); + } + + /** + @brief Constructor that takes a name and logging levels for console and Serilog. + @param name The name of the logger. + @param console_logging_level The logging level for console output. + @param serilog_logging_level The logging level for Serilog output. + */ + explicit serilog(const char *name, const logging_level console_logging_level, + const logging_level serilog_logging_level, logging_format l_format_mode = CONSOLE) + : format_mode(l_format_mode) { + initialize(name); + level = console_logging_level; + level_serilog = serilog_logging_level; + } + + /** + @brief Destructor. + */ + ~serilog() { + enrich_.clear(); + if (!static_instance_) { + // std::cout << "Remove last dynamic object" << std::endl; + std::lock_guard guard(logs_mutex_); + // std::cout << "Get logs_mutex_ OK" << std::endl; + unregister_logger(this); + + // std::cout << "unregister logger OK" << std::endl; + shared_instance().transfer_logs(serilog_dispatch_queue_); + shared_instance().format_mode = format_mode; + // std::cout << "transfer logger OK" << std::endl; + return; + } + s_terminating_ = true; + std::unique_lock lock{s_thread_finished_mutex_}; + s_thread_finished_.wait(lock); + // std::cout << std::string{"Destroy serilog"} << std::endl; + } + + serilog(const serilog &) = delete; + + serilog(serilog &&) = delete; + + serilog &operator=(const serilog &) = delete; + + serilog &operator=(serilog &&) = delete; + + /** + @brief Sends events to the handler. + */ + static void send_events_handler() { + bool has_data(false); + std::stringstream string_stream; + { + std::lock_guard static_guard(s_loggers_mutex_); + if (_s_loggers.empty()) + return; + int64_t index = static_cast(_s_loggers.size()) - 1; + + while (index >= 0) { + const auto &logger = _s_loggers[index]; + std::lock_guard guard(logger->logs_mutex_); + + while (!logger->serilog_dispatch_queue_.empty()) { + // std::cout << "new message : " << logger->format_mode << std::endl; + if (logger->format_mode == SEQ) { + has_data = true; + string_stream << logger->serilog_dispatch_queue_.back()->to_raw_json_entry() << "\n"; + } + + delete logger->serilog_dispatch_queue_.back(); + logger->serilog_dispatch_queue_.pop_back(); + } + + --index; + } + } + if (has_data) { + try { + std::cout << string_stream.str(); + } catch (const std::exception &e) { + log_error("Error while trying to ingest logs:", {{"What", utils::json_string{e.what()}}}); + throw; + } + } + } + +private: + static void init(const logging_level verbosity, const logging_level serilog_verbosity, + const size_t dispatch_interval = 1000) { + if (s_initialized_) + return; + + s_initialized_ = true; + base_level = verbosity; + base_level_serilog = serilog_verbosity; + _s_dispatch_interval = std::chrono::milliseconds(dispatch_interval); + // std::cout << "start serilog thread" << std::endl; + shared_instance().start_thread(); + } + +public: + void add_property(std::string key, utils::json_string val) { + properties_.emplace_back(std::move(key), std::move(val)); + } + + static void add_shared_property(std::string key, utils::json_string val) { + s_shared_properties_.emplace_back(std::move(key), std::move(val)); + } + + void enrich(std::function enrich) { enrich_.push_back(std::move(enrich)); } + + static void add_shared_enrich(std::function enrich) { + s_enrich_.push_back(std::move(enrich)); + } + + /** + @brief Log a verbose message with properties. + @param message The message to log. + @param properties Additional properties for the log entry. + */ + void verbose(std::string message, serilog_properties_vector_t &&properties) const { + instance_log_generic(std::move(message), std::move(properties)); + } + + /** + @brief Log a debug message with properties. + @param message The message to log. + @param properties Additional properties for the log entry. + */ + void debug(std::string message, serilog_properties_vector_t &&properties) const { + instance_log_generic(std::move(message), std::move(properties)); + } + + /** + @brief Log an info message with properties. + @param message The message to log. + @param properties Additional properties for the log entry. + */ + void info(std::string message, serilog_properties_vector_t &&properties) const { + instance_log_generic(std::move(message), std::move(properties)); + } + + /** + @brief Log a warning message with properties. + @param message The message to log. + @param properties Additional properties for the log entry. + */ + void warning(std::string message, serilog_properties_vector_t &&properties) const { + instance_log_generic(std::move(message), std::move(properties)); + } + + /** + @brief Log an error message with properties. + @param message The message to log. + @param properties Additional properties for the log entry. + */ + void error(std::string message, serilog_properties_vector_t &&properties) const { + instance_log_generic(std::move(message), std::move(properties)); + } + + /** + @brief Log a fatal message with properties. + @param message The message to log. + @param properties Additional properties for the log entry. + */ + void fatal(std::string message, serilog_properties_vector_t &&properties) const { + instance_log_generic(std::move(message), std::move(properties)); + } + + /** + @brief Log a verbose message without properties. + @param message The message to log. + */ + void verbose(std::string message) const { instance_log_generic(std::move(message)); } + + /** + @brief Log a debug message without properties. + @param message The message to log. + */ + void debug(std::string message) const { instance_log_generic(std::move(message)); } + + /** + @brief Log an info message without properties. + @param message The message to log. + */ + void info(std::string message) const { instance_log_generic(std::move(message)); } + + /** + @brief Log a warning message without properties. + @param message The message to log. + */ + void warning(std::string message) const { instance_log_generic(std::move(message)); } + + /** + @brief Log an error message without properties. + @param message The message to log. + */ + void error(std::string message) const { instance_log_generic(std::move(message)); } + + /** + @brief Log a fatal message without properties. + @param message The message to log. + */ + void fatal(std::string message) const { instance_log_generic(std::move(message)); } + + /** + @brief Log a static verbose message. + @param message The message to log. + */ + static void log_verbose(std::string message) { + shared_instance().instance_log_generic(std::move(message)); + } + + /** + @brief Log a static debug message. + @param message The message to log. + */ + static void log_debug(std::string message) { + shared_instance().instance_log_generic(std::move(message)); + } + + /** + @brief Log a static info message. + @param message The message to log. + */ + static void log_info(std::string message) { + shared_instance().instance_log_generic(std::move(message)); + } + + /** + @brief Log a static warning message. + @param message The message to log. + */ + static void log_warning(std::string message) { + shared_instance().instance_log_generic(std::move(message)); + } + + /** + @brief Log a static error message. + @param message The message to log. + */ + static void log_error(std::string message) { + shared_instance().instance_log_generic(std::move(message)); + } + + /** + @brief Log a static fatal message. + @param message The message to log. + */ + static void log_fatal(std::string message) { + shared_instance().instance_log_generic(std::move(message)); + } + + /** + @brief Log a static verbose message with properties. + @param message The message to log. + @param properties Additional properties for the log entry. + */ + static void log_verbose(std::string message, serilog_properties_vector_t &&properties) { + shared_instance().instance_log_generic(std::move(message), std::move(properties)); + } + + /** + @brief Log a static debug message with properties. + @param message The message to log. + @param properties Additional properties for the log entry. + */ + static void log_debug(std::string message, serilog_properties_vector_t &&properties) { + shared_instance().instance_log_generic(std::move(message), std::move(properties)); + } + + /** + @brief Log a static info message with properties. + @param message The message to log. + @param properties Additional properties for the log entry. + */ + static void log_info(std::string message, serilog_properties_vector_t &&properties) { + shared_instance().instance_log_generic(std::move(message), std::move(properties)); + } + + /** + @brief Log a static warning message with properties. + @param message The message to log. + @param properties Additional properties for the log entry. + */ + static void log_warning(std::string message, serilog_properties_vector_t &&properties) { + shared_instance().instance_log_generic(std::move(message), std::move(properties)); + } + + /** + @brief Log a static error message with properties. + @param message The message to log. + @param properties Additional properties for the log entry. + */ + static void log_error(std::string message, serilog_properties_vector_t &&properties) { + shared_instance().instance_log_generic(std::move(message), std::move(properties)); + } + + /** + @brief Log a static fatal message with properties. + @param message The message to log. + @param properties Additional properties for the log entry. + */ + static void log_fatal(std::string message, serilog_properties_vector_t &&properties) { + shared_instance().instance_log_generic(std::move(message), std::move(properties)); + } + + // endregion +private: + /** + * @brief Flag to check if the logger is initialized. + */ + inline static bool s_initialized_; + + /** + * @brief Flag to check if the logger is terminating. + */ + inline static bool s_terminating_; + + /** + * @brief Duration between log dispatches. + */ + inline static std::chrono::duration _s_dispatch_interval; + + /** + * @brief Mutex for thread finished condition variable. + */ + inline static std::mutex s_thread_finished_mutex_; + + /** + * @brief Condition variable for thread finished. + */ + inline static std::condition_variable s_thread_finished_; + + /** + * @brief Mutex for loggers. + */ + inline static std::mutex s_loggers_mutex_; + + /** + * @brief Vector containing all registered loggers. + */ + inline static std::vector _s_loggers; + + /** + * @brief Atomic integer to store the logger ID. + */ + inline static std::atomic_int32_t s_logger_id_{0}; + + /** + * @brief Shared properties for all loggers. + */ + inline static serilog_properties_vector_t s_shared_properties_; + + /** + * @brief Shared enrich functions for all loggers. + */ + inline static std::vector> s_enrich_; + + /** + * @brief Log dispatch queue. + */ + mutable std::vector serilog_dispatch_queue_; + + /** + * @brief Mutex for logs. + */ + mutable std::mutex logs_mutex_; + + /** + * @brief Flag to check if the logger is a static instance. + */ + bool static_instance_{false}; + + /** + * @brief Logger name. + */ + char name_[32]{"Default\0"}; + + /** + * @brief Logger properties. + */ + serilog_properties_vector_t properties_; + + /** + * @brief Enrich functions for the logger. + */ + std::vector> enrich_; + + /** + * @brief Logger ID. + */ + const int32_t id_ = s_logger_id_++; + + /** + * @brief Constructor for the logger. + * @param bool Flag to indicate if the logger is a static instance. + */ + serilog(bool) { + s_initialized_ = false; + s_terminating_ = false; + base_level = logging_level::fatal; + base_level_serilog = logging_level::verbose; + _s_dispatch_interval = std::chrono::milliseconds(100); + static_instance_ = true; + register_logger(this); + } + + /** + * @brief Static method to handle sending log events in a loop. + */ + static void send_events_loop_handler() { + // std::cout << "starting serilog thread" << std::endl; + while (!s_terminating_) { + std::this_thread::sleep_for(_s_dispatch_interval); + send_events_handler(); + } + // std::cout << "ending serilog thread" << std::endl; + s_thread_finished_.notify_all(); + } + + /** + * @brief Generic logging function for a specific log level. + * @param L Log level. + * @param message Log message. + * @param properties Log properties. + */ + template + void instance_log_generic(std::string message, serilog_properties_vector_t &&properties) const { + if (L < level && L < level_serilog) + return; + enqueue(std::move(message), make_context(L, properties)); + } + + /** + * @brief Generic logging function for a specific log level without properties. + * @param L Log level. + * @param message Log message. + */ + template void instance_log_generic(std::string message) const { + if (L < level && L < level_serilog) + return; + enqueue(std::move(message), make_context(L)); + } + + /** + * @brief Start the log dispatch loop in a separate thread. + */ + void start_thread() const { + if (!static_instance_) + return; + std::thread t(&serilog::send_events_loop_handler); + t.detach(); + } + + /** + * @brief Get the shared logger instance. + * @return serilog& The shared logger instance. + */ + [[nodiscard]] static serilog &shared_instance() { + static serilog instance(true); + return instance; + } + + /** + * @brief Create a serilog_context with properties. + * @param level_ Logging level. + * @param properties Log properties. + * @return serilog_context The created context. + */ + serilog_context make_context(logging_level level_, serilog_properties_vector_t properties) const { + auto ctx = serilog_context(level_, std::move(properties), name_); + ctx.append(properties_); + ctx.append(s_shared_properties_); + if (!enrich_.empty()) { + for (auto &enrich : enrich_) { + enrich(ctx); + } + } + if (!s_enrich_.empty()) { + for (auto &enrich : s_enrich_) { + enrich(ctx); + } + } + return ctx; + } + + /** + * @brief Create a serilog_context without properties. + * @param level_ Logging level. + * @return serilog_context The created context. + */ + serilog_context make_context(logging_level level_) const { + auto ctx = serilog_context(level_, properties_, name_); + ctx.append(s_shared_properties_); + if (!enrich_.empty()) { + for (auto &enrich : enrich_) { + enrich(ctx); + } + } + if (!s_enrich_.empty()) { + for (auto &enrich : s_enrich_) { + enrich(ctx); + } + } + return ctx; + } + + /** + * @brief Enqueue a log entry. + * @param message Log message. + * @param context_ Log context. + */ + void enqueue(std::string message, serilog_context &&context_) const { + auto *entry = new serilog_log_entry(std::move(message), std::move(context_)); + + if (entry->context.level >= level_serilog) { + std::lock_guard guard(logs_mutex_); + serilog_dispatch_queue_.push_back(entry); + } + + if (entry->context.level >= level && format_mode == CONSOLE) { + static constexpr char esc_char = 27; + std::stringstream ss; + ss << entry->time << "\t" << entry->context.logger_name << "\t[" + << logging_level_strings_short[entry->context.level] << "]\t" << esc_char << "[1m" << entry->message() + << esc_char << "[0m\t\t"; + if (!entry->context.empty()) { + for (size_t i = 0; i < entry->context.size(); ++i) { + ss << entry->context[i].first << "=" << entry->context[i].second.str_val << " "; + } + } + ss << std::endl; + if (entry->context.level > logging_level::warning) { + std::cerr << ss.str(); + std::cerr.flush(); + } else { + std::cout << ss.str(); + std::cout.flush(); + } + } + } + + /** + * @brief Transfer logs from a queue to the serilog_dispatch_queue. + * @param queue_ Queue containing log entries. + */ + void transfer_logs(std::vector &queue_) const { + std::lock_guard guard(logs_mutex_); + serilog_dispatch_queue_.reserve(serilog_dispatch_queue_.size() + queue_.size()); + serilog_dispatch_queue_.insert(serilog_dispatch_queue_.end(), queue_.begin(), queue_.end()); + } + + /** + * @brief Register a logger. + * @param logger Logger to be registered. + */ + static void register_logger(serilog *logger) { + std::lock_guard guard(s_loggers_mutex_); + _s_loggers.push_back(logger); + } + + /** + * @brief Unregister a logger. + * @param logger Logger to be unregistered. + */ + static void unregister_logger(serilog *logger) { + std::lock_guard guard(s_loggers_mutex_); + const auto pos = + std::find_if(_s_loggers.begin(), _s_loggers.end(), [&](auto &l_logger) { return logger == l_logger; }); + + if (pos != _s_loggers.end()) { + // std::cout << "Delete logger found" << std::endl; + _s_loggers.erase(pos); + } + } + + /** + * @brief Initialize the logger with a name. + * @param name Logger name. + */ + void initialize(const char *name) { + level = base_level; + level_serilog = base_level_serilog; + std::strcpy(name_, name); + register_logger(this); + shared_instance().init(level, level_serilog); + } +}; +} // namespace armonik::api::common::serilog diff --git a/packages/cpp/ArmoniK.Api.Common/header/serilog/serilog_typedef.h b/packages/cpp/ArmoniK.Api.Common/header/serilog/serilog_typedef.h new file mode 100644 index 000000000..1af410e7d --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/header/serilog/serilog_typedef.h @@ -0,0 +1,37 @@ +#pragma once + +#include "utils/StringsUtils.h" +#include + +/// @namespace armonik::api::common::serilog +/// @brief A namespace for serilog functionality in the Armonik API +namespace armonik::api::common::serilog { +/// @typedef serilog_properties_pair_t +/// @brief A pair containing a string as a key and a json_string as a value for serilog properties +using serilog_properties_pair_t = std::pair; + +/// @typedef serilog_properties_vector_t +/// @brief A vector of serilog_properties_pair_t for storing multiple serilog properties +using serilog_properties_vector_t = std::vector; + +/// @enum logging_level +/// @brief An enumeration representing the different logging levels for serilog +enum logging_level { + verbose = 0, + ///< Verbose logging level (lowest) + debug = 1, + ///< Debug logging level + info = 2, + ///< Information logging level + warning = 3, + ///< Warning logging level + error = 4, + ///< Error logging level + fatal = 5 ///< Fatal logging level (highest) +}; + +enum logging_format { + CONSOLE = 0, + SEQ = 1, +}; +} // namespace armonik::api::common::serilog diff --git a/packages/cpp/ArmoniK.Api.Common/header/utils/EnvConfiguration.h b/packages/cpp/ArmoniK.Api.Common/header/utils/EnvConfiguration.h new file mode 100644 index 000000000..eb609decc --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/header/utils/EnvConfiguration.h @@ -0,0 +1,46 @@ +/** + * @file EnvConfiguration.h + * @brief Header file for the EnvConfiguration class + */ + +#include "utils/IConfiguration.h" + +namespace armonik::api::common::utils { +/** + * @class EnvConfiguration + * @brief An implementation of IConfiguration that handles environment variables + */ +class EnvConfiguration : public IConfiguration { +public: + /** + * @brief Default constructor + */ + EnvConfiguration() = default; + + /** + * @brief Gets the value of an environment variable + * @param string The name of the environment variable + * @return The value of the environment variable, or an empty string if not found + */ + [[nodiscard]] std::string get(const std::string &string) const override { + std::string value = std::getenv(string.c_str()); + if (!value.empty()) { + return value; + } + throw std::runtime_error("Can't get server address !"); + } + + /** + * @brief Sets the value of an environment variable + * @param string The name of the environment variable + * @param value The value to set + */ + void set(const std::string &string, const std::string &value) override {} + + /** + * @brief Copies the values of another IConfiguration object into this one + * @param other The IConfiguration object to copy from + */ + void set(const IConfiguration &other) override {} +}; +} // namespace armonik::api::common::utils diff --git a/packages/cpp/ArmoniK.Api.Common/header/utils/GuuId.h b/packages/cpp/ArmoniK.Api.Common/header/utils/GuuId.h new file mode 100644 index 000000000..e03ae65d5 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/header/utils/GuuId.h @@ -0,0 +1,47 @@ +/** + * @file GuuId.h + * @brief Header file for the GuuId class, providing a UUID generation method. + */ + +#pragma once + +#include +#include +#include + +/** + * @brief The armonik::api::common::utils namespace provides utility classes and functions for the Armonik API. + */ +namespace armonik::api::common::utils { +/** + * @class GuuId + * @brief The GuuId class provides a static method for generating UUIDs. + */ +class GuuId { +public: + /** + * @brief Generates a random UUID string. + * + * This method generates a random UUID string, following the format "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx". + * + * @return A std::string containing the generated UUID. + */ + static std::string generate_uuid() { + std::random_device rd; + std::mt19937 gen(rd()); + std::uniform_int_distribution<> dis(0, 15); + + const std::string chars = "0123456789abcdef"; + std::string uuid = ""; + + for (int i = 0; i < 32; i++) { + uuid += chars[dis(gen)]; + if (i == 7 || i == 11 || i == 15 || i == 19) { + uuid += '-'; + } + } + + return uuid; + } +}; +} // namespace armonik::api::common::utils diff --git a/packages/cpp/ArmoniK.Api.Common/header/utils/IConfiguration.h b/packages/cpp/ArmoniK.Api.Common/header/utils/IConfiguration.h new file mode 100644 index 000000000..7624fcc0a --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/header/utils/IConfiguration.h @@ -0,0 +1,72 @@ +/** + * @file IConfiguration.h + * @brief Interface for a configuration class that stores and manages key-value pairs. + */ + +#pragma once + +#include +#include +#include + +namespace armonik::api::common::options { +class ComputePlane; +} + +namespace armonik::api::common::utils { +/** + * @class IConfiguration + * @brief Interface for a configuration class that stores and manages key-value pairs. + */ +class IConfiguration { +public: + /** + * @brief Default constructor. + */ + IConfiguration() = default; + + /** + * @brief Default virtual destructor. + */ + virtual ~IConfiguration() = default; + + /** + * @brief Get the value associated with the given key. + * @param string Key to look up. + * @return The value associated with the key, as a string. + */ + [[nodiscard]] virtual std::string get(const std::string &string) const = 0; + + /** + * @brief Set the value associated with the given key. + * @param string Key to set the value for. + * @param value Value to set for the key. + */ + virtual void set(const std::string &string, const std::string &value) = 0; + + /** + * @brief Set the values from another IConfiguration object. + * @param other IConfiguration object to copy values from. + */ + virtual void set(const IConfiguration &other) = 0; + + /** + * @brief Add JSON configuration from a file. + * @param file_path Path to the JSON file. + * @return Reference to the current IConfiguration object. + */ + IConfiguration &add_json_configuration(const std::string &file_path); + + /** + * @brief Add environment variable configuration. + * @return Reference to the current IConfiguration object. + */ + IConfiguration &add_env_configuration(); + + /** + * @brief Get the current ComputePlane configuration. + * @return A ComputePlane object representing the current configuration. + */ + options::ComputePlane get_compute_plane(); +}; +} // namespace armonik::api::common::utils diff --git a/packages/cpp/ArmoniK.Api.Common/header/utils/JsonConfiguration.h b/packages/cpp/ArmoniK.Api.Common/header/utils/JsonConfiguration.h new file mode 100644 index 000000000..0f1faa68a --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/header/utils/JsonConfiguration.h @@ -0,0 +1,42 @@ +#pragma once +/** + * @file JsonConfiguration.h + * @brief Definition of a JSON configuration class that inherits from IConfiguration. + */ + +#include "utils/IConfiguration.h" + +namespace armonik::api::common::utils { +/** + * @class JsonConfiguration + * @brief JSON configuration class that inherits from IConfiguration. + */ +class JsonConfiguration : public IConfiguration { +public: + /** + * @brief Constructor that takes a JSON string. + * @param string JSON string to be used for configuration. + */ + explicit JsonConfiguration(const std::string &string) {} + + /** + * @brief Get the value associated with the given key. + * @param string Key to look up. + * @return The value associated with the key, as a string. + */ + [[nodiscard]] std::string get(const std::string &string) const override { return ""; } + + /** + * @brief Set the value associated with the given key. + * @param string Key to set the value for. + * @param value Value to set for the key. + */ + void set(const std::string &string, const std::string &value) override {} + + /** + * @brief Set the values from another IConfiguration object. + * @param other IConfiguration object to copy values from. + */ + void set(const IConfiguration &other) override {} +}; +} // namespace armonik::api::common::utils diff --git a/packages/cpp/ArmoniK.Api.Common/header/utils/RootConfiguration.h b/packages/cpp/ArmoniK.Api.Common/header/utils/RootConfiguration.h new file mode 100644 index 000000000..4fa72013b --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/header/utils/RootConfiguration.h @@ -0,0 +1,51 @@ +/** + * @file RootConfiguration.h + * @brief Definition of a root configuration class that inherits from IConfiguration. + */ + +#pragma once +#include "utils/IConfiguration.h" + +namespace armonik::api::common::utils { +/** + * @class RootConfiguration + * @brief Root configuration class that inherits from IConfiguration. + */ +class RootConfiguration : public IConfiguration { +public: + /** + * @brief Default constructor. + */ + RootConfiguration() = default; + + /** + * @brief Get the value associated with the given key. + * @param key Key to look up. + * @return The value associated with the key, as a string. + */ + [[nodiscard]] std::string get(const std::string &key) const override { + auto pair = options_.find(key); + + return (pair != options_.end()) ? (*pair).second : ""; + } + + /** + * @brief Set the value associated with the given key. + * @param key Key to set the value for. + * @param value Value to set for the key. + */ + void set(const std::string &key, const std::string &value) override { options_[key] = value; } + + /** + * @brief Set the values from another IConfiguration object. + * @param other IConfiguration object to copy values from. + */ + void set(const IConfiguration &other) override {} + +protected: + /** + * @brief Storage for the key-value pairs. + */ + std::unordered_map options_; +}; +} // namespace armonik::api::common::utils diff --git a/packages/cpp/ArmoniK.Api.Common/header/utils/StringsUtils.h b/packages/cpp/ArmoniK.Api.Common/header/utils/StringsUtils.h new file mode 100644 index 000000000..bae7436dd --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/header/utils/StringsUtils.h @@ -0,0 +1,133 @@ +/** + * @file + * @brief Contains utility structures for string manipulationand JSON string creation + */ + +#pragma once + +#include +#include + +/** @namespace armonik::api::common::utils + * @brief A namespace for common utility structures used within the Armonik API + */ +namespace armonik::api::common::utils { +/** @struct string_tools + * @brief A structure containing static string utility methods + */ +struct string_tools { + /** @brief Escapes special characters in a JSON string + * @param input The input string + * @return A JSON-safe escaped string + */ + static inline auto escape_json(const std::string &input) -> std::string { + std::ostringstream output; + + for (const char c : input) { + switch (c) { + case '\"': + output << "\\\""; + break; + case '\\': + output << "\\\\"; + break; + case '\b': + output << "\\b"; + break; + case '\f': + output << "\\f"; + break; + case '\n': + output << "\\n"; + break; + case '\t': + output << "\\t"; + break; + default: + if (c < '\x20') { + output << "\\u" << std::hex << std::setw(4) << std::setfill('0') << static_cast(c); + } else { + output << c; + } + } + } + + return output.str(); + } +}; + +/** @struct json_string + * @brief A structure for creating JSON-safe string objects + */ +struct json_string { + json_string() = default; + + /** @brief Constructs a JSON-safe string object from a null-terminated char array + * @param value A pointer to a null-terminated char array + */ + explicit json_string(const char *value) : str_val(value == nullptr ? "" : value) {} + + /** @brief Constructs a JSON-safe string object from a std::string + * @param value A std::string object + */ + explicit json_string(const std::string &value) : str_val(value) {} + + /** @brief Constructs a JSON-safe string object from an rvalue std::string + * @param value An rvalue std::string object + */ + explicit json_string(std::string &&value) : str_val(value) {} + + /** @brief Constructs a JSON-safe string object from an rvalue uint8_t + * @param value An rvalue uint8_t object + */ + explicit json_string(uint8_t &&value) : str_val(std::to_string(value)) {} + + /** @brief Constructs a JSON-safe string object from an rvalue uint32_t + * @param value An rvalue uint32_t object + */ + explicit json_string(uint32_t &&value) : str_val(std::to_string(value)) {} + + /** @brief Constructs a JSON-safe string object from an rvalue uint64_t + * @param value An rvalue uint64_t object + */ + explicit json_string(uint64_t &&value) : str_val(std::to_string(value)) {} + + /** @brief Constructs a JSON-safe string object from an rvalue int8_t + * @param value An rvalue int8_t object + */ + explicit json_string(int8_t &&value) : str_val(std::to_string(value)) {} + + /** @brief Constructs a JSON + ** @brief Constructs a JSON-safe string object from an rvalue int32_t + * @param value An rvalue int32_t object + */ + explicit json_string(int32_t &&value) : str_val(std::to_string(value)) {} + + /** @brief Constructs a JSON-safe string object from an rvalue int64_t + * @param value An rvalue int64_t object + */ + explicit json_string(int64_t &&value_) : str_val(std::to_string(value_)) {} + + /** @brief Constructs a JSON-safe string object from a template parameter + * @tparam T The template parameter type + * @param value A value of type T + */ + template json_string(T value) { + std::ostringstream ss; + ss << value; + str_val = ss.str(); + } + + /** @brief Constructs a JSON-safe string object from a reference to a template parameter + * @tparam T The template parameter type + * @param value A const reference to a value of type T + */ + template explicit json_string(const T &value) { + std::ostringstream ss; + ss << value; + str_val = ss.str(); + } + + std::string str_val; ///< The JSON-safe string value +}; +} // namespace armonik::api::common::utils diff --git a/packages/cpp/ArmoniK.Api.Common/source/utils/IConfiguration.cpp b/packages/cpp/ArmoniK.Api.Common/source/utils/IConfiguration.cpp new file mode 100644 index 000000000..c010b2c38 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Common/source/utils/IConfiguration.cpp @@ -0,0 +1,22 @@ +#include "utils/IConfiguration.h" + +#include "options/ComputePlane.h" +#include "utils/EnvConfiguration.h" +#include "utils/JsonConfiguration.h" + +namespace armonik::api::common::utils { +IConfiguration &IConfiguration::add_json_configuration(const std::string &file_path) { + JsonConfiguration json_configuration(file_path); + + return *this; +} + +IConfiguration &IConfiguration::add_env_configuration() { + EnvConfiguration env_config; + + return *this; +} + +options::ComputePlane IConfiguration::get_compute_plane() { return *this; } + +} // namespace armonik::api::common::utils diff --git a/packages/cpp/ArmoniK.Api.Tests/CMakeLists.txt b/packages/cpp/ArmoniK.Api.Tests/CMakeLists.txt new file mode 100644 index 000000000..548dea028 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Tests/CMakeLists.txt @@ -0,0 +1,83 @@ +set(PROJECT_NAME Armonik.Api.Tests) +set(NAMESPACE Armonik::Api::Tests) + +# Trouver les packages requis +find_package(Protobuf REQUIRED) +find_package(gRPC CONFIG REQUIRED) +find_package(Threads) + +SET(SOURCES_FILES_DIR "${CMAKE_CURRENT_SOURCE_DIR}/source") +SET(HEADER_FILES_DIR "${CMAKE_CURRENT_SOURCE_DIR}/header") + +FILE(GLOB_RECURSE SRC_CLIENT_FILES ${SOURCES_FILES_DIR}/*.cpp) +FILE(GLOB_RECURSE HEADER_CLIENT_FILES ${HEADER_FILES_DIR}/*.h) + +#file(MAKE_DIRECTORY ${BUILD_DIR}/${PROJECT_NAME}) + +add_executable(${PROJECT_NAME} ${SRC_CLIENT_FILES} ${HEADER_CLIENT_FILES}) + +set_property(TARGET ${PROJECT_NAME} PROPERTY CXX_STANDARD 17) + +target_link_libraries(${PROJECT_NAME} PUBLIC protobuf::libprotobuf gRPC::grpc++_unsecure Armonik.Api.Client Armonik.Api.Common GTest::gtest_main GTest::gmock_main) +set(PROTO_BINARY_DIR "${BUILD_DIR}/${PROJECT_NAME}/") +set(PROTO_IMPORT_DIRS "${PROTO_FILES_DIR}") + +target_include_directories(${PROJECT_NAME} + PUBLIC + "$" + "$") + +set_property(TARGET ${PROJECT_NAME} PROPERTY VERSION ${version}) +set_property(TARGET ${PROJECT_NAME} PROPERTY SOVERSION 0) +set_property(TARGET ${PROJECT_NAME} PROPERTY + INTERFACE_${PROJECT_NAME}_MAJOR_VERSION 0) +set_property(TARGET ${PROJECT_NAME} APPEND PROPERTY + COMPATIBLE_INTERFACE_STRING ${PROJECT_NAME}_MAJOR_VERSION +) + +# gTest support +include(FetchContent) +FetchContent_Declare( + googletest + URL https://github.com/google/googletest/archive/03597a01ee50ed33e9dfd640b249b4be3799d395.zip +) +# For Windows: Prevent overriding the parent project's compiler/linker settings +set(gtest_force_shared_crt ON CACHE BOOL "" FORCE) +FetchContent_MakeAvailable(googletest) + +include(GoogleTest) +gtest_discover_tests(${PROJECT_NAME}) + +# generate the version file for the config file +write_basic_package_version_file( + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}ConfigVersion.cmake" + VERSION "${version}" + COMPATIBILITY AnyNewerVersion +) + +install(TARGETS ${PROJECT_NAME} + EXPORT ${PROJECT_NAME}Targets + LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} + ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} + RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} + INCLUDES DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} + ) + +#FILE(GLOB INCLUDE_FILES ${PROTO_BINARY_DIR}/*.h) +install(FILES ${INCLUDE_FILES} DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) + +install(EXPORT ${PROJECT_NAME}Targets + FILE ${PROJECT_NAME}Targets.cmake + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) + +configure_package_config_file(${CMAKE_CURRENT_SOURCE_DIR}/Config.cmake.in + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}Config.cmake" + INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) + +install(FILES + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}Config.cmake" + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}ConfigVersion.cmake" + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) diff --git a/packages/cpp/ArmoniK.Api.Tests/Config.cmake.in b/packages/cpp/ArmoniK.Api.Tests/Config.cmake.in new file mode 100644 index 000000000..7f2ad6b8f --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Tests/Config.cmake.in @@ -0,0 +1,5 @@ +@PACKAGE_INIT@ + +include("${CMAKE_INSTALL_PREFIX}/lib/cmake/Armonik.Api.Tests/Armonik.Api.TestsTargets.cmake") + +check_required_components(Armonik.Api.Tests) \ No newline at end of file diff --git a/packages/cpp/ArmoniK.Api.Tests/header/SubmitterCLientTest.h b/packages/cpp/ArmoniK.Api.Tests/header/SubmitterCLientTest.h new file mode 100644 index 000000000..06a23803b --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Tests/header/SubmitterCLientTest.h @@ -0,0 +1,295 @@ +#pragma once +#include +#include + +#include "gmock/gmock.h" + +#include "submitter_common.pb.h" +#include "submitter_service.grpc.pb.h" + +#include "submitter/SubmitterClient.h" + +/** + * @brief Aims to mock the gRPC client stub + * + */ +class MockStubInterface : public armonik::api::grpc::v1::submitter::Submitter::StubInterface { +public: + MOCK_METHOD(::grpc::Status, GetServiceConfiguration, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::Empty &request, + ::armonik::api::grpc::v1::Configuration *response)); + MOCK_METHOD(::grpc::Status, CreateSession, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter::CreateSessionRequest &request, + ::armonik::api::grpc::v1::submitter::CreateSessionReply *response)); + MOCK_METHOD(::grpc::Status, CancelSession, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::Session &request, + ::armonik::api::grpc::v1::Empty *response)); + MOCK_METHOD(::grpc::Status, ListTasks, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::TaskFilter &request, + ::armonik::api::grpc::v1::TaskIdList *response)); + MOCK_METHOD(::grpc::Status, CreateSmallTasks, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter::CreateSmallTaskRequest &request, + ::armonik::api::grpc ::v1::submitter::CreateTaskReply *response)); + MOCK_METHOD(::grpc::Status, ListSessions, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::SessionFilter &request, + ::armonik::api::grpc::v1::submitter::SessionIdList *response)); + MOCK_METHOD(::grpc::Status, CountTasks, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::TaskFilter &request, + ::armonik::api::grpc::v1::Count *response)); + MOCK_METHOD(::grpc::Status, TryGetTaskOutput, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::TaskOutputRequest &request, + ::armonik::api::grpc::v1::Output *response)); + MOCK_METHOD(::grpc::Status, WaitForAvailability, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::ResultRequest &request, + ::armonik::api::grpc::v1::submitter::AvailabilityReply *response)); + MOCK_METHOD(::grpc::Status, WaitForCompletion, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::WaitRequest &request, + ::armonik::api::grpc::v1::Count *response)); + MOCK_METHOD(::grpc::Status, CancelTasks, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::TaskFilter &request, + ::armonik::api::grpc::v1::Empty *response)); + MOCK_METHOD(::grpc::Status, GetTaskStatus, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter::GetTaskStatusRequest &request, + ::armonik::api::grpc::v1::submitter::GetTaskStatusReply *response)); + MOCK_METHOD(::grpc::Status, GetResultStatus, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter::GetResultStatusRequest &request, + ::armonik::api::grpc ::v1::submitter::GetResultStatusReply *response)); + + MOCK_METHOD(void, GetServiceConfiguration, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::Empty *request, + ::armonik::api::grpc::v1::Configuration *response, std::function)); + MOCK_METHOD(void, GetServiceConfiguration, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::Empty *request, + ::armonik::api::grpc::v1::Configuration *response, ::grpc::ClientUnaryReactor *reactor)); + MOCK_METHOD(void, CreateSession, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter::CreateSessionRequest *request, + ::armonik::api::grpc::v1::submitter::CreateSessionReply *response, std::function)); + MOCK_METHOD(void, CreateSession, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter::CreateSessionRequest *request, + ::armonik::api::grpc::v1::submitter::CreateSessionReply *response, ::grpc::ClientUnaryReactor *reactor)); + MOCK_METHOD(void, CancelSession, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::Session *request, + ::armonik::api::grpc::v1::Empty *response, std ::function)); + MOCK_METHOD(void, CancelSession, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::Session *request, + ::armonik::api::grpc::v1::Empty *response, ::grpc::ClientUnaryReactor *reactor)); + MOCK_METHOD(void, CreateSmallTasks, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter::CreateSmallTaskRequest *request, + ::armonik::api::grpc ::v1::submitter::CreateTaskReply *response, std::function)); + MOCK_METHOD(void, CreateSmallTasks, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter::CreateSmallTaskRequest *request, + ::armonik::api::grpc ::v1::submitter::CreateTaskReply *response, ::grpc::ClientUnaryReactor *reactor)); + MOCK_METHOD(void, CreateLargeTasks, + (::grpc::ClientContext * context, ::armonik::api::grpc::v1::submitter::CreateTaskReply *response, + ::grpc::ClientWriteReactor<::armonik::api::grpc::v1::submitter::CreateLargeTaskRequest> *reactor)); + MOCK_METHOD(void, ListTasks, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::TaskFilter *request, + ::armonik::api::grpc::v1::TaskIdList *response, std::function)); + MOCK_METHOD(void, ListTasks, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::TaskFilter *request, + ::armonik::api::grpc::v1::TaskIdList *response, ::grpc::ClientUnaryReactor *reactor)); + MOCK_METHOD(void, ListSessions, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::SessionFilter *request, + ::armonik::api::grpc::v1::submitter::SessionIdList *response, std::function)); + MOCK_METHOD(void, ListSessions, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::SessionFilter *request, + ::armonik::api::grpc::v1::submitter::SessionIdList *response, ::grpc::ClientUnaryReactor *reactor)); + MOCK_METHOD(void, CountTasks, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::TaskFilter *request, + ::armonik::api::grpc::v1::Count *response, std::function)); + MOCK_METHOD(void, CountTasks, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::TaskFilter *request, + ::armonik::api::grpc::v1::Count *response, ::grpc::ClientUnaryReactor *reactor)); + MOCK_METHOD(void, TryGetResultStream, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::ResultRequest *request, + ::grpc::ClientReadReactor<::armonik::api::grpc::v1::submitter::ResultReply> *reactor)); + MOCK_METHOD(void, TryGetTaskOutput, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::TaskOutputRequest *request, + ::armonik::api::grpc::v1::Output *response, std::function)); + MOCK_METHOD(void, TryGetTaskOutput, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::TaskOutputRequest *request, + ::armonik::api::grpc::v1::Output *response, ::grpc::ClientUnaryReactor *reactor)); + MOCK_METHOD(void, WaitForAvailability, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::ResultRequest *request, + ::armonik::api::grpc::v1::submitter::AvailabilityReply *response, std::function)); + MOCK_METHOD(void, WaitForAvailability, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::ResultRequest *request, + ::armonik::api::grpc::v1::submitter::AvailabilityReply *response, ::grpc::ClientUnaryReactor *reactor)); + MOCK_METHOD(void, WaitForCompletion, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::WaitRequest *request, + ::armonik::api::grpc::v1::Count *response, std::function)); + MOCK_METHOD(void, WaitForCompletion, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::WaitRequest *request, + ::armonik::api::grpc::v1::Count *response, ::grpc::ClientUnaryReactor *reactor)); + MOCK_METHOD(void, CancelTasks, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::TaskFilter *request, + ::armonik::api::grpc::v1::Empty *response, std::function)); + MOCK_METHOD(void, CancelTasks, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter::TaskFilter *request, + ::armonik::api::grpc::v1::Empty *response, ::grpc::ClientUnaryReactor *reactor)); + MOCK_METHOD(void, GetTaskStatus, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter::GetTaskStatusRequest *request, + ::armonik::api::grpc::v1::submitter::GetTaskStatusReply *response, std::function)); + MOCK_METHOD(void, GetTaskStatus, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter::GetTaskStatusRequest *request, + ::armonik::api::grpc::v1::submitter::GetTaskStatusReply *response, ::grpc::ClientUnaryReactor *reactor)); + MOCK_METHOD(void, GetResultStatus, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter::GetResultStatusRequest *request, + ::armonik::api::grpc ::v1::submitter::GetResultStatusReply *response, + std::function)); + MOCK_METHOD(void, GetResultStatus, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter::GetResultStatusRequest *request, + ::armonik::api::grpc ::v1::submitter::GetResultStatusReply *response, + ::grpc::ClientUnaryReactor *reactor)); + + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::Configuration> *, + AsyncGetServiceConfigurationRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::Empty &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::Configuration> *, + PrepareAsyncGetServiceConfigurationRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::Empty &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::submitter::CreateSessionReply> *, + AsyncCreateSessionRaw, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter ::CreateSessionRequest &request, ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::submitter::CreateSessionReply> *, + PrepareAsyncCreateSessionRaw, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter ::CreateSessionRequest &request, ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::Empty> *, AsyncCancelSessionRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::Session &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::Empty> *, + PrepareAsyncCancelSessionRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::Session &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::submitter::CreateTaskReply> *, + AsyncCreateSmallTasksRaw, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter ::CreateSmallTaskRequest &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::submitter::CreateTaskReply> *, + PrepareAsyncCreateSmallTasksRaw, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter ::CreateSmallTaskRequest &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientWriterInterface<::armonik::api::grpc::v1::submitter::CreateLargeTaskRequest> *, + CreateLargeTasksRaw, + (::grpc::ClientContext * context, ::armonik::api::grpc::v1::submitter::CreateTaskReply *response)); + MOCK_METHOD(::grpc::ClientAsyncWriterInterface<::armonik::api::grpc::v1::submitter::CreateLargeTaskRequest> *, + AsyncCreateLargeTasksRaw, + (::grpc::ClientContext * context, ::armonik::api::grpc::v1::submitter::CreateTaskReply *response, + ::grpc::CompletionQueue *cq, void *tag)); + MOCK_METHOD(::grpc::ClientAsyncWriterInterface<::armonik::api::grpc::v1::submitter::CreateLargeTaskRequest> *, + PrepareAsyncCreateLargeTasksRaw, + (::grpc::ClientContext * context, ::armonik::api::grpc::v1::submitter::CreateTaskReply *response, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::TaskIdList> *, AsyncListTasksRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter ::TaskFilter &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::TaskIdList> *, + PrepareAsyncListTasksRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter ::TaskFilter &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::submitter::SessionIdList> *, + AsyncListSessionsRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter ::SessionFilter &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::submitter::SessionIdList> *, + PrepareAsyncListSessionsRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter ::SessionFilter &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::Count> *, AsyncCountTasksRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter ::TaskFilter &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::Count> *, PrepareAsyncCountTasksRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter ::TaskFilter &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientReaderInterface<::armonik::api::grpc::v1::submitter::ResultReply> *, TryGetResultStreamRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::ResultRequest &request)); + MOCK_METHOD(::grpc::ClientAsyncReaderInterface<::armonik::api::grpc::v1::submitter::ResultReply> *, + AsyncTryGetResultStreamRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::ResultRequest &request, + ::grpc::CompletionQueue *cq, void *tag)); + MOCK_METHOD(::grpc::ClientAsyncReaderInterface<::armonik::api::grpc::v1::submitter::ResultReply> *, + PrepareAsyncTryGetResultStreamRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::ResultRequest &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::Output> *, AsyncTryGetTaskOutputRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::TaskOutputRequest &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::Output> *, + PrepareAsyncTryGetTaskOutputRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::TaskOutputRequest &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::submitter::AvailabilityReply> *, + AsyncWaitForAvailabilityRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::ResultRequest &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::submitter::AvailabilityReply> *, + PrepareAsyncWaitForAvailabilityRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::ResultRequest &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::Count> *, AsyncWaitForCompletionRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter ::WaitRequest &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::Count> *, + PrepareAsyncWaitForCompletionRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter ::WaitRequest &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::Empty> *, AsyncCancelTasksRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter ::TaskFilter &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::Empty> *, PrepareAsyncCancelTasksRaw, + (::grpc::ClientContext * context, const ::armonik::api::grpc::v1::submitter ::TaskFilter &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::submitter::GetTaskStatusReply> *, + AsyncGetTaskStatusRaw, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter ::GetTaskStatusRequest &request, ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::submitter::GetTaskStatusReply> *, + PrepareAsyncGetTaskStatusRaw, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter ::GetTaskStatusRequest &request, ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::submitter::GetResultStatusReply> *, + AsyncGetResultStatusRaw, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter ::GetResultStatusRequest &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD(::grpc::ClientAsyncResponseReaderInterface<::armonik::api::grpc::v1::submitter::GetResultStatusReply> *, + PrepareAsyncGetResultStatusRaw, + (::grpc::ClientContext * context, + const ::armonik::api::grpc::v1::submitter ::GetResultStatusRequest &request, + ::grpc::CompletionQueue *cq)); + MOCK_METHOD((::grpc::ClientReaderWriterInterface<::armonik::api::grpc::v1::submitter ::WatchResultRequest, + ::armonik::api::grpc::v1::submitter::WatchResultStream> *), + WatchResultsRaw, (::grpc::ClientContext * context)); + MOCK_METHOD((::grpc::ClientAsyncReaderWriterInterface<::armonik::api::grpc::v1::submitter::WatchResultRequest, + ::armonik::api::grpc::v1::submitter::WatchResultStream> *), + AsyncWatchResultsRaw, (::grpc::ClientContext * context, ::grpc::CompletionQueue *cq, void *tag)); + MOCK_METHOD((::grpc::ClientAsyncReaderWriterInterface<::armonik::api::grpc::v1::submitter::WatchResultRequest, + ::armonik::api::grpc::v1::submitter::WatchResultStream> *), + PrepareAsyncWatchResultsRaw, (::grpc::ClientContext * context, ::grpc::CompletionQueue *cq)); +}; + +/** + * @brief Initializes task options creates channel with server address + * + * @param channel The gRPC channel to communicate with the server. + * @param default_task_options The default task options. + */ +void init(std::shared_ptr &channel, armonik::api::grpc::v1::TaskOptions &task_options); diff --git a/packages/cpp/ArmoniK.Api.Tests/source/SubmitterCLientTest.cpp b/packages/cpp/ArmoniK.Api.Tests/source/SubmitterCLientTest.cpp new file mode 100644 index 000000000..0872d3999 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Tests/source/SubmitterCLientTest.cpp @@ -0,0 +1,185 @@ +#include +#include +#include + +#include + +#include +#include + +#include "SubmitterClientTest.h" + +#include "submitter/SubmitterClient.h" +#include "submitter_service.grpc.pb.h" + +#include "serilog/serilog.h" +#include "utils/EnvConfiguration.h" +#include "utils/GuuId.h" +#include "utils/StringsUtils.h" + +using armonik::api::common::utils::IConfiguration; +using armonik::api::grpc::v1::TaskOptions; +using armonik::api::grpc::v1::submitter::CreateSessionReply; +using armonik::api::grpc::v1::submitter::CreateSessionRequest; +using armonik::api::grpc::v1::submitter::Submitter; +using grpc::Channel; +using grpc::ClientContext; +using grpc::Status; +using namespace armonik::api::common::utils; + +using ::testing::_; +using ::testing::AtLeast; + +using namespace armonik::api::common::serilog; + +/** + * @brief Initializes task options creates channel with server address + * + * @param channel The gRPC channel to communicate with the server. + * @param default_task_options The default task options. + */ +void init(std::shared_ptr &channel, TaskOptions &default_task_options) { + + EnvConfiguration configuration; + // auto server = std::make_shared(configuration_t); + + configuration.add_json_configuration("appsetting.json").add_env_configuration(); + + std::string server_address = configuration.get("ArmoniK_Client_Server"); + + std::cout << " Server address " << server_address << std::endl; + + channel = grpc::CreateChannel(server_address, grpc::InsecureChannelCredentials()); + + // stub_ = Submitter::NewStub(channel); + + default_task_options.mutable_options()->insert({"key1", "value1"}); + default_task_options.mutable_options()->insert({"key2", "value2"}); + default_task_options.mutable_max_duration()->set_seconds(3600); + default_task_options.mutable_max_duration()->set_nanos(0); + default_task_options.set_max_retries(3); + default_task_options.set_priority(1); + default_task_options.set_partition_id("cpp"); + default_task_options.set_application_name("my-app"); + default_task_options.set_application_version("1.0"); + default_task_options.set_application_namespace("my-namespace"); + default_task_options.set_application_service("my-service"); + default_task_options.set_engine_type("Unified"); +} + +TEST(testMock, createSession) { + // MockStubInterface stub; + std::shared_ptr channel; + + ClientContext context; + CreateSessionReply reply; + CreateSessionRequest request; + + const std::vector &partition_ids = {"cpp"}; + + TaskOptions task_options; + init(channel, task_options); + + ASSERT_EQ(task_options.partition_id(), "cpp"); + + std::unique_ptr stub = Submitter::NewStub(channel); + // EXPECT_CALL(*stub, CreateSession(_, _, _)).Times(AtLeast(1)); + SubmitterClient submitter(std::move(stub)); + std::string session_id = submitter.create_session(task_options, partition_ids); + + std::cout << "create_session response: " << session_id << std::endl; + + ASSERT_FALSE(session_id.empty()); +} + +TEST(testMock, submitTask) { + + serilog log(logging_format::CONSOLE); + + std::cout << "Serilog closed" << std::endl; + + log.enrich([&](serilog_context &ctx) { ctx.add("threadid", std::this_thread::get_id()); }); + log.enrich([&](serilog_context &ctx) { ctx.add("fieldTestValue", 1); }); + log.add_property("time", time(nullptr)); + + ::putenv("GRPC_DNS_RESOLVER=native"); + + std::cout << "Starting client..." << std::endl; + + CreateSessionRequest request; + TaskOptions task_options; + + std::shared_ptr channel; + init(channel, task_options); + + // MockStubInterface stub; + std::unique_ptr stub = Submitter::NewStub(channel); + + *request.mutable_default_task_option() = task_options; + request.add_partition_ids(task_options.partition_id()); + + // EXPECT_CALL(*stub, CreateSession(_, _, _)).Times(AtLeast(1)); + // EXPECT_CALL(*stub, GetServiceConfiguration(_, _, _)).Times(AtLeast(1)); + // EXPECT_CALL(*stub, CreateLargeTasksRaw(_, _)).Times(AtLeast(1)); + + CreateSessionReply reply; + grpc::ClientContext context; + + SubmitterClient submitter(std::move(stub)); + const std::vector &partition_ids = {"cpp"}; + std::string session_id = submitter.create_session(task_options, partition_ids); + + ASSERT_FALSE(session_id.empty()); + + try { + std::vector payloads; + + for (int i = 0; i < 10; i++) { + payload_data data; + data.keys = armonik::api::common::utils::GuuId::generate_uuid(); + data.payload = {'a', 'r', 'm', 'o', 'n', 'i', 'k'}; + data.dependencies = {}; + payloads.push_back(data); + } + const auto [task_ids, failed_task_ids] = + submitter.submit_tasks_with_dependencies(session_id, task_options, payloads, 5); + for (const auto &task_id : task_ids) { + std::stringstream out; + out << "Generate task_ids : " << task_id; + log.info(out.str()); + } + for (const auto &failed_task_id : failed_task_ids) { + std::stringstream out; + out << "Failed task_ids : " << failed_task_id; + log.info(out.str()); + } + } catch (std::exception &e) { + log.error(e.what()); + throw; + } + log.info("Stopping client...OK"); +} + +TEST(testMock, getResult) { + // MockStubInterface stub; + std::shared_ptr channel; + + CreateSessionReply reply; + CreateSessionRequest request; + + const std::vector &partition_ids = {"cpp"}; + + TaskOptions task_options; + armonik::api::grpc::v1::ResultRequest result_request; + + init(channel, task_options); + // EXPECT_CALL(*stub, GetServiceConfiguration(_, _, _)).Times(AtLeast(1)); + // EXPECT_CALL(*stub, TryGetResultStreamRaw(_, _)).Times(AtLeast(1)); + + std::unique_ptr stub = Submitter::NewStub(channel); + SubmitterClient submitter(std::move(stub)); + + auto result = submitter.get_result_async(result_request); + + ASSERT_FALSE(result.get().empty()); +} diff --git a/packages/cpp/ArmoniK.Api.Tests/source/main.cpp b/packages/cpp/ArmoniK.Api.Tests/source/main.cpp new file mode 100644 index 000000000..eaad385a4 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Tests/source/main.cpp @@ -0,0 +1,8 @@ +#include +#include + +int main(int argc, char **argv) { + testing::InitGoogleTest(&argc, argv); + testing::InitGoogleMock(&argc, argv); + return RUN_ALL_TESTS(); +} diff --git a/packages/cpp/ArmoniK.Api.Worker.Tests/CMakeLists.txt b/packages/cpp/ArmoniK.Api.Worker.Tests/CMakeLists.txt new file mode 100644 index 000000000..e311adcbf --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Worker.Tests/CMakeLists.txt @@ -0,0 +1,71 @@ +set(PROJECT_NAME Armonik.Api.Worker.Tests) +set(NAMESPACE Armonik::Api::Worker::Tests) + +# Trouver les packages requis +find_package(Protobuf REQUIRED) +find_package(gRPC CONFIG REQUIRED) +find_package(Threads) + +SET(SOURCES_FILES_DIR "${CMAKE_CURRENT_SOURCE_DIR}/source") +SET(HEADER_FILES_DIR "${CMAKE_CURRENT_SOURCE_DIR}/header") + + +FILE(GLOB_RECURSE SRC_CLIENT_FILES ${SOURCES_FILES_DIR}/*.cpp) +FILE(GLOB_RECURSE HEADER_CLIENT_FILES ${HEADER_FILES_DIR}/*.h) + +#file(MAKE_DIRECTORY ${BUILD_DIR}/${PROJECT_NAME}) + +add_executable(${PROJECT_NAME} ${SRC_CLIENT_FILES} ${HEADER_CLIENT_FILES}) + +set_property(TARGET ${PROJECT_NAME} PROPERTY CXX_STANDARD 17) + +target_link_libraries(${PROJECT_NAME} PUBLIC protobuf::libprotobuf gRPC::grpc++_unsecure Armonik.Api.Worker Armonik.Api.Common) +set(PROTO_BINARY_DIR "${BUILD_DIR}/${PROJECT_NAME}/") +set(PROTO_IMPORT_DIRS "${PROTO_FILES_DIR}") + +target_include_directories(${PROJECT_NAME} + PUBLIC + "$" + "$") + +set_property(TARGET ${PROJECT_NAME} PROPERTY VERSION ${version}) +set_property(TARGET ${PROJECT_NAME} PROPERTY SOVERSION 0) +set_property(TARGET ${PROJECT_NAME} PROPERTY + INTERFACE_${PROJECT_NAME}_MAJOR_VERSION 0) +set_property(TARGET ${PROJECT_NAME} APPEND PROPERTY + COMPATIBLE_INTERFACE_STRING ${PROJECT_NAME}_MAJOR_VERSION +) + +# generate the version file for the config file +write_basic_package_version_file( + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}ConfigVersion.cmake" + VERSION "${version}" + COMPATIBILITY AnyNewerVersion +) + +install(TARGETS ${PROJECT_NAME} + EXPORT ${PROJECT_NAME}Targets + LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} + ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} + RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} + INCLUDES DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} + ) + +#FILE(GLOB INCLUDE_FILES ${PROTO_BINARY_DIR}/*.h) +install(FILES ${INCLUDE_FILES} DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) + +install(EXPORT ${PROJECT_NAME}Targets + FILE ${PROJECT_NAME}Targets.cmake + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) + +configure_package_config_file(${CMAKE_CURRENT_SOURCE_DIR}/Config.cmake.in + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}Config.cmake" + INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) + +install(FILES + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}Config.cmake" + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}ConfigVersion.cmake" + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) diff --git a/packages/cpp/ArmoniK.Api.Worker.Tests/Config.cmake.in b/packages/cpp/ArmoniK.Api.Worker.Tests/Config.cmake.in new file mode 100644 index 000000000..7f2ad6b8f --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Worker.Tests/Config.cmake.in @@ -0,0 +1,5 @@ +@PACKAGE_INIT@ + +include("${CMAKE_INSTALL_PREFIX}/lib/cmake/Armonik.Api.Tests/Armonik.Api.TestsTargets.cmake") + +check_required_components(Armonik.Api.Tests) \ No newline at end of file diff --git a/packages/cpp/ArmoniK.Api.Worker.Tests/source/main.cpp b/packages/cpp/ArmoniK.Api.Worker.Tests/source/main.cpp new file mode 100644 index 000000000..a19c39f99 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Worker.Tests/source/main.cpp @@ -0,0 +1,101 @@ +#include +#include +#include + +#include + +#include "grpcpp/support/sync_stream.h" +#include "objects.pb.h" + +#include "utils/RootConfiguration.h" +#include "utils/WorkerServer.h" +#include "worker_common.grpc.pb.h" +#include "worker_service.grpc.pb.h" + +using grpc::Channel; +using grpc::ClientContext; +using grpc::Status; + +using armonik::api::common::utils::IConfiguration; +using armonik::api::grpc::v1::TaskOptions; + +using namespace armonik::api::grpc::v1::worker; +using namespace armonik::api::worker; +using namespace armonik::api::common::utils; + +/** + * @brief Implements the Worker service. + */ +class WorkerServiceImpl final : public Worker::Service { +private: + armonik::api::common::serilog::serilog logger; + +public: + /** + * @brief Constructs a WorkerServiceImpl object. + */ + WorkerServiceImpl() : logger(armonik::api::common::serilog::logging_format::SEQ) { + logger.info("Build Service WorkerServiceImpl"); + logger.add_property("class", "WorkerServiceImpl"); + logger.add_property("Worker", "ArmoniK.Api.Cpp"); + } + + /** + * @brief Implements the Process method of the Worker service. + * + * @param context The ServerContext object. + * @param reader The ServerReader object. + * @param response The ProcessReply object. + * + * @return The status of the method. + */ + Status Process(::grpc::ServerContext *context, + ::grpc::ServerReader<::armonik::api::grpc::v1::worker::ProcessRequest> *reader, + ::armonik::api::grpc::v1::worker::ProcessReply *response) override { + // Implementation of the Process method + logger.info("Receive new request From C++ Worker"); + auto output = armonik::api::grpc::v1::Output(); + *output.mutable_ok() = armonik::api::grpc::v1::Empty(); + ProcessRequest req; + reader->Read(&req); + *response->mutable_output() = output; + + logger.info("Finish call C++"); + + return grpc::Status::OK; + } + + /** + * @brief Implements the HealthCheck method of the Worker service. + * + * @param context The ServerContext object. + * @param request The Empty object. + * @param response The HealthCheckReply object. + * + * @return The status of the method. + */ + Status HealthCheck(::grpc::ServerContext *context, const ::armonik::api::grpc::v1::Empty *request, + ::armonik::api::grpc::v1::worker::HealthCheckReply *response) override { + // Implementation of the HealthCheck method + logger.info("HealthCheck request OK"); + + response->set_status(HealthCheckReply_ServingStatus_SERVING); + + return Status::OK; + } +}; + +int main(int argc, char **argv) { + std::cout << "Starting C++ worker..." << std::endl; + + std::shared_ptr config = std::make_shared(); + + config->set("ComputePlane__WorkerChannel__Address", "/cache/armonik_worker.sock"); + config->set("ComputePlane__AgentChannel__Address", "/cache/armonik_agent.sock"); + + config->get_compute_plane(); + WorkerServer::create(config)->run(); + + std::cout << "Stooping Server..." << std::endl; + return 0; +} diff --git a/packages/cpp/ArmoniK.Api.Worker/CMakeLists.txt b/packages/cpp/ArmoniK.Api.Worker/CMakeLists.txt new file mode 100644 index 000000000..7d16ff778 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Worker/CMakeLists.txt @@ -0,0 +1,122 @@ +set(PROJECT_NAME ArmoniK.Api.Worker) +set(NAMESPACE ArmoniK::Api::Worker) +set(version 0.1.0) + +set(PROTO_FILES + "worker_service.proto" + "agent_service.proto" + "agent_common.proto" + "worker_common.proto") + +set(PROTO_DEPS + "objects.proto" + "result_status.proto" + "task_status.proto" + "session_status.proto" + "sort_direction.proto") + +foreach(file ${PROTO_FILES} ${PROTO_DEPS}) + configure_file("${PROTO_FILES_DIR}/${file}" "${BUILD_DIR}/${PROJECT_NAME}/${file}" COPYONLY) +endforeach() +list(TRANSFORM PROTO_FILES PREPEND ${BUILD_DIR}/${PROJECT_NAME}/) + +set(CMAKE_FIND_DEBUG_MODE FALSE) +# Trouver les packages requis +find_package(Protobuf REQUIRED) +find_package(gRPC CONFIG REQUIRED) +find_package(Threads) + +SET(SOURCES_FILES_DIR "${CMAKE_CURRENT_SOURCE_DIR}/source") +SET(HEADER_FILES_DIR "${CMAKE_CURRENT_SOURCE_DIR}/header") + +FILE(GLOB_RECURSE SRC_CLIENT_FILES ${SOURCES_FILES_DIR}/*.cpp) +FILE(GLOB_RECURSE HEADER_CLIENT_FILES ${HEADER_FILES_DIR}/*.h) + +file(MAKE_DIRECTORY ${BUILD_DIR}/${PROJECT_NAME}) + +add_library(${PROJECT_NAME} STATIC ${PROTO_FILES} ${SRC_CLIENT_FILES} ${HEADER_CLIENT_FILES}) + +target_link_libraries(${PROJECT_NAME} PUBLIC protobuf::libprotobuf gRPC::grpc++_unsecure ArmoniK.Api.Common) + +target_include_directories(${PROJECT_NAME} + PUBLIC + "$" + "$" + "$" +) + +set(PROTO_BINARY_DIR "${BUILD_DIR}/${PROJECT_NAME}") +set(PROTO_IMPORT_DIRS "${PROTO_FILES_DIR}") + +protobuf_generate( + TARGET ${PROJECT_NAME} + OUT_VAR PROTO_GENERATED_FILES + IMPORT_DIRS ${BUILD_DIR}/${PROJECT_NAME} + PROTOC_OUT_DIR "${PROTO_BINARY_DIR}") +set_source_files_properties(${PROTO_GENERATED_FILES} PROPERTIES SKIP_UNITY_BUILD_INCLUSION on) + +get_target_property(grpc_cpp_plugin_location gRPC::grpc_cpp_plugin LOCATION) + +protobuf_generate( + TARGET ${PROJECT_NAME} + OUT_VAR PROTO_GENERATED_FILES + LANGUAGE grpc + GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc + PLUGIN "protoc-gen-grpc=${grpc_cpp_plugin_location}" + # PLUGIN_OPTIONS "generate_mock_code=true" + IMPORT_DIRS ${BUILD_DIR}/${PROJECT_NAME} + PROTOC_OUT_DIR "${PROTO_BINARY_DIR}") + +set_source_files_properties(${PROTO_GENERATED_FILES} PROPERTIES SKIP_UNITY_BUILD_INCLUSION on) + +target_include_directories(${PROJECT_NAME} + PUBLIC + "$" + "$") + +target_include_directories(${PROJECT_NAME} + PUBLIC + "$" + "$") + +set_property(TARGET ${PROJECT_NAME} PROPERTY VERSION ${version}) +set_property(TARGET ${PROJECT_NAME} PROPERTY SOVERSION 0) +set_property(TARGET ${PROJECT_NAME} PROPERTY + INTERFACE_${PROJECT_NAME}_MAJOR_VERSION 0) +set_property(TARGET ${PROJECT_NAME} APPEND PROPERTY + COMPATIBLE_INTERFACE_STRING ${PROJECT_NAME}_MAJOR_VERSION +) + +# generate the version file for the config file +write_basic_package_version_file( + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}ConfigVersion.cmake" + VERSION "${version}" + COMPATIBILITY AnyNewerVersion +) + +install(TARGETS ${PROJECT_NAME} + EXPORT ${PROJECT_NAME}Targets + LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} + ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} + RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} + INCLUDES DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} + ) + +FILE(GLOB INCLUDE_FILES ${PROTO_BINARY_DIR}/*.h) +install(FILES ${INCLUDE_FILES} DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) + +install(EXPORT ${PROJECT_NAME}Targets + FILE ${PROJECT_NAME}Targets.cmake + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) + +configure_package_config_file(${CMAKE_CURRENT_SOURCE_DIR}/Config.cmake.in + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}Config.cmake" + INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) + +install(FILES + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}Config.cmake" + "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}ConfigVersion.cmake" + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME} +) diff --git a/packages/cpp/ArmoniK.Api.Worker/Config.cmake.in b/packages/cpp/ArmoniK.Api.Worker/Config.cmake.in new file mode 100644 index 000000000..751f562ff --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Worker/Config.cmake.in @@ -0,0 +1,5 @@ +@PACKAGE_INIT@ + +include("${CMAKE_INSTALL_PREFIX}/lib/cmake/Armonik.Api.Client/Armonik.Api.ClientTargets.cmake") + +check_required_components(Armonik.Api.Client) \ No newline at end of file diff --git a/packages/cpp/ArmoniK.Api.Worker/header/utils/WorkerServer.h b/packages/cpp/ArmoniK.Api.Worker/header/utils/WorkerServer.h new file mode 100644 index 000000000..d0572b485 --- /dev/null +++ b/packages/cpp/ArmoniK.Api.Worker/header/utils/WorkerServer.h @@ -0,0 +1,93 @@ +/** + * @file WorkerServer.h + * @brief Contains the WorkerServer class, which represents the worker server for ArmoniK API. + */ +#pragma once +#include +#include +#include +#include + +#include "grpcpp/server_builder.h" + +#include "agent_common.pb.h" +#include "agent_service.grpc.pb.h" + +#include "options/ComputePlane.h" +#include "serilog/SerilogContext.h" +#include "serilog/serilog.h" +#include "utils/IConfiguration.h" + +using namespace armonik::api::grpc::v1::agent; + +namespace armonik::api::worker { +/** + * @class WorkerServer + * @brief Represents the worker server for ArmoniK API. + */ +class WorkerServer { +public: + common::serilog::serilog logger; + +private: + ::grpc::ServerBuilder builder_; + std::shared_ptr configuration_; + +public: + /** + * @brief Constructor for the WorkerServer class. + * @param configuration A shared pointer to the IConfiguration object. + */ + WorkerServer(std::shared_ptr configuration) + : configuration_(std::move(configuration)) { + logger.enrich([&](common::serilog::serilog_context &ctx) { ctx.add("threadId", std::this_thread::get_id()); }); + + logger.add_property("container", "ArmoniK.Worker"); + } + + /** + * @brief Create a WorkerServer instance with the given configuration. + * @tparam Worker The worker class to be used + * @tparam Collection The collection class to be used + * @param configuration Shared pointer to the IConfiguration object + * @param service_configurator Function pointer for the service configurator + * @return A shared pointer to the created WorkerServer instance + */ + template + static std::shared_ptr + create(const std::shared_ptr configuration, + [[maybe_unused]] std::function service_configurator = nullptr) { + configuration->add_json_configuration("appsetting.json").add_env_configuration(); + auto worker_server = std::make_shared(configuration); + worker_server->logger.info("Creating worker"); + common::options::ComputePlane compute_plane(*configuration); + + worker_server->builder_.AddListeningPort(std::string(compute_plane.get_server_address()), + ::grpc::InsecureServerCredentials()); + worker_server->builder_.SetMaxReceiveMessageSize(-1); + + worker_server->logger.info("Initialize and register worker"); + + // Create a gRPC channel to communicate with the server + worker_server->channel = + CreateChannel(std::string(compute_plane.get_agent_address()), ::grpc::InsecureChannelCredentials()); + + // Create a stub for the Submitter service + worker_server->agent_stub = Agent::NewStub(worker_server->channel); + + worker_server->builder_.RegisterService(new Worker()); + worker_server->logger.info("Finish to register new worker"); + + return worker_server; + } + + std::unique_ptr<::grpc::Server> instance_server; ///< Unique pointer to the gRPC server instance + std::shared_ptr<::grpc::Channel> channel; ///< Shared pointer to the gRPC channel + std::unique_ptr agent_stub; ///< Proxy to communicate with the agent + + void run() { + instance_server = builder_.BuildAndStart(); + instance_server->Wait(); + } +}; +} // namespace armonik::api::worker diff --git a/packages/cpp/CMakeLists.txt b/packages/cpp/CMakeLists.txt new file mode 100644 index 000000000..07cbd8048 --- /dev/null +++ b/packages/cpp/CMakeLists.txt @@ -0,0 +1,34 @@ +cmake_minimum_required(VERSION 3.25) + +set(version 0.1.0) + +set(SOLUTION_NAME ArmoniK.Api) +project(${SOLUTION_NAME} C CXX) +if (UNIX) + set(BUILD_DIR "/app/build") + set(PROTO_FILES_DIR "/app/proto") +elseif (WIN32) + set(BUILD_DIR "${CMAKE_SOURCE_DIR}/build") + set(PROTO_FILES_DIR "${CMAKE_SOURCE_DIR}/../../Protos/V1") + list(APPEND CMAKE_PREFIX_PATH "${CMAKE_SOURCE_DIR}/tools/win64/lib/cmake/grpc") +endif (UNIX) + +option(BUILD_TEST OFF) + +# make cache variables for install destinations +include(GNUInstallDirs) +include(CMakePackageConfigHelpers) + +set(CMAKE_CXX_STANDARD 17) + +add_subdirectory(ArmoniK.Api.Common) +add_subdirectory(ArmoniK.Api.Worker) +add_subdirectory(ArmoniK.Api.Client) + + +if (BUILD_TEST OR WIN32) + add_subdirectory(ArmoniK.Api.Worker.Tests) + add_subdirectory(ArmoniK.Api.Tests) +endif() + + diff --git a/packages/cpp/CMakeSettings.json b/packages/cpp/CMakeSettings.json new file mode 100644 index 000000000..a9c794aa7 --- /dev/null +++ b/packages/cpp/CMakeSettings.json @@ -0,0 +1,15 @@ +{ + "configurations": [ + { + "name": "x64-Debug", + "generator": "Ninja", + "configurationType": "Debug", + "inheritEnvironments": [ "msvc_x64_x64" ], + "buildRoot": "${projectDir}\\out\\build\\${name}", + "installRoot": "${projectDir}\\out\\install\\${name}", + "cmakeCommandArgs": "", + "buildCommandArgs": "", + "ctestCommandArgs": "" + } + ] +} \ No newline at end of file diff --git a/packages/cpp/tools/Dockerfile.ubuntu b/packages/cpp/tools/Dockerfile.ubuntu new file mode 100644 index 000000000..00b7bfab9 --- /dev/null +++ b/packages/cpp/tools/Dockerfile.ubuntu @@ -0,0 +1,34 @@ +# Use the latest version of Ubuntu 20.04 as the base image +FROM ubuntu:23.04 + +# Install dependencies +RUN apt-get update && DEBIAN_FRONTEND="noninteractive" TZ="Europe/London" apt-get install -y \ + gcc \ + g++ \ + make \ + build-essential \ + cmake \ + libc-ares-dev \ + protobuf-compiler-grpc \ + grpc-proto \ + libgrpc-dev \ + libgrpc++-dev + +# Set environment variables for protobuf +ENV protobuf_BUILD_TESTS=OFF + +# Update PATH with the new directories +ENV PATH="/app/install/lib:$PATH" +ENV PATH="/app/install/bin:$PATH" + +# Print the PATH variable +RUN echo $PATH + +# Set the working directory for building protobuf +WORKDIR /app/build + +# Set the default command to build the client using CMake and make +CMD ["bash", "-c", "cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/app/install -DBUILD_TEST=OFF /app/source/ && make -j $(nproc) install"] + +# Uncomment the line below if you want the container to start in the bash shell by default +# ENTRYPOINT ["bash"] diff --git a/packages/cpp/tools/Dockerfile.worker b/packages/cpp/tools/Dockerfile.worker new file mode 100644 index 000000000..d8be23be8 --- /dev/null +++ b/packages/cpp/tools/Dockerfile.worker @@ -0,0 +1,102 @@ +# Start with the latest Alpine base image for the build stage +FROM alpine:3.14 AS builder +ARG GRPC_VERSION=v1.54.0 + +# Install all the necessary dependencies required for the build process +# These include tools and libraries for building and compiling the source code +RUN apk update && apk add --no-cache \ + git \ + gcc \ + g++ \ + build-base \ + autoconf \ + automake \ + libtool \ + curl \ + c-ares \ + c-ares-dev \ + make \ + cmake \ + unzip \ + linux-headers + +# Clone the gRPC repository using the specified version +WORKDIR /tmp +RUN git clone -b $GRPC_VERSION https://github.com/grpc/grpc + +# Build the gRPC libraries and binaries +WORKDIR /tmp/grpc +RUN git submodule update --init +RUN mkdir -p cmake/build && \ + cd cmake/build && \ + cmake -DCMAKE_INSTALL_PREFIX=/app/install \ + -DgRPC_INSTALL=ON \ + -DCMAKE_BUILD_TYPE=Release \ + ../.. && \ + make -j $(nproc) && \ + make install && \ + ln -sf /app/install/lib/libgrpc++.so /app/install/lib/libgrpc++.so.1 && \ + ln -sf /app/install/lib/libgrpc++_reflection.so /app/install/lib/libgrpc++_reflection.so.1 + rm -rf /tmp/grpc + +# Update the PATH environment variable to include the gRPC libraries and binaries +ENV PATH="/app/install/lib:$PATH" +ENV PATH="/app/install/bin:$PATH" + +# Display the updated PATH environment variable +RUN echo $PATH + +# Copy the application source files into the image +WORKDIR /app/source +COPY ./packages/cpp/ArmoniK.Api.Common ./ArmoniK.Api.Common +COPY ./packages/cpp/ArmoniK.Api.Worker ./ArmoniK.Api.Worker +COPY ./packages/cpp/ArmoniK.Api.Worker.Tests ./ArmoniK.Api.Worker.Tests +COPY ./packages/cpp/CMakeLists.txt ./ + +# Copy the Protocol Buffer definition files into the image +WORKDIR /app/proto +COPY ./Protos/V1/ /app/proto + +# Build the application using the copied source files and protobuf definitions +WORKDIR /app/builder +RUN cmake "-DCMAKE_INSTALL_PREFIX=/app/install" /app/source/ +RUN make -j $(nproc) install + +# Start with the latest Alpine base image for the final stage +FROM alpine:3.14 +# Install all the necessary dependencies required for the build process +# These include tools and libraries for building and compiling the source code +RUN apk update && apk add --no-cache \ + git \ + gcc \ + g++ \ + build-base \ + autoconf \ + automake \ + libtool \ + curl \ + c-ares \ + c-ares-dev \ + make \ + cmake \ + unzip \ + linux-headers +# Create a non-root user and group for running the application +# This is a security best practice to avoid running applications as the root user +RUN addgroup -g 5000 -S armonikuser && adduser -D -h /home/armonikuser -u 5000 -G armonikuser --shell /bin/sh armonikuser && mkdir /cache && chown armonikuser: /cache +USER armonikuser + +# Copy the application files, libraries, and binaries from the builder image to the final image +COPY --from=builder /app /app + +# Update the PATH environment variable to include the application libraries and binaries +ENV PATH="/app/install/lib:$PATH" +ENV PATH="/app/install/bin:$PATH" + +# Set the entrypoint for the application's test executable +# This is the command that will be executed when the container is run +ENTRYPOINT ["bash", "-c", "/app/install/bin/Armonik.Api.Worker.Tests"] + +# Uncomment the line below if you want to run a shell instead of the application's test executable +# This can be useful for debugging purposes +# ENTRYPOINT ["bash"] diff --git a/packages/cpp/tools/build-worker.sh b/packages/cpp/tools/build-worker.sh new file mode 100644 index 000000000..d429542c2 --- /dev/null +++ b/packages/cpp/tools/build-worker.sh @@ -0,0 +1,31 @@ +#!/bin/bash + +# Declare a variable that will store the image tag for our Docker image +IMAGE_TAG="armonik-api-cpp:0.1.0" + +# Determine the full path of the script being executed +script_path=$(readlink -f "${BASH_SOURCE:-$0}") + +# Extract the directory from the script path +script_dir=$(dirname $script_path) + +# Print the script directory +echo $script_dir + +# Set the working directory to be three levels above the script directory +working_dir=$script_dir/../../../ + +# Change to the working directory +cd $working_dir + +# Get the full path of the current working directory +working_dir=$(pwd -P) + +# Print a message explaining that we are building the worker image in the root directory +echo "To build worker image. Change to root directory where Protos are" + +# Print the working directory +echo "Change to directory [${working_dir}]" + +# Build the Docker image using the specified Dockerfile and tag it with the IMAGE_TAG variable +docker build --rm -t ${IMAGE_TAG} -f packages/cpp/tools/Dockerfile.worker . diff --git a/packages/cpp/tools/compile.sh b/packages/cpp/tools/compile.sh new file mode 100755 index 000000000..ad33f2e31 --- /dev/null +++ b/packages/cpp/tools/compile.sh @@ -0,0 +1,41 @@ +#!/bin/sh + +set -x + +# Set the image tag for Docker +IMAGE_TAG="${1:-ubuntu-grpc:v0.1}" + +# Get the absolute path of the current script and its directory +script_path="$(readlink -f "${BASH_SOURCE:-$0}")" +script_dir="$(dirname "$script_path")" + +# Set the working directory to the parent directory of the script +working_dir="$script_dir/../" +cd "$working_dir" +working_dir="$(pwd -P)" +cd - + +# Set the path to the protocol buffer (Protos) directory +proto_path="${script_dir}/../../../Protos/V1/" + +# Change to the Protos directory and store its absolute path +cd $proto_path +proto_dir="$(pwd -P)" +cd - + +# Create an install directory and store its absolute path +mkdir -p "${working_dir}/install" +cd "${working_dir}/install" +install_dir="$(pwd -P)" +cd - + +# Change to the working directory +cd "${working_dir}" + +# Check if the Docker image exists, and if not, build it +if [ -z "$(docker images -q "${IMAGE_TAG}" 2> /dev/null)" ]; then + docker build -t "${IMAGE_TAG}" -f tools/Dockerfile.ubuntu . +fi + +# Compile the project source using the Docker image +docker run -v "${proto_dir}:/app/proto" -v "${working_dir}:/app/source" -v "${install_dir}:/app/install" --rm "${IMAGE_TAG}"