Try fetching configuration from the Manager at start / reload

[sophie-cluml]

Implementation Details

• Based on whether a local configuration file is provided via command-line arguments, Crusher will determine whether it needs to fetch configuration from the Manager.
  • If a local configuration file is provided, Crusher will not fetch configuration from the Manager.
  • If a local configuration file is not provided, Crusher will proceed to fetch the configuration from the Manager.
• In cases where a local configuration file is used, the module will operate as it did previously.
• If the configuration needs to be fetched from the Manager, Crusher will attempt to establish a connection to the Manager. However, the following two operations must occur before the connection is established:
  • Parsing the necessary authentication and connection details from the CLI for communication with the Manager.
  • Initializing tracing to log to a temporary directory until the log_dir is defined. Please refer to Adjust Logging Behavior by Revised Logging Policy #117 for logging.
• The configuration fetched from the Manager should be used directly in memory without writing it to a temporary file.
• If fetching the configuration from the Manager fails, Crusher will not continuously retry. Instead, it will wait for a notification sent by the Manager triggered through the applyNode API logic.
• Manual(Added later) integration tests are needed for verifying the connection with the Manager, Giganto, and other key modules. These tests should cover various cases, including normal operations and scenarios where connections fail.

[sophie-cluml]

For log_dir, issue #117 has been created to introduce an updated logging policy for Crusher. I’ve crossed out the outdated log_dir details in the description and added a link to the new issue for reference.

[danbi2990]

@sophie-cluml
During testing, I encountered an edge case. After sending the series data to the Datalake, Crusher writes the last_timestamp_data to a file.

In my opinion, the updated last_timestamp_data should be sent to the Manager server to update the configuration. Otherwise, the next time Crusher fetches the configuration, the last_timestamp_data would be outdated.

As far as I know, it is currently impossible for an agent to update the configuration on the Manager server. I wonder if this issue can be handled without requiring Crusher to update the configuration directly.

I would appreciate your thoughts on this.

[sophie-cluml]

Thank you for raising the point. I think that point is valid. In that sense, I think it's worthwhile to invite Kone at this point.
@sehkone Could you guide us on how to handle the case mentioned in the #113 (comment) ?

Besides, in my opinion, on how to address the raised issue, I think we can implement this by opening a new GraphQL API. To be more concrete, two options instantly come to my mind. One is to open a new GraphQL API to be polled by the client. The other is to open a subscription GraphQL API.

Also, @danbi2990 Do you have any suggestions?

[danbi2990]

Assuming the GraphQL client establishes a new connection whenever needed, using the GraphQL API could become a bottleneck when sending series data to the data lake. Reusing a QUIC connection might be more efficient.
On second thought, sending a GraphQL request is acceptable as long as it is not awaited. However, as I understand it, the GraphQL API is considered public, which may not be necessary in this case.
Maybe we could use quinn::Connection::open_uni to send data to the Manager server.

On the other hand, how about moving the last_timestamp_data configuration to the data lake side? I've heard that the Datalake can scale horizontally, so making Crusher stateless—and using the Datalake's database as the single source of truth—could simplify its management.
I believe it would be better to keep the configuration on the Manager server.

That said, my opinions might lack a broader perspective. I’d appreciate hearing about any better options.

[danbi2990]

It was concluded that no update is needed in the Manager server because last_timestamp_data is a file path, not the data itself.

Instead, the configuration name (last_timestamp_data) will be checked to ensure it follows the convention used in other repositories.

[danbi2990]

In the Sensor, Datalake and Semi-supervised Engine repositories, there is no common convention for file paths (only directory configurations).
I think we can either use _file suffix (e.g., last_timestamp_file) or leave it as it is. Please let me know your opinion.