Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hookup for log/event messages to history server #218

Merged
merged 6 commits into from
Dec 3, 2024
Merged

Conversation

amitschang
Copy link
Member

@amitschang amitschang commented Nov 25, 2024

To utilize the existing history server for logs in addition to sensor data, the following steps were taken:

  • namespace all included loggers with the package name. This is done so we can modify the logger without affecting root
  • add a log handler class that emits to the history API
  • add a hookup between the above two within the device manager, along with config options
  • modify the history api to take a "kind" and "vial" parameter - kind differentiates between sensor reads and logs, and vial enables logs to be tied to a specific vial
  • change shape of sensor data to be row-per-vial, instead of row containing struct of vials (will require corresponding UI mod)

Resolves: #131, Resolves #196
Enables/relates to: #109, #132

Some examples:

curl -Hcontent-type:application/json -XPOST localhost:8080/history/'?t_start=1732570420&t_stop=1732570440' -d'{"kinds": ["event", "sensor"], "vials": [0, 1]}' | jq .
{
  "data": {
    "test": [
      {
        "timestamp": 1732570421.0073779,
        "kind": "sensor",
        "vial": 0,
        "data": {
          "name": "NoOpSensorDriver",
          "vial": 0,
          "raw": 1,
          "value": 2
        }
      },
      {
        "timestamp": 1732570421.009811,
        "kind": "sensor",
        "vial": 1,
        "data": {
          "name": "NoOpSensorDriver",
          "vial": 1,
          "raw": 1,
          "value": 2
        }
      }
    ]
  }
}
curl -Hcontent-type:application/json -XPOST localhost:8080/history/'?n_max=2' -d'{"kinds": ["log"]}
' | jq . 
{
  "data": {
    "test": [
      {
        "timestamp": 1732569561.055786,
        "kind": "log",
        "vial": null,
        "data": {
          "level": "WARNING",
          "message": "test from HW"
        }
      }
    ],
    "evolver": [
      {
        "timestamp": 1732569561.033202,
        "kind": "log",
        "vial": null,
        "data": {
          "level": "ERROR",
          "message": "Error in loop: committing proposals for <MagicMock spec='EffectorDriver' id='4987471824'>"
        }
      }
    ]
  }
}

Copy link

codecov bot commented Nov 25, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

📢 Thoughts on this report? Let us know!

@amitschang amitschang marked this pull request as ready for review November 26, 2024 19:51
@amitschang amitschang requested a review from a team as a code owner November 26, 2024 19:51
@amitschang amitschang added enhancement New feature or request history logging labels Nov 26, 2024
jamienoss
jamienoss previously approved these changes Dec 3, 2024
Copy link
Member

@jamienoss jamienoss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

Just some comments/suggestions.

Comment on lines +106 to +111
if data := device.get():
if isinstance(data, dict):
for vial, output in data.items():
self.history.put(name, "sensor", output, vial=vial)
else:
self.history.put(name, "sensor", data, vial=getattr(data, "vial", None))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[comment] I can see this being duplicated if logging is needed from somewhere else. It would be better if self.history.put could handle this.

@@ -8,6 +8,8 @@

class HistoricDatum(BaseModel):
timestamp: float
kind: str
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[comment] This would benefit from having a description.

vial = getattr(record, "vial", None)
record_dict = {"level": record.levelname, "message": record.getMessage()}
# TODO: add support for some extra fields (but these should be json-seralizable)
kind = "event" if record.levelno == EVENT else "log"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[comment] It might help the TODO implementer if we added here and now, how we might envision coding this, e.g., using a match, an enum, a mapping etc?

evolver/base.py Outdated
@@ -312,7 +312,7 @@ def init_descriptors(self, **non_config_kwargs):
init_and_set_vars_from_descriptors(self, **non_config_kwargs)

def _setup_logger(self):
self.logger = logging.getLogger(self.name)
self.logger = logging.getLogger(f"{__package__}.{self.name}")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[comment] It might be slightly more robust against future changes if we added a setting for this rather than relying on the convention of it always being __package__.
E.g.:

# evolver.settings
class Settings(...):
  ROOT_LOGGER_NAME: str = __package__

@amitschang
Copy link
Member Author

Thanks. I made some corresponding changes, I think scope of them is small enough I will consider approval sticky and merge this in 😄

@amitschang amitschang merged commit e4dc599 into main Dec 3, 2024
7 checks passed
@amitschang amitschang deleted the gh-131-log-history branch December 3, 2024 17:38
amitschang added a commit to ssec-jhu/evolver-ui that referenced this pull request Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request history logging
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Record abort events Add endpoint to retrieve logs
2 participants