Monitoring and logging

[akainic]

No description provided.

[akainic]

• Alerting when validation failure occurs (maybe not PagerDuty, look more into CloudWatch)

[akainic]

@markolson @lihanli do we have any updates on this?

[lihanli]

devops added sentry to vets-api, alerts still need to be setup though

[akainic]

Things we need to check off here:

• Create and implement log management plan
• Define log collection and analysis architecture
• Test log collection and analysis architecture
• Define production versus debugging logging standards
• Document method to enable debugging log level on production system
• Test enabling different logging levels
• Verify correct log messages produced at different logging levels

[lihanli]

devops is adding splunk which will be our log collection and analysis architecture https://github.com/department-of-veterans-affairs/devops/issues/617

the rails logger allows different logging levels. we are outputting all log levels in the production log to have access to all possible information.

logger.debug "Person attributes hash: #{@person.attributes.inspect}"
logger.info "Processing the request..."
logger.fatal "Terminating application, raised unrecoverable error!!!"

Document method to enable debugging log level on production system

Rails.application.configure do
  config.log_level = :debug
end

[akainic]

Follow up with devops and make sure we can access the logs and are getting alerts

[akainic]

@knkski wanted to loop you into this ticket and make sure that we can access the logs for edu-benefits and that we're getting alerts for issues. Do we need to get access to anything in particular for this?

[lihanli]

we have access to the logs through cloudwatch. me and mark both have access and tested it out already. i still need to set up sentry access and test out alerts though

[lihanli]

i'm registered on sentry and subscribed to alerts now

[knkski]

If anybody else needs CloudWatch access, just file a ticket over in the devops repo. As far as alerting, we are sending alerts currently whenever the server gives a 500 error. Will the validation failure alerts that you'd like above be included in that?

[akainic]

Got it, thanks @lihanli and @knkski for that information! I think we need some kind of alerting when a validation failure occurs, but I'm not sure what the right tool is for this. @lihanli do you have thoughts on this?

[lihanli]

i made a pr to add the validation errors to sentry which can send out alerts when they happen department-of-veterans-affairs/vets-api#404

[akainic]

Awesome, thanks @lihanli!

[akainic]

As all of these items are checked off now, going to close this ticket.