Add New Relic Application agent for catalog

[jbrown-xentity]

User Story

As a catalog administrator, I want need catalog configured with New Relic so that I can can monitor and evaluate blocking requests/weak points in the system.

Acceptance Criteria

• GIVEN the catalog playbook is run on staging and production
  WHEN I look at New Relic UI
  THEN I can see the different applications and machines
  AND a breakdown of the different components of the application

Background

We currently have language-specific playbooks to deploy and configure New Relic for applications. However, this has some deficiencies. The platform has to know details about the application, environments, app names, configuration settings. The platform has to know where to install hooks within the application (wsgi, worker scripts, etc). The platform might install conflicting versions of dependencies, outside of application pinned dependencies. The platform installing New Relic might cause an extra restart, prolonging any service disruption during deploy.

Related to #174.

Security Considerations (required)

None, New Relic is already installed, just tweaking the configuration.

Sketch

• Remove the python-agent-ansible role from catalog-next playbook
• Add new relic python to dependency list
• Add new relic addition/configuration to gunicorn (look for best practices)
• Use environment variables instead of config file so usage depends on environment and not changing config file (see current config for what can/should be overwritten)
  • Turn off monitor mode in local environment.

[jbrown-xentity]

Specific to catalog, we need to:

• Remove the python-agent-ansible role from catalog-next playbook
• Add new relic python to dependency list
• Add new relic addition/configuration to gunicorn (look for best practices)
• Use environment variables instead of config file so usage depends on environment and not changing config file (see current config for what can/should be overwritten)
  • Turn off monitor mode in local environment.

[jbrown-xentity]

Not sure how to proceed. Gunicorn implementation is only done at the ansible level as part of supervisor, the docker system implements the OKFN docker build which starts the application using UWSGI (or possibly paster serve?). Not sure what the best practice is at this point to add new relic in as part of the application. @adborden

[adborden]

Just summarizing the conversation from sync, not sure if you have any updates since then:

This is probably a two-part story (or two stories). First, let's add a start.sh script that calls newrelic-admin and gunicorn. Then update datagov-deploy to run the start.sh as part of the application (update the supervisor configuration).

The second part would be to get the start.sh integrated into the development workflow. That could end up being a number of things. Maybe we replace OKFN image with a Dockerfile and call start.sh OR maybe it's a feature request for the OKFN image to call a start.sh and then we just overwrite the start.sh file as part of the docker build.

[jbrown-xentity]

Current work is being tested on the feature/new-relic branch.
Was able to get version of the bash script working from supervisor on sandbox. Still need to add the following:

• Configure path to include /usr/lib/ckan/bin/gunicorn and new-relic from python install in ansible, or make script variable to pass full path.
• Add new relic variables in vault as environment variables in the catalog playbook
  • newrelic_license_key
  • newrelic_app_name
  • monitor_mode
  • log_file
  • log_level
  • host (NEW_RELIC_HOST)

[jbrown-xentity]

@adborden it looks like configuring the path is untenable, I will just map variables between docker and ansible in the environment and call it a day. Should I use the environment setting in supervisor to specify the values, as that file is already a template in datagov-deploy and is the go-between on platform and application? Or should this be specified somewhere else, as some kind of script?

Update: The environment installs a .env script into the local directory for the ckan web server script, and the script looks for and tries to read in that file if it exists.

[jbrown-xentity]

Requested added feature upstream.