stats endpoint is timing out

[jakebolam]

Describe the bug
The stats endpoint is timing out:
https://gkioebvccg.execute-api.us-east-1.amazonaws.com/prod/probot/stats

Causing https://probot.github.io/apps/all-contributors/ to show incorrect installations.
https://raw.githubusercontent.com/probot/probot.github.io/master/_apps/all-contributors.md

From the screenshot below:

We can see that the duration of the endpoint has been slowly increasing with the number of installs. Once we crossed the 30 second mark (in the last week of February) the lambda is taking longer to process than the 30second hard timeout from API Gateway, and the gateway times out.

To Reproduce
Steps to reproduce the behavior:

1. Hit end point

Expected behavior
Endpoint returns quickly and promptly to allow stat reporting.

Screenshots

[jakebolam]

Probot has the same issue: probot/probot#380

[jakebolam]

I've just merged #133 to add in-memory caching, this does not fix the root issue, and is not a long term solution to have accurate reporting on https://probot.github.io/apps/all-contributors/

(During the period we expect the sync to happen, also added a cron to keep the lambda warm: #169)

[jakebolam]

@gr2m any idea how we could make the endpoint code more efficient? i.e. are there endpoints from GitHub we should be using instead

https://github.com/all-contributors/all-contributors-bot/blob/master/src/serverless-stats.js

[gr2m]

As mentioned in Slack

I think instead of using the app to get these stats, I’d use something like Travis Cron job to get the stats on a schedule. With the latest Octokit & throttle/retry plugins, it shouldn’t be a problem in terms of hitting rate / abuse limits, but it will take a long time

I would turn off stats using the environment variable, I don’t like the endpoint to be in probot at all the way it is. GitHub should provide better APIs to get these statistics, and they are aware of the problem.

[gr2m]

[jakebolam]

Shipped in #175