Automatically create monitors when using Redbeat

[bensaufley]

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

We're using Python with Celery and Redbeat and following the docs for setup. Using either the autodiscover or manual @monitor instrumentation outlined in the docs, we are not having monitors in our Sentry UI automatically created. But manually creating them confirms that the data does come into Sentry—as soon as they're created we can see data for them in the UI, but only from that point forward.

I've been talking to Sentry support about this and was asked to create an Issue. Unfortunately our repo is private but our setup is pretty straightforward. Here's the gist of what we've got:

requirements.txt:

celery==5.2.6
celery-redbeat==2.0.0

worker.py

from celery import Celery
from celery.signals import celeryd_init
from celery.schedules import crontab
import sentry_sdk
from sentry_sdk.crons.decorator import monitor
from sentry_sdk.integrations.celery import CeleryIntegration

from myco import _my_celery_task, get_env_var, LockInUse, get_redis_url

app = Celery(__name__, broker=get_redis_url(database='celery'))

@celeryd_init.connect
def init_sentry(**_kwargs):
    env = get_env_var('APP_ENV', False)
    if not env:
        print('APP_ENV not set, defaulting to development')
        env = 'development'
    if env not in ['development', 'test']:
        sentry_sdk.init(
            dsn="<[email protected]>", # obviously this is redacted

            environment=env,
            release=get_env_var('SENTRY_RELEASE_VERSION', False) or 'unknown',

            ignore_errors=[LockInUse],

            integrations=[CeleryIntegration()],

            # Set traces_sample_rate to 1.0 to capture 100%
            # of transactions for performance monitoring.
            # We recommend adjusting this value in production,
            traces_sample_rate=0.005,
        )

# ...


@app.task
@monitor(monitor_slug='my_celery_task')
def my_celery_task():
    return _my_celery_task()

# ...

app.conf.redbeat_redis_url = get_redis_url(database='celery')
app.conf.beat_scheduler = 'redbeat.RedBeatScheduler'
app.conf.beat_schedule = {
    'my-celery-task': {
        'task': 'worker.my_celery_task',
        'schedule': crontab(minute='0', hour='0', day_of_week='1'),
    },
    # ...
}

Expected Result

Cron job monitors are automatically created in Sentry on deploy, or at least when they execute

Actual Result

No monitors are created

Product Area

Crons

Link

No response

DSN

No response

Version

No response

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[getsantry]

Routing to @getsentry/product-owners-crons for triage ⏲️

[gaprl]

@antonpirker could you take a look into this please when you're back?

[IevgeniiB]

Hi! Just adding my observations because I have been directed to this ticket by support.

I noticed a similar problem - I am using cron monitors for multiple automations that don't use celery. I used to have celery and django integrations enabled by default (because of historical reasons that don't make sense anymore) and decided to disable them for my scripts. It resulted in a better performance on the cron monitors' side.

See the screenshot below:

The gray line is at the point when the integrations were disabled.

As you can see - it significantly improved the stability of crons, but didn't fix the issue entirely. Is there anything else in addition to celery that I should watch out for?

The version of sentry sdk for python that I use is 1.39.1

[antonpirker]

@IevgeniiB please create a issue in https://github.com/getsentry/sentry-python/ because it is something different the this issue. Thanks!

[antonpirker]

@bensaufley looks like task creation does not work when using redbeat. I will have a look.

[antonpirker]

What I know now:

For Celery beat we patch celery.Beat.Scheduler.apply_entry to add the monitor-config to check-ins. In Redbeats scheduler this function is not used, but instead this: https://github.com/sibson/redbeat/blob/v2.0.0/redbeat/schedulers.py#L450

So we need to patch this function to make it work when Redbeats scheduler is used.

[IevgeniiB]

@IevgeniiB please create a issue in https://github.com/getsentry/sentry-python/ because it is something different the this issue. Thanks!

@antonpirker thank you! I've submitted a new issue with more info here: #2617

[antonpirker]

Ok, we will add support for Redbeat. When it will be done, this issue will be updated of course.