webhook fails with 500

[mmmaly]

Madrill reports, and Rrnning the webook from command line confirms, that server ends with:

500 Internal Server Error

more info in mail.

[martinmacko47]

access.log

"POST /mandrill/webhook/?secret=Kcf*****jle HTTP/1.1" 500 5829 "-" "Mandrill-Webhook/1.0"

error.log

mod_wsgi (pid=10854): Exception occurred processing WSGI script '/var/www/chcemvediet/chcemvediet/chcemvediet/wsgi.py'.
Traceback (most recent call last):
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 187, in __call__
    response = self.get_response(request)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/core/handlers/base.py", line 199, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/core/handlers/base.py", line 231, in handle_uncaught_exception
    'request': request
  File "/usr/lib/python2.7/logging/__init__.py", line 1178, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/lib/python2.7/logging/__init__.py", line 1271, in _log
    self.handle(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 1281, in handle
    self.callHandlers(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 1321, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 749, in handle
    self.emit(record)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/utils/log.py", line 132, in emit
    connection=self.connection())
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/core/mail/__init__.py", line 98, in mail_admins
    mail.send(fail_silently=fail_silently)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/core/mail/message.py", line 286, in send
    return self.get_connection(fail_silently).send_messages([self])
  File "/var/www/chcemvediet/chcemvediet/poleno/mail/backend.py", line 20, in send_messages
    self._enqueue(message)
  File "/var/www/chcemvediet/chcemvediet/poleno/mail/backend.py", line 84, in _enqueue
    msg.save()
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/base.py", line 589, in save
    force_update=force_update, update_fields=update_fields)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/base.py", line 617, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/base.py", line 698, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/base.py", line 731, in _do_insert
    using=using, raw=raw)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/manager.py", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/query.py", line 921, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 920, in execute_sql
    cursor.execute(sql, params)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 129, in execute
    return self.cursor.execute(query, args)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (2006, 'MySQL server has gone away')

[martinmacko47]

Ok, tak to nie je skutocna chyba. Toto je chyba, ze sa nepodarilo reportovat inu chybu, ktora sa snazila reportovat, ze ina chyba, ktora sa snazila reportovat, ... az kym databaza netimeoutla. Je chyba, ze sa snazime kazdu chybu hned reportnut a ked sa nepodari odoslat email s chybou, tak sa to zacykli. Mali sme chyby pekne akumulovat a posielat raz za cas, ako som povodne mal.

[martinmacko47]

Este ze git vie revertnut aj random commit daleko zahrabany v historii: Revert "Use standard Django error reports"

[martinmacko47]

Skutocna chyba:

[2016-03-21 23:08:21,953] django.request ERROR Internal Server Error: /mandrill/webhook/
Traceback (most recent call last):
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/views/decorators/http.py", line 41, in inner
    return func(request, *args, **kwargs)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/www/chcemvediet/chcemvediet/poleno/utils/views.py", line 78, in wrapped_view
    return view(request, *args, **kwargs)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/transaction.py", line 394, in inner
    return func(*args, **kwargs)
  File "/var/www/chcemvediet/chcemvediet/poleno/mail/transports/mandrill/views.py", line 66, in webhook
    webhook_event.send(sender=None, event_type=event['event'], data=event)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 198, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/var/www/chcemvediet/chcemvediet/poleno/mail/transports/mandrill/signals.py", line 87, in inbound_email_webhook_event
    message.save()
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/base.py", line 589, in save
    force_update=force_update, update_fields=update_fields)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/base.py", line 617, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/base.py", line 698, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/base.py", line 731, in _do_insert
    using=using, raw=raw)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/manager.py", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/query.py", line 921, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 920, in execute_sql
    cursor.execute(sql, params)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 134, in execute
    six.reraise(utils.IntegrityError, utils.IntegrityError(*tuple(e.args)), sys.exc_info()[2])
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 129, in execute
    return self.cursor.execute(query, args)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/var/www/chcemvediet/chcemvediet/env/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
IntegrityError: (1048, "Column 'text' cannot be null")

[martinmacko47]

Hadam fixnute. Fix som deployol. Pockam kym to mandrill submitne opatovne, resp. pls klikni v mandrill rozhrani, nech to resubmitne. Rucne cez curl to uz nesubmituj, nech nevyrobime nasobne maily.

Od teraz by nam aj naozaj mali chodit maily s chybami na servri. Raz za hodinu report s tym, co sa za ten cas stalo.

[mmmaly]

Vyzera, ze preslo. Akurat tam svieti este ina (starsia) chyba, 413 Request Entity Too Large, neviem, ci to s tym suvisi.

[mmmaly]

aha, uh, 31MB. To asi nemohlo prejst. Vieme to pozriet, ze co to je, kto to posiela?

[martinmacko47]

Od vcera vecera nejake webhooky boli a zatial ziaden nespadol. Ak nejaky spadne mal by prijst mail na admin@chv, ale pre istotu pozriem aj logy na servri.

To 31MB je z akeho datumu? Po neprejde ani cez apache, kedze uz tam je nizsi limit. Ak si vies z mandrillu stiahnut mandrill-webhook.txt pre ten request, tak si ho vies rucne rozparsovat a pozriet, co poslali. Je to len urlencoded json. Ale ak je to nejaky stary webhook, tak nemyslim, ze mandrill bude tie data drzat.

Otazka je, co s tym webhookom potom mandrill spravil. Zrejme to casom vzdal postovat. Co potom spravil s mailom? Vratil bounce ako nedoruceny, alebo len zahodil?

[martinmacko47]

Po polroku mozeme povedat, ze to asi funguje.