diff --git a/docs/getting_started.rst b/docs/getting_started.rst index b2b47cd53b6..a7d40589736 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -83,33 +83,66 @@ Then in your ``conf.py``: .. _this blog post: http://ericholscher.com/blog/2016/mar/15/dont-use-markdown-for-technical-docs/ +.. _connect-account: + +Sign Up and Connect an External Account +--------------------------------------- + +.. TODO Update this with GitLab support later + +If you are going to import a repository from GitHub or Bitbucket, you should +connect your account to your provider first. Connecting your account allows for +easier importing and enables Read the Docs to configure your repository webhooks +automatically. + +To connect your account, got to your *Settings* dashboard and select *Connected +Services*. From here, you'll be able to connect to your GitHub or Bitbucket +account. This process will ask you to authorize a connection to Read the Docs, +that allows us to read information about and clone your repositories. + .. _import-docs: Import Your Docs ---------------- -`Sign up`_ for an account on RTD, then `log in`_. Visit your dashboard_ and click -Import_ to add your project to the site. Fill in the name and description, then -specify where your repository is located. This is normally the URL or path name -you'd use to checkout, clone, or branch your code. Some examples: +To import a repository, visit your dashboard_ and click Import_. + +If you have a connected account, you will see a list of your repositories that +we are able to import. To import one of these projects, just click the import +icon next to the repository you'd like to import. This will bring up a form that +is already filled with your project's information. Feel free to edit any of +these properties, and the click **Next** to build your documentation. + +Manually Import Your Docs +~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you do not have a connected account, you will need select **Import Manually** +and enter the information for your repository yourself. You will also need to +manually configure the webhook for your repository as well. When importing your +project, you will be asked for the repository URL, along with some other +information for you new project. The URL is normally the URL or path name you'd +use to checkout, clone, or branch your repository. Some examples: * Git: ``http://github.com/ericholscher/django-kong.git`` -* Subversion: ``http://varnish-cache.org/svn/trunk`` * Mercurial: ``https://bitbucket.org/ianb/pip`` +* Subversion: ``http://varnish-cache.org/svn/trunk`` * Bazaar: ``lp:pasta`` -Add an optional homepage URL and some tags, then click "Create". +Add an optional homepage URL and some tags, and then click **Next**. + +Once your project is created, you'll need to manually configure the repository +webhook if you would like to have new changesets to trigger builds for your +project on Read the Docs. Go to your project's **Integrations** page to +configure a new webhook, or see :ref:`our steps for webhook creation +` for more information on this process. Within a few seconds your code will automatically be fetched from your public repository, and the documentation will be built. Check out our :doc:`builds` page to learn more about how we build your docs, and to troubleshoot any issues that arise. -If you want to keep your code updated as you commit, -configure your code repository to hit our `Post Commit Hooks`_. -This will rebuild your docs every time you push your code. - -We support multiple versions of your code. You can read more about how to use this well on our :doc:`versions` page. +Read the Docs will host multiple versions of your code. You can read more about +how to use this well on our :doc:`versions` page. If you have any more trouble, don't hesitate to reach out to us. The :doc:`support` page has more information on getting in touch. @@ -126,4 +159,3 @@ If you have any more trouble, don't hesitate to reach out to us. The :doc:`suppo .. _log in: http://readthedocs.org/accounts/login .. _dashboard: http://readthedocs.org/dashboard .. _Import: http://readthedocs.org/dashboard/import -.. _Post Commit Hooks: http://readthedocs.org/docs/read-the-docs/en/latest/webhooks.html diff --git a/docs/webhooks.rst b/docs/webhooks.rst index d22cac4688b..24e9b9279cb 100644 --- a/docs/webhooks.rst +++ b/docs/webhooks.rst @@ -1,78 +1,135 @@ Webhooks ======== -Webhooks are pretty amazing, and help to turn the web into a push instead of -pull platform. We have support for hitting a URL whenever you commit to your -project and we will try and rebuild your docs. This only rebuilds them if -something has changed, so it is cheap on the server side. As anyone who has -worked with push knows, pushing a doc update to your repo and watching it get -updated within seconds is an awesome feeling. +The primary method that Read the Docs uses to detect changes to your +documentation is through the use of *webhooks*. Webhooks are configured with +your repository provider, such as GitHub or Bitbucket, and with each commit, +merge, or other change to your repository, Read the Docs is notified. When we +receive a webhook notification, we determine if the change is related to an +active version for your project, and if it is, a build is triggered for that +version. -GitHub ---------- +.. _integration-detail: -If your project is hosted on GitHub, you can easily add a hook that will rebuild -your docs whenever you push updates: +Webhook Integrations +-------------------- -* Go to the "Settings" page for your project -* Click "Integrations & Services" -* In the "Services" section, click "Add service" -* In the list of available services, click "ReadTheDocs" -* Leave "Active" checked -* Click "Add service" +You'll find a list of configured webhook integrations on your project's admin +dashboard, under **Integrations**. You can select any of these integrations to +see the *integration detail page*. This page has additional configuration +details and a list of HTTP exchanges that have taken place for the integration. -.. note:: The GitHub URL in your Read the Docs project must match the URL on GitHub. The URL is case-sensitive. +.. _webhook-creation: -If you ever need to manually set the webhook on GitHub, -you can point it at ``https://readthedocs.org/github``. +Webhook creation +---------------- -Bitbucket ------------ +If you import a project using a :ref:`connected account `, a +webhook will be set up automatically for your repository. However, if your +project was not imported through a connected account, you may need to +manually configure a webhook for your project. + +To manually set up a webhook, click **Add integration** on your project's +**Integrations** admin dashboard page and select the integration type you'd like +to add. After you have added the integration, you'll see a URL for the +integration on the :ref:`integration detail page `. Use this +URL when setting up a new webhook with your provider -- these steps vary +depending on the provider: + +GitHub +~~~~~~ + +* Go to the **Settings** page for your project +* Click **Webhooks** and then **Add webhook** +* For **Payload URL**, use the URL of the integration on Read the Docs, found on + the :ref:`integration detail page ` page +* For **Content type**, both *application/json* and + *application/x-www-form-urlencoded* work +* Select **Just the push event** +* Finish by clicking **Add webhook** -If your project is hosted on Bitbucket, you can easily add a hook that will rebuild -your docs whenever you push updates: +.. note:: The webhook secret is not yet respected -* Go to the "admin" page for your project -* Click "Services" -* In the available service hooks, select "Read the Docs" -* Click "Add service" +Bitbucket +~~~~~~~~~ -If you ever need to manually set the webhook on Bitbucket, -you can point it at ``https://readthedocs.org/bitbucket``. +* Go to the **Settings** page for your project +* Click **Webhooks** and then **Add webhook** +* For **URL**, use the URL of the integration on Read the Docs, found on the + :ref:`integration detail page ` page +* Under **Triggers**, **Repository push** should be selected +* Finish by clicking **Save** GitLab ---------- +~~~~~~ + +* Go to the **Settings** page for your project +* Click **Integrations** +* For **URL**, use the URL of the integration on Read the Docs, found on the + :ref:`integration detail page ` page +* Leave the default **Push events** selected +* Finish by clicking **Add Webhook** -If your project is hosted on GitLab, you can easily add a hook that will rebuild -your docs whenever you push updates. +Using the generic API integration +--------------------------------- -* Go to the "Settings" page for your project -* Click "Integrations" -* In the "URL" section, enter ``https://readthedocs.org/gitlab`` -* Leave the default "Push events" selected -* Click "Add Webhook" +For repositories that are not hosted with a supported provider, we also offer a +generic API endpoint for triggering project builds. Similar to webhook +integrations, this integration has a specific URL, found on the +:ref:`integration detail page `. -Others ------- +Token authentication is required to use the generic endpoint, you will find this +token on the integration details page. The token should be passed in as a +request parameter, either as form data or as part of JSON data input. -Your ReadTheDocs project detail page has your post-commit hook on it; it will -look something along the lines of ``http://readthedocs.org/build/``. -Regardless of which revision control system you use, you can just hit this URL -to kick off a rebuild. +Parameters +~~~~~~~~~~ -The following parameters available to customize the behavior of custom webhooks: +This endpoint accepts the following arguments during an HTTP POST: -* ``'version_slug'``: The build version to trigger build for (defaults to ``'latest'``) +branches + The names of the branches to trigger builds for. This can either be an array + of branch name strings, or just a single branch name string. - Example:: + Default: **latest** - $ curl -X POST --data "version_slug=$VERSION" https://readthedocs.org/build/$PROJECT_NAME +token + The integration token. You'll find this value on the + :ref:`integration detail page ` page. -You could make this part of a hook using Git_, Subversion_, Mercurial_, or -Bazaar_, perhaps through a simple script that accesses the build URL using -``wget`` or ``curl``. +For example, the cURL command to build the ``dev`` branch, using the token +``1234``, would be:: + + curl -X POST -d "branches=dev" -d "token=1234" https://readthedocs.org/api/v2/webhook/example-project/1/ + +A command like the one above could be called from a cron job or from a hook +inside Git_, Subversion_, Mercurial_, or Bazaar_. .. _Git: http://www.kernel.org/pub/software/scm/git/docs/githooks.html .. _Subversion: http://mikewest.org/2006/06/subversion-post-commit-hooks-101 .. _Mercurial: http://hgbook.red-bean.com/read/handling-repository-events-with-hooks.html .. _Bazaar: http://wiki.bazaar.canonical.com/BzrHooks + +Authentication +~~~~~~~~~~~~~~ + +This endpoint requires authentication. If authenticating with an integration +token, a check will determine if the token is valid and matches the given +project. If instead an authenticated user is used to make this request, a check +will be performed to ensure the authenticated user is an owner of the project. + +Debugging webhooks +------------------ + +If you are experiencing problems with an existing webhook, you may be able to +use the integration detail page to help debug the issue. Each project +integration, such as a webhook or the generic API endpoint, stores the HTTP +exchange that takes place between Read the Docs and the external source. You'll +find a list of these exchanges in any of the integration detail pages. + +Resyncing webhooks +------------------ + +It might be necessary to re-establish a webhook if you are noticing problems. +To resync a webhook from Read the Docs, visit the integration detail page and +follow the directions for re-syncing your repository webhook. diff --git a/media/css/core.css b/media/css/core.css index 30b2e23b947..6e921acff5d 100644 --- a/media/css/core.css +++ b/media/css/core.css @@ -119,6 +119,12 @@ h3 > span.link-help { float: right; } +form.form-wide input[type='text'], +form.form-wide select, +form.form-wide textarea { + width: 100%; +} + /* content */ #content { padding-top: 50px; } @@ -925,6 +931,22 @@ body .edit-toggle { display: none; } .navigable ul input[type=text] { width: 164px; } +div.button-bar ul { + list-style: none; + text-align: right; +} + +div.button-bar ul li { + display: inline-block; +} + +div.button-bar li a.button, +div.button-bar li input[type="submit"], +div.button-bar li input[type="button"], +div.button-bar li button { + margin-top: .5em; + margin-bottom: .5em; +} select.dropdown { display: none; } .dropdown > a { font-family: "ff-meta-web-pro", "ff-meta-web-pro-1", "ff-meta-web-pro-2", Arial, "Helvetica Neue", sans-serif; color: #666; font-weight: bold; padding: 8px 15px; border: none; background: #e6e6e6 url(../images/gradient.png) repeat-x bottom left; margin: 30px 5px 20px 0; text-shadow: 0 1px 0 rgba(255, 255, 255, 1); border: 1px solid #bfbfbf; display: block; text-decoration: none; box-shadow: 0 1px 0 rgba(0, 0, 0, 0.1), 0 1px 0 rgba(255, 255, 255, 0.5) inset; -moz-box-shadow: 0 1px 0 rgba(0, 0, 0, 0.1), 0 1px 0 rgba(255, 255, 255, 0.5) inset; -webkit-box-shadow: 0 1px 0 rgba(0, 0, 0, 0.1), 0 1px 0 rgba(255, 255, 255, 0.5) inset; } @@ -1044,28 +1066,40 @@ div.module-list-wrapper.httpexchanges li span.status.status-fail { background: #a55; } +div.integration-details { + margin: 1em; +} + +div.integration-details dl dt, div.httpexchange dl dt { - display: inline-block; - font-weight: bold; - font-family: 'inconsolata', 'bitstream vera sans mono', 'andale mono', 'lucida console', monospace; - font-size: .9em; + display: inline-block; + font-weight: bold; +} +div.httpexchange dl dt { + font-family: 'inconsolata', 'bitstream vera sans mono', 'andale mono', 'lucida console', monospace; + font-size: .9em; } +div.integration-details dl dd, +div.httpexchange dl dd { + display: inline; + font-family: 'inconsolata', 'bitstream vera sans mono', 'andale mono', 'lucida console', monospace; +} div.httpexchange dl dd { - display: inline; - font-family: 'inconsolata', 'bitstream vera sans mono', 'andale mono', 'lucida console', monospace; - font-size: .9em; + font-size: .9em; } + +div.integration-details dl dd:after, div.httpexchange dl dd:after { - display: block; - content: ''; + display: block; + content: ''; } div.httpexchange div.highlight pre { - padding: 1em; - background: #f4f4f4; - border: 1px solid #ccc; - font-size: .9em; + padding: 1em; + background: #f4f4f4; + border: 1px solid #ccc; + font-size: .9em; } /* Pygments */ diff --git a/readthedocs/constants.py b/readthedocs/constants.py index 03c2fa531b4..35cca627432 100644 --- a/readthedocs/constants.py +++ b/readthedocs/constants.py @@ -9,4 +9,5 @@ 'lang_slug': LANGUAGES_REGEX, 'version_slug': VERSION_SLUG_REGEX, 'filename_slug': '(?:.*)', + 'integer_pk': r'[\d]+', } diff --git a/readthedocs/core/fields.py b/readthedocs/core/fields.py new file mode 100644 index 00000000000..822ad5ca151 --- /dev/null +++ b/readthedocs/core/fields.py @@ -0,0 +1,9 @@ +"""Shared model fields and defaults""" + +import binascii +import os + + +def default_token(): + """Generate default value for token field""" + return binascii.hexlify(os.urandom(20)).decode() diff --git a/readthedocs/integrations/admin.py b/readthedocs/integrations/admin.py new file mode 100644 index 00000000000..1bbdaed68a0 --- /dev/null +++ b/readthedocs/integrations/admin.py @@ -0,0 +1,110 @@ +"""Integration admin models""" + +from django.contrib import admin +from django.core import urlresolvers +from django.utils.safestring import mark_safe +from pygments.formatters import HtmlFormatter + +from .models import Integration, HttpExchange + + +def pretty_json_field(field, description, include_styles=False): + # There is some styling here because this is easier than reworking how the + # admin is getting stylesheets. We only need minimal styles here, and there + # isn't much user impact to these styles as well. + def inner(_, obj): + styles = '' + if include_styles: + formatter = HtmlFormatter(style='colorful') + styles = '' + return mark_safe('
{1}
{2}'.format( + 'float: left;', + obj.formatted_json(field), + styles, + )) + + inner.short_description = description + return inner + + +class HttpExchangeAdmin(admin.ModelAdmin): + + """Admin model for HttpExchange + + This adds some read-only display to the admin model. + """ + + readonly_fields = [ + 'date', + 'status_code', + 'pretty_request_headers', + 'pretty_request_body', + 'pretty_response_headers', + 'pretty_response_body', + ] + fields = readonly_fields + list_display = [ + 'related_object', + 'date', + 'status_code', + 'failed_icon', + ] + + pretty_request_headers = pretty_json_field( + 'request_headers', + 'Request headers', + include_styles=True, + ) + pretty_request_body = pretty_json_field( + 'request_body', + 'Request body', + ) + pretty_response_headers = pretty_json_field( + 'response_headers', + 'Response headers', + ) + pretty_response_body = pretty_json_field( + 'response_body', + 'Response body', + ) + + def failed_icon(self, obj): + return not obj.failed + + failed_icon.boolean = True + failed_icon.short_description = 'Passed' + + +class IntegrationAdmin(admin.ModelAdmin): + + """Admin model for Integration + + Because of some problems using JSONField with admin model inlines, this + instead just links to the queryset. + """ + + search_fields = ('project__slug', 'project__name') + readonly_fields = ['exchanges'] + + def exchanges(self, obj): + """Manually make an inline-ish block + + JSONField doesn't do well with fieldsets for whatever reason. This is + just to link to the exchanges. + """ + url = urlresolvers.reverse('admin:{0}_{1}_changelist'.format( + HttpExchange._meta.app_label, # pylint: disable=protected-access + HttpExchange._meta.model_name, # pylint: disable=protected-access + )) + return mark_safe('{3} HTTP transactions'.format( + url, + 'integrations', + obj.pk, + obj.exchanges.count(), + )) + + exchanges.short_description = 'HTTP exchanges' + + +admin.site.register(HttpExchange, HttpExchangeAdmin) +admin.site.register(Integration, IntegrationAdmin) diff --git a/readthedocs/integrations/migrations/0002_add-webhook.py b/readthedocs/integrations/migrations/0002_add-webhook.py new file mode 100644 index 00000000000..9d215886f58 --- /dev/null +++ b/readthedocs/integrations/migrations/0002_add-webhook.py @@ -0,0 +1,26 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.9.12 on 2017-03-29 21:29 +from __future__ import unicode_literals + +from django.db import migrations, models +import django.db.models.deletion +import jsonfield.fields + + +class Migration(migrations.Migration): + + dependencies = [ + ('integrations', '0001_add_http_exchange'), + ] + + operations = [ + migrations.CreateModel( + name='Integration', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('integration_type', models.CharField(choices=[(b'github_webhook', 'GitHub incoming webhook'), (b'bitbucket_webhook', 'Bitbucket incoming webhook'), (b'gitlab_webhook', 'GitLab incoming webhook'), (b'api_webhook', 'Generic API incoming webhook')], max_length=32, verbose_name='Type')), + ('provider_data', jsonfield.fields.JSONField(verbose_name='Provider data')), + ('project', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='integrations', to='projects.Project')), + ], + ), + ] diff --git a/readthedocs/integrations/models.py b/readthedocs/integrations/models.py index d0a0abcc4a4..80d8d8a9342 100644 --- a/readthedocs/integrations/models.py +++ b/readthedocs/integrations/models.py @@ -2,18 +2,21 @@ import json import uuid +import re from django.db import models, transaction -from django.utils.translation import ugettext_lazy as _ -from django.contrib.contenttypes.fields import GenericForeignKey +from django.contrib.contenttypes.fields import GenericForeignKey, GenericRelation from django.contrib.contenttypes.models import ContentType from django.utils.safestring import mark_safe +from django.utils.translation import ugettext_lazy as _ from rest_framework import status from jsonfield import JSONField from pygments import highlight from pygments.lexers import JsonLexer from pygments.formatters import HtmlFormatter +from readthedocs.core.fields import default_token +from readthedocs.projects.models import Project from .utils import normalize_request_payload @@ -21,6 +24,12 @@ class HttpExchangeManager(models.Manager): """HTTP exchange manager methods""" + # Filter rules for request headers to remove from the output + REQ_FILTER_RULES = [ + re.compile('^X-Forwarded-.*$', re.I), + re.compile('^X-Real-Ip$', re.I), + ] + @transaction.atomic def from_exchange(self, req, resp, related_object, payload=None): """Create object from Django request and response objects @@ -54,6 +63,11 @@ def from_exchange(self, req, resp, related_object, payload=None): if key.startswith('HTTP_') ) request_headers['Content-Type'] = req.content_type + # Remove unwanted headers + for filter_rule in self.REQ_FILTER_RULES: + for key in request_headers.keys(): + if filter_rule.match(key): + del request_headers[key] response_payload = resp.data if hasattr(resp, 'data') else resp.content try: @@ -75,14 +89,16 @@ def from_exchange(self, req, resp, related_object, payload=None): return obj def delete_limit(self, related_object, limit=10): - # pylint: disable=protected-access - queryset = self.filter( - content_type=ContentType.objects.get( - app_label=related_object._meta.app_label, - model=related_object._meta.model_name, - ), - object_id=related_object.pk - ) + if isinstance(related_object, Integration): + queryset = self.filter(integrations=related_object) + else: + queryset = self.filter( + content_type=ContentType.objects.get( + app_label=related_object._meta.app_label, # pylint: disable=protected-access + model=related_object._meta.model_name, # pylint: disable=protected-access + ), + object_id=related_object.pk + ) for exchange in queryset[limit:]: exchange.delete() @@ -126,7 +142,9 @@ def formatted_json(self, field): """Try to return pretty printed and Pygment highlighted code""" value = getattr(self, field) or '' try: - json_value = json.dumps(json.loads(value), sort_keys=True, indent=2) + if not isinstance(value, dict): + value = json.loads(value) + json_value = json.dumps(value, sort_keys=True, indent=2) formatter = HtmlFormatter() html = highlight(json_value, JsonLexer(), formatter) return mark_safe(html) @@ -140,3 +158,156 @@ def formatted_request_body(self): @property def formatted_response_body(self): return self.formatted_json('response_body') + + +class IntegrationQuerySet(models.QuerySet): + + """Return a subclass of Integration, based on the integration type + + .. note:: + This doesn't affect queries currently, only fetching of an object + """ + + def _get_subclass(self, integration_type): + # Build a mapping of integration_type -> class dynamically + class_map = dict( + (cls.integration_type_id, cls) + for cls in self.model.__subclasses__() + if hasattr(cls, 'integration_type_id') + ) + return class_map.get(integration_type) + + def _get_subclass_replacement(self, original): + """Replace model instance on Integration subclasses + + This is based on the ``integration_type`` field, and is used to provide + specific functionality to and integration via a proxy subclass of the + Integration model. + """ + cls_replace = self._get_subclass(original.integration_type) + if cls_replace is None: + return original + new = cls_replace() + for k, v in original.__dict__.items(): + new.__dict__[k] = v + return new + + def get(self, *args, **kwargs): + original = super(IntegrationQuerySet, self).get(*args, **kwargs) + return self._get_subclass_replacement(original) + + def subclass(self, instance): + return self._get_subclass_replacement(instance) + + def create(self, *args, **kwargs): # pylint: disable=unused-argument + """Override of create method to use subclass instance instead + + Instead of using the underlying Integration model to create this + instance, we get the correct subclass to use instead. This allows for + overrides to ``save`` and other model functions on object creation. + """ + model_cls = self._get_subclass(kwargs.get('integration_type')) + if model_cls is None: + model_cls = self.model + obj = model_cls(**kwargs) + self._for_write = True + obj.save(force_insert=True, using=self.db) + return obj + + +class Integration(models.Model): + + """Inbound webhook integration for projects""" + + GITHUB_WEBHOOK = 'github_webhook' + BITBUCKET_WEBHOOK = 'bitbucket_webhook' + GITLAB_WEBHOOK = 'gitlab_webhook' + API_WEBHOOK = 'api_webhook' + + WEBHOOK_INTEGRATIONS = ( + (GITHUB_WEBHOOK, _('GitHub incoming webhook')), + (BITBUCKET_WEBHOOK, _('Bitbucket incoming webhook')), + (GITLAB_WEBHOOK, _('GitLab incoming webhook')), + (API_WEBHOOK, _('Generic API incoming webhook')), + ) + + INTEGRATIONS = WEBHOOK_INTEGRATIONS + + project = models.ForeignKey(Project, related_name='integrations') + integration_type = models.CharField( + _('Integration type'), + max_length=32, + choices=INTEGRATIONS + ) + provider_data = JSONField(_('Provider data')) + exchanges = GenericRelation( + 'HttpExchange', + related_query_name='integrations' + ) + + objects = IntegrationQuerySet.as_manager() + + # Integration attributes + has_sync = False + + def __unicode__(self): + return (_('{0} for {1}') + .format(self.get_integration_type_display(), self.project.name)) + + +class GitHubWebhook(Integration): + + integration_type_id = Integration.GITHUB_WEBHOOK + has_sync = True + + class Meta: + proxy = True + + @property + def can_sync(self): + try: + return all((k in self.provider_data) for k in ['id', 'url']) + except (ValueError, TypeError): + return False + + +class BitbucketWebhook(Integration): + + integration_type_id = Integration.BITBUCKET_WEBHOOK + has_sync = True + + class Meta: + proxy = True + + @property + def can_sync(self): + try: + return all((k in self.provider_data) for k in ['uuid', 'url']) + except (ValueError, TypeError): + return False + + +class GenericAPIWebhook(Integration): + + integration_type_id = Integration.API_WEBHOOK + has_sync = False + + class Meta: + proxy = True + + def save(self, *args, **kwargs): + """Ensure model has token data before saving""" + try: + token = self.provider_data.get('token') + except (AttributeError, TypeError): + token = None + finally: + if token is None: + token = default_token() + self.provider_data = {'token': token} + super(GenericAPIWebhook, self).save(*args, **kwargs) + + @property + def token(self): + """Get or generate a secret token for authentication""" + return self.provider_data.get('token') diff --git a/readthedocs/oauth/services/base.py b/readthedocs/oauth/services/base.py index 019a51faf91..809adbafb2d 100644 --- a/readthedocs/oauth/services/base.py +++ b/readthedocs/oauth/services/base.py @@ -125,6 +125,9 @@ def create_organization(self, fields): def setup_webhook(self, project): raise NotImplementedError + def update_webhook(self, project, integration): + raise NotImplementedError + @classmethod def is_project_service(cls, project): """Determine if this is the service the project is using diff --git a/readthedocs/oauth/services/bitbucket.py b/readthedocs/oauth/services/bitbucket.py index 10a96fcc1c2..a4edae63802 100644 --- a/readthedocs/oauth/services/bitbucket.py +++ b/readthedocs/oauth/services/bitbucket.py @@ -11,6 +11,7 @@ BitbucketOAuth2Adapter) from readthedocs.builds import utils as build_utils +from readthedocs.integrations.models import Integration from ..models import RemoteOrganization, RemoteRepository from .base import Service @@ -174,6 +175,22 @@ def paginate(self, url): results.extend(self.paginate(next_url)) return results + def get_webhook_data(self, project, integration): + """Get webhook JSON data to post to the API""" + return json.dumps({ + 'description': 'Read the Docs ({domain})'.format(domain=settings.PRODUCTION_DOMAIN), + 'url': 'https://{domain}{path}'.format( + domain=settings.PRODUCTION_DOMAIN, + path=reverse( + 'api_webhook', + kwargs={'project_slug': project.slug, + 'integration_pk': integration.pk} + ) + ), + 'active': True, + 'events': ['repo:push'], + }) + def setup_webhook(self, project): """Set up Bitbucket project webhook for project @@ -184,18 +201,12 @@ def setup_webhook(self, project): """ session = self.get_session() owner, repo = build_utils.get_bitbucket_username_repo(url=project.repo) - data = json.dumps({ - 'description': 'Read the Docs ({domain})'.format(domain=settings.PRODUCTION_DOMAIN), - 'url': 'https://{domain}{path}'.format( - domain=settings.PRODUCTION_DOMAIN, - path=reverse( - 'api_webhook_bitbucket', - kwargs={'project_slug': project.slug} - ) - ), - 'active': True, - 'events': ['repo:push'], - }) + integration, _ = Integration.objects.get_or_create( + project=project, + integration_type=Integration.BITBUCKET_WEBHOOK, + ) + data = self.get_webhook_data(project, integration) + resp = None try: resp = session.post( ('https://api.bitbucket.org/2.0/repositories/{owner}/{repo}/hooks' @@ -204,15 +215,65 @@ def setup_webhook(self, project): headers={'content-type': 'application/json'} ) if resp.status_code == 201: + recv_data = resp.json() + integration.provider_data = recv_data + integration.save() log.info('Bitbucket webhook creation successful for project: %s', project) return (True, resp) - except RequestException: + # Catch exceptions with request or deserializing JSON + except (RequestException, ValueError): log.error('Bitbucket webhook creation failed for project: %s', project, exc_info=True) else: log.error('Bitbucket webhook creation failed for project: %s', project) - log.debug('Bitbucket webhook creation failure response: %s', - resp.content) + try: + log.debug('Bitbucket webhook creation failure response: %s', + resp.json()) + except ValueError: + pass + return (False, resp) + + def update_webhook(self, project, integration): + """Update webhook integration + + :param project: project to set up webhook for + :type project: Project + :param integration: Webhook integration to update + :type integration: Integration + :returns: boolean based on webhook set up success, and requests Response object + :rtype: (Bool, Response) + """ + session = self.get_session() + data = self.get_webhook_data(project, integration) + resp = None + try: + # Expect to throw KeyError here if provider_data is invalid + url = integration.provider_data['links']['self']['href'] + resp = session.put( + url, + data=data, + headers={'content-type': 'application/json'} + ) + if resp.status_code == 200: + recv_data = resp.json() + integration.provider_data = recv_data + integration.save() + log.info('Bitbucket webhook update successful for project: %s', + project) + return (True, resp) + # Catch exceptions with request or deserializing JSON + except (KeyError, RequestException, ValueError): + log.error('Bitbucket webhook update failed for project: %s', + project, exc_info=True) + else: + log.error('Bitbucket webhook update failed for project: %s', + project) + # Response data should always be JSON, still try to log if not though + try: + debug_data = resp.json() + except ValueError: + debug_data = resp.content + log.debug('Bitbucket webhook update failure response: %s', debug_data) return (False, resp) diff --git a/readthedocs/oauth/services/github.py b/readthedocs/oauth/services/github.py index da8171e1508..c7527999a21 100644 --- a/readthedocs/oauth/services/github.py +++ b/readthedocs/oauth/services/github.py @@ -11,6 +11,7 @@ from allauth.socialaccount.providers.github.views import GitHubOAuth2Adapter from readthedocs.builds import utils as build_utils +from readthedocs.integrations.models import Integration from readthedocs.restapi.client import api from ..models import RemoteOrganization, RemoteRepository @@ -159,31 +160,40 @@ def paginate(self, url): result.extend(self.paginate(next_url)) return result - def setup_webhook(self, project): - """Set up GitHub project webhook for project - - :param project: project to set up webhook for - :type project: Project - :returns: boolean based on webhook set up success, and requests Response object - :rtype: (Bool, Response) - """ - session = self.get_session() - owner, repo = build_utils.get_github_username_repo(url=project.repo) - data = json.dumps({ + def get_webhook_data(self, project, integration): + """Get webhook JSON data to post to the API""" + return json.dumps({ 'name': 'web', 'active': True, 'config': { 'url': 'https://{domain}{path}'.format( domain=settings.PRODUCTION_DOMAIN, path=reverse( - 'api_webhook_github', - kwargs={'project_slug': project.slug} + 'api_webhook', + kwargs={'project_slug': project.slug, + 'integration_pk': integration.pk} ) ), 'content_type': 'json', }, 'events': ['push', 'pull_request'], }) + + def setup_webhook(self, project): + """Set up GitHub project webhook for project + + :param project: project to set up webhook for + :type project: Project + :returns: boolean based on webhook set up success, and requests Response object + :rtype: (Bool, Response) + """ + session = self.get_session() + owner, repo = build_utils.get_github_username_repo(url=project.repo) + integration, _ = Integration.objects.get_or_create( + project=project, + integration_type=Integration.GITHUB_WEBHOOK, + ) + data = self.get_webhook_data(project, integration) resp = None try: resp = session.post( @@ -194,18 +204,70 @@ def setup_webhook(self, project): ) # GitHub will return 200 if already synced if resp.status_code in [200, 201]: + recv_data = resp.json() + integration.provider_data = recv_data + integration.save() log.info('GitHub webhook creation successful for project: %s', project) return (True, resp) - except RequestException: + # Catch exceptions with request or deserializing JSON + except (RequestException, ValueError): log.error('GitHub webhook creation failed for project: %s', project, exc_info=True) pass else: log.error('GitHub webhook creation failed for project: %s', project) + # Response data should always be JSON, still try to log if not though + try: + debug_data = resp.json() + except ValueError: + debug_data = resp.content log.debug('GitHub webhook creation failure response: %s', - resp.content) + resp.json()) + return (False, resp) + + def update_webhook(self, project, integration): + """Update webhook integration + + :param project: project to set up webhook for + :type project: Project + :param integration: Webhook integration to update + :type integration: Integration + :returns: boolean based on webhook update success, and requests Response object + :rtype: (Bool, Response) + """ + session = self.get_session() + data = self.get_webhook_data(project, integration) + url = integration.provider_data.get('url') + resp = None + try: + resp = session.patch( + url, + data=data, + headers={'content-type': 'application/json'} + ) + # GitHub will return 200 if already synced + if resp.status_code in [200, 201]: + recv_data = resp.json() + integration.provider_data = recv_data + integration.save() + log.info('GitHub webhook creation successful for project: %s', + project) + return (True, resp) + # Catch exceptions with request or deserializing JSON + except (RequestException, ValueError): + log.error('GitHub webhook update failed for project: %s', + project, exc_info=True) + pass + else: + log.error('GitHub webhook update failed for project: %s', + project) + try: + log.debug('GitHub webhook creation failure response: %s', + resp.json()) + except ValueError: + pass return (False, resp) @classmethod diff --git a/readthedocs/oauth/utils.py b/readthedocs/oauth/utils.py index 9fbca0952b8..3418bd9062e 100644 --- a/readthedocs/oauth/utils.py +++ b/readthedocs/oauth/utils.py @@ -3,14 +3,25 @@ from django.contrib import messages from django.utils.translation import ugettext_lazy as _ -from readthedocs.oauth.services import registry +from readthedocs.integrations.models import Integration +from readthedocs.oauth.services import registry, GitHubService, BitbucketService log = logging.getLogger(__name__) +SERVICE_MAP = { + Integration.GITHUB_WEBHOOK: GitHubService, + Integration.BITBUCKET_WEBHOOK: BitbucketService, +} + def attach_webhook(project, request=None): - """Add post-commit hook on project import""" + """Add post-commit hook on project import + This is a brute force approach to adding a webhook to a repository. We try + all accounts until we set up a webhook. This should remain around for legacy + connections -- that is, projects that do not have a remote repository them + and were not set up with a VCS provider. + """ for service_cls in registry: if service_cls.is_project_service(project): service = service_cls @@ -46,3 +57,27 @@ def attach_webhook(project, request=None): )) ) return False + + +def update_webhook(project, integration, request=None): + """Update a specific project integration instead of brute forcing""" + service_cls = SERVICE_MAP.get(integration.integration_type) + if service_cls is None: + return None + account = project.remote_repository.account + service = service_cls(request.user, account) + updated, __ = service.update_webhook(project, integration) + if updated: + messages.success(request, _('Webhook activated')) + project.has_valid_webhook = True + project.save() + return True + else: + messages.error( + request, + _('Webhook activation failed. ' + 'Make sure you have the necessary permissions.') + ) + project.has_valid_webhook = False + project.save() + return False diff --git a/readthedocs/projects/forms.py b/readthedocs/projects/forms.py index 68752e21819..bb373f40a82 100644 --- a/readthedocs/projects/forms.py +++ b/readthedocs/projects/forms.py @@ -15,12 +15,13 @@ from readthedocs.builds.constants import TAG from readthedocs.core.utils import trigger_build, slugify -from readthedocs.redirects.models import Redirect +from readthedocs.integrations.models import Integration from readthedocs.oauth.models import RemoteRepository from readthedocs.projects import constants from readthedocs.projects.exceptions import ProjectSpamError from readthedocs.projects.models import Project, EmailHook, WebHook, Domain from readthedocs.privacy.loader import AdminPermission +from readthedocs.redirects.models import Redirect class ProjectForm(forms.ModelForm): @@ -544,6 +545,33 @@ def clean_canonical(self): return canonical +class IntegrationForm(forms.ModelForm): + + """Form to add an integration + + This limits the choices of the integration type to webhook integration types + """ + + project = forms.CharField(widget=forms.HiddenInput(), required=False) + + class Meta: + model = Integration + exclude = ['provider_data', 'exchanges'] + + def __init__(self, *args, **kwargs): + self.project = kwargs.pop('project', None) + super(IntegrationForm, self).__init__(*args, **kwargs) + # Alter the integration type choices to only provider webhooks + self.fields['integration_type'].choices = Integration.WEBHOOK_INTEGRATIONS + + def clean_project(self): + return self.project + + def save(self, commit=True): + self.instance = Integration.objects.subclass(self.instance) + return super(IntegrationForm, self).save(commit) + + class ProjectAdvertisingForm(forms.ModelForm): """Project promotion opt-out form""" diff --git a/readthedocs/projects/urls/private.py b/readthedocs/projects/urls/private.py index f3ebedf3bcb..33a11079a28 100644 --- a/readthedocs/projects/urls/private.py +++ b/readthedocs/projects/urls/private.py @@ -2,13 +2,14 @@ from django.conf.urls import url +from readthedocs.constants import pattern_opts from readthedocs.projects.views import private from readthedocs.projects.views.private import ( ProjectDashboard, ImportView, ProjectUpdate, ProjectAdvancedUpdate, DomainList, DomainCreate, DomainDelete, DomainUpdate, - IntegrationList, IntegrationExchangeDetail, IntegrationWebhookSync, - ProjectAdvertisingUpdate) + IntegrationList, IntegrationCreate, IntegrationDetail, IntegrationDelete, + IntegrationExchangeDetail, IntegrationWebhookSync, ProjectAdvertisingUpdate) from readthedocs.projects.backends.views import ImportWizardView, ImportDemoView @@ -129,15 +130,37 @@ urlpatterns += domain_urls integration_urls = [ - url(r'^(?P[-\w]+)/integrations/$', + url(r'^(?P{project_slug})/integrations/$'.format(**pattern_opts), IntegrationList.as_view(), name='projects_integrations'), - url(r'^(?P[-\w]+)/integrations/exchange/(?P[-\w]+)/$', + url(r'^(?P{project_slug})/integrations/sync/$'.format(**pattern_opts), + IntegrationWebhookSync.as_view(), + name='projects_integrations_webhooks_sync'), + url((r'^(?P{project_slug})/integrations/create/$' + .format(**pattern_opts)), + IntegrationCreate.as_view(), + name='projects_integrations_create'), + url((r'^(?P{project_slug})/' + r'integrations/(?P{integer_pk})/$' + .format(**pattern_opts)), + IntegrationDetail.as_view(), + name='projects_integrations_detail'), + url((r'^(?P{project_slug})/' + r'integrations/(?P{integer_pk})/' + r'exchange/(?P[-\w]+)/$' + .format(**pattern_opts)), IntegrationExchangeDetail.as_view(), - name='projects_integrations_exchange_detail'), - url(r'^(?P[-\w]+)/integrations/sync/$', + name='projects_integrations_exchanges_detail'), + url((r'^(?P{project_slug})/' + r'integrations/(?P{integer_pk})/sync/$' + .format(**pattern_opts)), IntegrationWebhookSync.as_view(), - name='projects_integrations_sync'), + name='projects_integrations_webhooks_sync'), + url((r'^(?P{project_slug})/' + r'integrations/(?P{integer_pk})/delete/$' + .format(**pattern_opts)), + IntegrationDelete.as_view(), + name='projects_integrations_delete'), ] urlpatterns += integration_urls diff --git a/readthedocs/projects/views/private.py b/readthedocs/projects/views/private.py index 9dbb346c9c6..888c1ce944b 100644 --- a/readthedocs/projects/views/private.py +++ b/readthedocs/projects/views/private.py @@ -27,17 +27,18 @@ from readthedocs.builds.models import VersionAlias from readthedocs.core.utils import trigger_build, broadcast from readthedocs.core.mixins import ListViewWithForm -from readthedocs.integrations.models import HttpExchange +from readthedocs.integrations.models import HttpExchange, Integration from readthedocs.projects.forms import ( ProjectBasicsForm, ProjectExtraForm, ProjectAdvancedForm, UpdateProjectForm, SubprojectForm, build_versions_form, UserForm, EmailHookForm, TranslationForm, - RedirectForm, WebHookForm, DomainForm, ProjectAdvertisingForm) + RedirectForm, WebHookForm, DomainForm, IntegrationForm, + ProjectAdvertisingForm) from readthedocs.projects.models import Project, EmailHook, WebHook, Domain from readthedocs.projects.views.base import ProjectAdminMixin, ProjectSpamMixin from readthedocs.projects import constants, tasks from readthedocs.oauth.services import registry -from readthedocs.oauth.utils import attach_webhook +from readthedocs.oauth.utils import attach_webhook, update_webhook from readthedocs.core.mixins import LoginRequiredMixin from readthedocs.projects.signals import project_import @@ -665,13 +666,33 @@ class DomainDelete(DomainMixin, DeleteView): pass -class IntegrationMixin(object): +class IntegrationMixin(ProjectAdminMixin, PrivateViewMixin): - """Project external service mixin for listing webhook objects + """Project external service mixin for listing webhook objects""" - This mixin will be used more once we have modeling around webhooks and - external integrations. - """ + model = Integration + integration_url_field = 'integration_pk' + form_class = IntegrationForm + + def get_queryset(self): + return self.get_integration_queryset() + + def get_object(self): + return self.get_integration() + + def get_integration_queryset(self): + self.project = self.get_project() + return self.model.objects.filter(project=self.project) + + def get_integration(self): + """Return project integration determined by url kwarg""" + if self.integration_url_field not in self.kwargs: + return None + return get_object_or_404( + Integration, + pk=self.kwargs[self.integration_url_field], + project=self.get_project(), + ) def get_success_url(self): return reverse('projects_integrations', args=[self.get_project().slug]) @@ -682,33 +703,63 @@ def get_template_names(self): return 'projects/integration{0}.html'.format(self.template_name_suffix) -class IntegrationExchangeMixin(ProjectAdminMixin, PrivateViewMixin): +class IntegrationList(IntegrationMixin, ListView): + pass - """Project webhook exchange mixin for listing exchange objects""" - model = HttpExchange - lookup_url_kwarg = 'exchange_pk' +class IntegrationCreate(IntegrationMixin, CreateView): - def get_queryset(self): - self.project = self.get_project() - return self.model.objects.filter( - content_type=ContentType.objects.filter( - app_label='projects', - model='project' - ), - object_id=self.project.pk + def get_success_url(self): + return reverse( + 'projects_integrations_detail', + kwargs={ + 'project_slug': self.get_project().slug, + 'integration_pk': self.object.id, + } ) -class IntegrationList(IntegrationMixin, IntegrationExchangeMixin, ListView): - pass +class IntegrationDetail(IntegrationMixin, DetailView): + + # Some of the templates can be combined, we'll avoid duplicating templates + SUFFIX_MAP = { + Integration.GITHUB_WEBHOOK: 'webhook', + Integration.GITLAB_WEBHOOK: 'webhook', + Integration.BITBUCKET_WEBHOOK: 'webhook', + Integration.API_WEBHOOK: 'generic_webhook', + } + def get_template_names(self): + if self.template_name: + return self.template_name + integration_type = self.get_integration().integration_type + suffix = self.SUFFIX_MAP.get(integration_type, integration_type) + return ('projects/integration_{0}{1}.html' + .format(suffix, self.template_name_suffix)) + + +class IntegrationDelete(IntegrationMixin, DeleteView): + + def get(self, request, *args, **kwargs): + return self.http_method_not_allowed(request, *args, **kwargs) + + +class IntegrationExchangeDetail(IntegrationMixin, DetailView): -class IntegrationExchangeDetail(IntegrationMixin, IntegrationExchangeMixin, DetailView): + model = HttpExchange + lookup_url_kwarg = 'exchange_pk' template_name = 'projects/integration_exchange_detail.html' + def get_queryset(self): + return self.model.objects.filter( + integrations=self.get_integration() + ) + + def get_object(self): + return DetailView.get_object(self) -class IntegrationWebhookSync(PrivateViewMixin, ProjectAdminMixin, GenericView): + +class IntegrationWebhookSync(IntegrationMixin, GenericView): """Resync a project webhook @@ -717,7 +768,14 @@ class IntegrationWebhookSync(PrivateViewMixin, ProjectAdminMixin, GenericView): def post(self, request, *args, **kwargs): # pylint: disable=unused-argument - attach_webhook(project=self.get_project(), request=request) + if 'integration_pk' in kwargs: + integration = self.get_integration() + update_webhook(self.get_project(), integration, request=request) + else: + # This is a brute force form of the webhook sync, if a project has a + # webhook or a remote repository object, the user should be using + # the per-integration sync instead. + attach_webhook(project=self.get_project(), request=request) return HttpResponseRedirect(self.get_success_url()) def get_success_url(self): diff --git a/readthedocs/restapi/urls.py b/readthedocs/restapi/urls.py index 5b3152d93c0..de85d60894d 100644 --- a/readthedocs/restapi/urls.py +++ b/readthedocs/restapi/urls.py @@ -62,18 +62,22 @@ ] integration_urls = [ - url(r'webhook/github/(?P{project_slug})/'.format(**pattern_opts), + url(r'webhook/github/(?P{project_slug})/$'.format(**pattern_opts), integrations.GitHubWebhookView.as_view(), name='api_webhook_github'), - url(r'webhook/gitlab/(?P{project_slug})/'.format(**pattern_opts), + url(r'webhook/gitlab/(?P{project_slug})/$'.format(**pattern_opts), integrations.GitLabWebhookView.as_view(), name='api_webhook_gitlab'), - url(r'webhook/bitbucket/(?P{project_slug})/'.format(**pattern_opts), + url(r'webhook/bitbucket/(?P{project_slug})/$'.format(**pattern_opts), integrations.BitbucketWebhookView.as_view(), name='api_webhook_bitbucket'), - url(r'webhook/generic/(?P{project_slug})/'.format(**pattern_opts), - integrations.GenericWebhookView.as_view(), + url(r'webhook/generic/(?P{project_slug})/$'.format(**pattern_opts), + integrations.APIWebhookView.as_view(), name='api_webhook_generic'), + url((r'webhook/(?P{project_slug})/' + r'(?P{integer_pk})/$'.format(**pattern_opts)), + integrations.WebhookView.as_view(), + name='api_webhook'), ] urlpatterns += function_urls diff --git a/readthedocs/restapi/views/integrations.py b/readthedocs/restapi/views/integrations.py index 73a9d589d42..97bf5b06a81 100644 --- a/readthedocs/restapi/views/integrations.py +++ b/readthedocs/restapi/views/integrations.py @@ -13,7 +13,7 @@ from readthedocs.core.views.hooks import build_branches from readthedocs.core.signals import (webhook_github, webhook_bitbucket, webhook_gitlab) -from readthedocs.integrations.models import HttpExchange +from readthedocs.integrations.models import HttpExchange, Integration from readthedocs.integrations.utils import normalize_request_payload from readthedocs.projects.models import Project @@ -29,21 +29,26 @@ class WebhookMixin(object): permission_classes = (permissions.AllowAny,) renderer_classes = (JSONRenderer,) + integration = None + integration_type = None def post(self, request, project_slug, format=None): """Set up webhook post view with request and project objects""" self.request = request self.project = None try: - self.project = Project.objects.get(slug=project_slug) - resp = self.handle_webhook() - if resp is None: - log.info('Unhandled webhook event') - resp = {'detail': 'Unhandled webhook event'} - resp = Response(resp) + self.project = self.get_project(slug=project_slug) except Project.DoesNotExist: raise NotFound('Project not found') - return resp + self.data = self.get_data() + resp = self.handle_webhook() + if resp is None: + log.info('Unhandled webhook event') + resp = {'detail': 'Unhandled webhook event'} + return Response(resp) + + def get_project(self, **kwargs): + return Project.objects.get(**kwargs) def finalize_response(self, req, *args, **kwargs): """If the project was set on POST, store an HTTP exchange""" @@ -52,22 +57,38 @@ def finalize_response(self, req, *args, **kwargs): HttpExchange.objects.from_exchange( req, resp, - related_object=self.project, - payload=self.get_payload(), + related_object=self.get_integration(), + payload=self.data, ) return resp + def get_data(self): + """Normalize posted data""" + return normalize_request_payload(self.request) + def handle_webhook(self): """Handle webhook payload""" raise NotImplementedError - def get_payload(self): - """Don't specify any special handling of the payload data + def get_integration(self): + """Get or create an inbound webhook to track webhook requests - The exchange will record ``request.data`` instead of assume any - special handling of the payload data + We shouldn't need this, but to support legacy webhooks, we can't assume + that a webhook has ever been created on our side. Most providers don't + pass the webhook ID in either, so we default to just finding *any* + integration from the provider. This is not ideal, but the + :py:class:`WebhookView` view solves this by performing a lookup on the + integration instead of guessing. """ - return None + # `integration` can be passed in as an argument to `as_view`, as it is + # in `WebhookView` + if self.integration is not None: + return self.integration + integration, _ = Integration.objects.get_or_create( + project=self.project, + integration_type=self.integration_type, + ) + return integration def get_response_push(self, project, branches): """Build branches on push events and return API response @@ -111,24 +132,25 @@ class GitHubWebhookView(WebhookMixin, APIView): } """ - def get_payload(self): + integration_type = Integration.GITHUB_WEBHOOK + + def get_data(self): if self.request.content_type == 'application/x-www-form-urlencoded': try: return json.loads(self.request.data['payload']) - except ValueError: + except (ValueError, KeyError): pass - return normalize_request_payload(self.request) + return super(GitHubWebhookView, self).get_data() def handle_webhook(self): - data = self.get_payload() # Get event and trigger other webhook events event = self.request.META.get('HTTP_X_GITHUB_EVENT', 'push') webhook_github.send(Project, project=self.project, - data=data, event=event) + data=self.data, event=event) # Handle push events and trigger builds if event == GITHUB_PUSH: try: - branches = [data['ref'].replace('refs/heads/', '')] + branches = [self.data['ref'].replace('refs/heads/', '')] return self.get_response_push(self.project, branches) except KeyError: raise ParseError('Parameter "ref" is required') @@ -149,6 +171,8 @@ class GitLabWebhookView(WebhookMixin, APIView): } """ + integration_type = Integration.GITLAB_WEBHOOK + def handle_webhook(self): # Get event and trigger other webhook events event = self.request.data.get('object_kind', GITLAB_PUSH) @@ -186,6 +210,8 @@ class BitbucketWebhookView(WebhookMixin, APIView): } """ + integration_type = Integration.BITBUCKET_WEBHOOK + def handle_webhook(self): # Get event and trigger other webhook events event = self.request.META.get('HTTP_X_EVENT_KEY', BITBUCKET_PUSH) @@ -202,9 +228,23 @@ def handle_webhook(self): raise ParseError('Invalid request') -class GenericWebhookView(WebhookMixin, APIView): +class IsAuthenticatedOrHasToken(permissions.IsAuthenticated): - """Generic webhook consumer + """Allow authenticated users and requests with token auth through + + This does not check for instance-level permissions, as the check uses + methods from the view to determine if the token matches. + """ + + def has_permission(self, request, view): + has_perm = (super(IsAuthenticatedOrHasToken, self) + .has_permission(request, view)) + return has_perm or 'token' in request.data + + +class APIWebhookView(WebhookMixin, APIView): + + """API webhook consumer Expects the following JSON:: @@ -213,12 +253,73 @@ class GenericWebhookView(WebhookMixin, APIView): } """ + integration_type = Integration.API_WEBHOOK + permission_classes = [IsAuthenticatedOrHasToken] + + def get_project(self, **kwargs): + """Get authenticated user projects, or token authed projects + + Allow for a user to either be authed to receive a project, or require + the integration token to be specified as a POST argument. + """ + # If the user is not an admin of the project, fall back to token auth + if self.request.user.is_authenticated(): + try: + return (Project.objects + .for_admin_user(self.request.user) + .get(**kwargs)) + except Project.DoesNotExist: + pass + # Recheck project and integration relationship during token auth check + token = self.request.data.get('token') + if token: + integration = self.get_integration() + obj = Project.objects.get(**kwargs) + is_valid = ( + integration.project == obj and + token == getattr(integration, 'token', None) + ) + if is_valid: + return obj + raise Project.DoesNotExist() + def handle_webhook(self): try: - branches = list(self.request.data.get( + branches = self.request.data.get( 'branches', [self.project.get_default_branch()] - )) + ) + if isinstance(branches, basestring): + branches = [branches] return self.get_response_push(self.project, branches) except TypeError: raise ParseError('Invalid request') + + +class WebhookView(APIView): + + """This is the main webhook view for webhooks with an ID + + The handling of each view is handed off to another view. This should only + ever get webhook requests for established webhooks on our side. The other + views can receive webhooks for unknown webhooks, as all legacy webhooks will + be. + """ + + VIEW_MAP = { + Integration.GITHUB_WEBHOOK: GitHubWebhookView, + Integration.GITLAB_WEBHOOK: GitLabWebhookView, + Integration.BITBUCKET_WEBHOOK: BitbucketWebhookView, + Integration.API_WEBHOOK: APIWebhookView, + } + + def post(self, request, project_slug, integration_pk): + """Set up webhook post view with request and project objects""" + integration = get_object_or_404( + Integration, + project__slug=project_slug, + pk=integration_pk, + ) + view_cls = self.VIEW_MAP[integration.integration_type] + view = view_cls.as_view(integration=integration) + return view(request, project_slug) diff --git a/readthedocs/rtd_tests/tests/test_api.py b/readthedocs/rtd_tests/tests/test_api.py index 9092ccba34b..332268ec158 100644 --- a/readthedocs/rtd_tests/tests/test_api.py +++ b/readthedocs/rtd_tests/tests/test_api.py @@ -11,6 +11,7 @@ from allauth.socialaccount.models import SocialAccount from readthedocs.builds.models import Build, Version +from readthedocs.integrations.models import Integration from readthedocs.projects.models import Project from readthedocs.oauth.models import RemoteRepository, RemoteOrganization @@ -414,3 +415,69 @@ def test_bitbucket_invalid_webhook(self, trigger_build): ) self.assertEqual(resp.status_code, 200) self.assertEqual(resp.data['detail'], 'Unhandled webhook event') + + def test_generic_api_fails_without_auth(self, trigger_build): + client = APIClient() + resp = client.post( + '/api/v2/webhook/generic/{0}/'.format(self.project.slug), + {}, + format='json', + ) + self.assertEqual(resp.status_code, 403) + self.assertEqual( + resp.data['detail'], + 'Authentication credentials were not provided.' + ) + + def test_generic_api_respects_token_auth(self, trigger_build): + client = APIClient() + integration = Integration.objects.create( + project=self.project, + integration_type=Integration.API_WEBHOOK + ) + self.assertIsNotNone(integration.token) + resp = client.post( + '/api/v2/webhook/{0}/{1}/'.format(self.project.slug, integration.pk), + {'token': integration.token}, + format='json', + ) + self.assertEqual(resp.status_code, 200) + self.assertTrue(resp.data['build_triggered']) + # Test nonexistent branch + resp = client.post( + '/api/v2/webhook/{0}/{1}/'.format(self.project.slug, integration.pk), + {'token': integration.token, 'branches': 'nonexistent'}, + format='json', + ) + self.assertEqual(resp.status_code, 200) + self.assertFalse(resp.data['build_triggered']) + + def test_generic_api_respects_basic_auth(self, trigger_build): + client = APIClient() + user = get(User) + self.project.users.add(user) + client.force_authenticate(user=user) + resp = client.post( + '/api/v2/webhook/generic/{0}/'.format(self.project.slug), + {}, + format='json', + ) + self.assertEqual(resp.status_code, 200) + self.assertTrue(resp.data['build_triggered']) + + def test_generic_api_falls_back_to_token_auth(self, trigger_build): + client = APIClient() + user = get(User) + client.force_authenticate(user=user) + integration = Integration.objects.create( + project=self.project, + integration_type=Integration.API_WEBHOOK + ) + self.assertIsNotNone(integration.token) + resp = client.post( + '/api/v2/webhook/{0}/{1}/'.format(self.project.slug, integration.pk), + {'token': integration.token}, + format='json', + ) + self.assertEqual(resp.status_code, 200) + self.assertTrue(resp.data['build_triggered']) diff --git a/readthedocs/rtd_tests/tests/test_integrations.py b/readthedocs/rtd_tests/tests/test_integrations.py index ad5f1fe48cb..db6c75a976f 100644 --- a/readthedocs/rtd_tests/tests/test_integrations.py +++ b/readthedocs/rtd_tests/tests/test_integrations.py @@ -5,7 +5,9 @@ from rest_framework.test import APIRequestFactory from rest_framework.response import Response -from readthedocs.integrations.models import HttpExchange +from readthedocs.integrations.models import ( + HttpExchange, Integration, GitHubWebhook +) from readthedocs.projects.models import Project @@ -21,18 +23,15 @@ def test_exchange_json_request_body(self): client = APIClient() client.login(username='super', password='test') project = fixture.get(Project, main_language_project=None) + integration = fixture.get(Integration, project=project, + integration_type=Integration.GITHUB_WEBHOOK, + provider_data='') resp = client.post( '/api/v2/webhook/github/{0}/'.format(project.slug), {'ref': 'exchange_json'}, format='json' ) - exchange = HttpExchange.objects.get( - content_type=ContentType.objects.filter( - app_label='projects', - model='project' - ), - object_id=project.pk - ) + exchange = HttpExchange.objects.get(integrations=integration) self.assertEqual( exchange.request_body, '{"ref": "exchange_json"}' @@ -57,18 +56,15 @@ def test_exchange_form_request_body(self): client = APIClient() client.login(username='super', password='test') project = fixture.get(Project, main_language_project=None) + integration = fixture.get(Integration, project=project, + integration_type=Integration.GITHUB_WEBHOOK, + provider_data='') resp = client.post( '/api/v2/webhook/github/{0}/'.format(project.slug), 'payload=%7B%22ref%22%3A+%22exchange_form%22%7D', content_type='application/x-www-form-urlencoded', ) - exchange = HttpExchange.objects.get( - content_type=ContentType.objects.filter( - app_label='projects', - model='project' - ), - object_id=project.pk - ) + exchange = HttpExchange.objects.get(integrations=integration) self.assertEqual( exchange.request_body, '{"ref": "exchange_form"}' @@ -93,15 +89,12 @@ def test_extraneous_exchanges_deleted_in_correct_order(self): client = APIClient() client.login(username='super', password='test') project = fixture.get(Project, main_language_project=None) + integration = fixture.get(Integration, project=project, + integration_type=Integration.GITHUB_WEBHOOK, + provider_data='') self.assertEqual( - HttpExchange.objects.filter( - content_type=ContentType.objects.get( - app_label='projects', - model='project', - ), - object_id=project.pk - ).count(), + HttpExchange.objects.filter(integrations=integration).count(), 0 ) @@ -119,23 +112,73 @@ def test_extraneous_exchanges_deleted_in_correct_order(self): ) self.assertEqual( - HttpExchange.objects.filter( - content_type=ContentType.objects.get( - app_label='projects', - model='project', - ), - object_id=project.pk - ).count(), + HttpExchange.objects.filter(integrations=integration).count(), 10 ) self.assertEqual( HttpExchange.objects.filter( - content_type=ContentType.objects.get( - app_label='projects', - model='project', - ), - object_id=project.pk, + integrations=integration, request_body='{"ref": "preserved"}', ).count(), 10 ) + + def test_request_headers_are_removed(self): + client = APIClient() + client.login(username='super', password='test') + project = fixture.get(Project, main_language_project=None) + integration = fixture.get(Integration, project=project, + integration_type=Integration.GITHUB_WEBHOOK, + provider_data='') + resp = client.post( + '/api/v2/webhook/github/{0}/'.format(project.slug), + {'ref': 'exchange_json'}, + format='json', + HTTP_X_FORWARDED_FOR='1.2.3.4', + HTTP_X_REAL_IP='5.6.7.8', + HTTP_X_FOO='bar', + ) + exchange = HttpExchange.objects.get(integrations=integration) + self.assertEqual( + exchange.request_headers, + {u'Content-Type': u'application/json; charset=None', + u'Cookie': u'', + u'X-Foo': u'bar'} + ) + + +class IntegrationModelTests(TestCase): + + def test_subclass_is_replaced_on_get(self): + project = fixture.get(Project, main_language_project=None) + integration = Integration.objects.create( + project=project, + integration_type=Integration.GITHUB_WEBHOOK + ) + integration = Integration.objects.get(pk=integration.pk) + self.assertIsInstance(integration, GitHubWebhook) + + def test_subclass_is_replaced_on_subclass(self): + project = fixture.get(Project, main_language_project=None) + integration = Integration.objects.create( + project=project, + integration_type=Integration.GITHUB_WEBHOOK + ) + integration = Integration.objects.subclass(integration) + self.assertIsInstance(integration, GitHubWebhook) + + def test_subclass_is_replaced_on_create(self): + project = fixture.get(Project, main_language_project=None) + integration = Integration.objects.create( + integration_type=Integration.GITHUB_WEBHOOK, + project=project, + ) + self.assertIsInstance(integration, GitHubWebhook) + + def test_generic_token(self): + project = fixture.get(Project, main_language_project=None) + integration = Integration.objects.create( + integration_type=Integration.API_WEBHOOK, + project=project, + ) + self.assertIsNotNone(integration.token) diff --git a/readthedocs/rtd_tests/tests/test_privacy_urls.py b/readthedocs/rtd_tests/tests/test_privacy_urls.py index 57dcae90127..2cfca20202a 100644 --- a/readthedocs/rtd_tests/tests/test_privacy_urls.py +++ b/readthedocs/rtd_tests/tests/test_privacy_urls.py @@ -6,7 +6,7 @@ from readthedocs.builds.models import Build, VersionAlias, BuildCommandResult from readthedocs.comments.models import DocumentComment, NodeSnapshot -from readthedocs.integrations.models import HttpExchange +from readthedocs.integrations.models import HttpExchange, Integration from readthedocs.projects.models import Project, Domain from readthedocs.oauth.models import RemoteRepository, RemoteOrganization from readthedocs.rtd_tests.utils import create_user @@ -128,9 +128,10 @@ def setUp(self): users=[self.owner], main_language_project=None) self.pip.add_subproject(self.subproject) self.pip.translations.add(self.subproject) + self.integration = get(Integration, project=self.pip, provider_data='') # For whatever reason, fixtures hates JSONField self.webhook_exchange = HttpExchange.objects.create( - related_object=self.pip, + related_object=self.integration, request_headers='{"foo": "bar"}', response_headers='{"foo": "bar"}', status_code=200, @@ -146,6 +147,7 @@ def setUp(self): 'child_slug': self.subproject.slug, 'build_pk': self.build.pk, 'domain_pk': self.domain.pk, + 'integration_pk': self.integration.pk, 'exchange_pk': self.webhook_exchange.pk, } @@ -223,6 +225,8 @@ class PrivateProjectAdminAccessTest(PrivateProjectMixin, TestCase): '/dashboard/pip/notifications/delete/': {'status_code': 405}, '/dashboard/pip/redirects/delete/': {'status_code': 405}, '/dashboard/pip/integrations/sync/': {'status_code': 405}, + '/dashboard/pip/integrations/1/sync/': {'status_code': 405}, + '/dashboard/pip/integrations/1/delete/': {'status_code': 405}, } def login(self): @@ -249,6 +253,8 @@ class PrivateProjectUserAccessTest(PrivateProjectMixin, TestCase): '/dashboard/pip/notifications/delete/': {'status_code': 405}, '/dashboard/pip/redirects/delete/': {'status_code': 405}, '/dashboard/pip/integrations/sync/': {'status_code': 405}, + '/dashboard/pip/integrations/1/sync/': {'status_code': 405}, + '/dashboard/pip/integrations/1/delete/': {'status_code': 405}, } # Filtered out by queryset on projects that we don't own. @@ -284,6 +290,7 @@ def setUp(self): self.snapshot = get(NodeSnapshot, node=self.comment.node) self.remote_org = get(RemoteOrganization) self.remote_repo = get(RemoteRepository, organization=self.remote_org) + self.integration = get(Integration, project=self.pip, provider_data='') self.default_kwargs = { 'project_slug': self.pip.slug, 'version_slug': self.pip.versions.all()[0].slug, @@ -300,6 +307,7 @@ def setUp(self): 'footer_html': {'data': {'project': 'pip', 'version': 'latest', 'page': 'index'}}, 'remoteorganization-detail': {'pk': self.remote_org.pk}, 'remoterepository-detail': {'pk': self.remote_repo.pk}, + 'api_webhook': {'integration_pk': self.integration.pk}, } self.response_data = { 'project-sync-versions': {'status_code': 403}, @@ -315,10 +323,11 @@ def setUp(self): 'api_project_search': {'status_code': 400}, 'api_section_search': {'status_code': 400}, 'api_sync_remote_repositories': {'status_code': 403}, + 'api_webhook': {'status_code': 405}, 'api_webhook_github': {'status_code': 405}, 'api_webhook_gitlab': {'status_code': 405}, 'api_webhook_bitbucket': {'status_code': 405}, - 'api_webhook_generic': {'status_code': 405}, + 'api_webhook_generic': {'status_code': 403}, 'remoteorganization-detail': {'status_code': 404}, 'remoterepository-detail': {'status_code': 404}, } diff --git a/readthedocs/templates/core/project_bar_base.html b/readthedocs/templates/core/project_bar_base.html index 14bc4e454fb..99d5b661850 100644 --- a/readthedocs/templates/core/project_bar_base.html +++ b/readthedocs/templates/core/project_bar_base.html @@ -28,12 +28,12 @@

{% if not project.has_valid_webhook and request.user|is_admin:project %}

- {% url "projects_integrations_sync" project.slug as sync_url %} + {% url "projects_integrations" project.slug as integrations_url %} {% blocktrans %} - This repository doesn't have a valid webhook set up, - commits won't trigger new builds for this project. -
- You can sync your webhook to fix this. + This repository doesn't have a valid webhook set up, + commits won't trigger new builds for this project. +
+ See your project integrations for more information. {% endblocktrans %}

{% endif %} diff --git a/readthedocs/templates/projects/integration_form.html b/readthedocs/templates/projects/integration_form.html new file mode 100644 index 00000000000..cc46a9b0355 --- /dev/null +++ b/readthedocs/templates/projects/integration_form.html @@ -0,0 +1,23 @@ +{% extends "projects/project_edit_base.html" %} + +{% load i18n %} + +{% block title %}{% trans "Integrations" %}{% endblock %} + +{% block nav-dashboard %} class="active"{% endblock %} + +{% block editing-option-edit-integrations %}class="active"{% endblock %} + +{% block project-integrations-active %}active{% endblock %} +{% block project_edit_content_header %}{% trans "Integrations" %}{% endblock %} + +{% block project_edit_content %} +
+ {% csrf_token %} + {{ form.as_p }} + +
+{% endblock %} diff --git a/readthedocs/templates/projects/integration_generic_webhook_detail.html b/readthedocs/templates/projects/integration_generic_webhook_detail.html new file mode 100644 index 00000000000..59ff097383c --- /dev/null +++ b/readthedocs/templates/projects/integration_generic_webhook_detail.html @@ -0,0 +1,18 @@ +{% extends "projects/integration_webhook_detail.html" %} + +{% load i18n %} + +{% block integration_details %} +

+ {% blocktrans %} + The following parameters are configured for this integration: + {% endblocktrans %} +

+ +
+
+
{% trans "Token" %}:
+
{{ integration.token }}
+
+
+{% endblock %} diff --git a/readthedocs/templates/projects/integration_list.html b/readthedocs/templates/projects/integration_list.html index f4a089dcc0a..99f05ebfda5 100644 --- a/readthedocs/templates/projects/integration_list.html +++ b/readthedocs/templates/projects/integration_list.html @@ -12,35 +12,34 @@ {% block project_edit_content_header %}{% trans "Integrations" %}{% endblock %} {% block project_edit_content %} -

- Manage external integrations for project webhooks. -

- -
- {% csrf_token %} - -
- -

Recent Activity

- -
+
+ +
+
+ +
+
{% endblock %} diff --git a/readthedocs/templates/projects/integration_webhook_detail.html b/readthedocs/templates/projects/integration_webhook_detail.html new file mode 100644 index 00000000000..26b00864b12 --- /dev/null +++ b/readthedocs/templates/projects/integration_webhook_detail.html @@ -0,0 +1,102 @@ +{% extends "projects/project_edit_base.html" %} + +{% load i18n %} + +{% block title %}{% trans "Integration" %}{% endblock %} + +{% block nav-dashboard %} class="active"{% endblock %} + +{% block editing-option-edit-integrations %}class="active"{% endblock %} + +{% block project-integrations-active %}active{% endblock %} +{% block project_edit_content_header %} + {% blocktrans with type=integration.get_integration_type_display %} + Integration - {{ type }} + {% endblocktrans %} +{% endblock %} + +{% block project_edit_content %} + {% if integration.has_sync and integration.can_sync %} +

+ {% blocktrans %} + This webhook was configured when this project was imported. If this + integration is not functioning correctly, try resyncing the webhook: + {% endblocktrans %} +

+ +
+ {% csrf_token %} + +
+ {% else %} + {% comment %} + Display information for manual webhook set up if either case is true: + + * Integration doesn't have the ability to sync + * Integration has ability to sync, but we don't have the data returned + from the provider API on webhook creation (or webhook was automatically + created on new data from an old webhook) + {% endcomment %} + {% if integration.has_sync and not integration.can_sync %} +

+ {% blocktrans %} + This integration was created automatically from an existing webhook + configured on your repository. To make any changes to this webhook, + you'll need to update the configuration there. You can use the + following address to manually configure this webhook. + {% endblocktrans %} +

+ {% else %} +

+ {% blocktrans %} + To manually configure this webhook with your provider, use the + following address: + {% endblocktrans %} +

+ {% endif %} + +

+ {% url 'api_webhook' project_slug=project.slug integration_pk=integration.pk as webhook_url %} + {{ PRODUCTION_DOMAIN }}{{ webhook_url }} +

+ + {% block integration_details %}{% endblock %} + +

+ {% blocktrans %} + For more information on manually configuring a webhook, refer to + + our webhook documentation + + {% endblocktrans %} +

+ {% endif %} + +

Recent Activity

+ +
+ +
+ +
+ {% csrf_token %} + +
+{% endblock %}