Skip to content

Commit

Permalink
Add webhook modeling to projects (#2764)
Browse files Browse the repository at this point in the history
* Add start of work to add webhook modeling for each integration

* Lint cleanup

* Add authentication to generic webhook

* Add tests, docs, and lint fixes

* Fix bitbucket operations for webhook create/update

* Doc updates

* Update integration token generator and make service debug logging more permissive

* Redirect user to integration detail page on create

* Add information for details page, drop automatic webhook mention
  • Loading branch information
agjohnson authored May 10, 2017
1 parent 125421e commit a66bd56
Show file tree
Hide file tree
Showing 25 changed files with 1,327 additions and 251 deletions.
56 changes: 44 additions & 12 deletions docs/getting_started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -83,33 +83,66 @@ Then in your ``conf.py``:

.. _this blog post: http://ericholscher.com/blog/2016/mar/15/dont-use-markdown-for-technical-docs/

.. _connect-account:

Sign Up and Connect an External Account
---------------------------------------

.. TODO Update this with GitLab support later
If you are going to import a repository from GitHub or Bitbucket, you should
connect your account to your provider first. Connecting your account allows for
easier importing and enables Read the Docs to configure your repository webhooks
automatically.

To connect your account, got to your *Settings* dashboard and select *Connected
Services*. From here, you'll be able to connect to your GitHub or Bitbucket
account. This process will ask you to authorize a connection to Read the Docs,
that allows us to read information about and clone your repositories.

.. _import-docs:

Import Your Docs
----------------

`Sign up`_ for an account on RTD, then `log in`_. Visit your dashboard_ and click
Import_ to add your project to the site. Fill in the name and description, then
specify where your repository is located. This is normally the URL or path name
you'd use to checkout, clone, or branch your code. Some examples:
To import a repository, visit your dashboard_ and click Import_.

If you have a connected account, you will see a list of your repositories that
we are able to import. To import one of these projects, just click the import
icon next to the repository you'd like to import. This will bring up a form that
is already filled with your project's information. Feel free to edit any of
these properties, and the click **Next** to build your documentation.

Manually Import Your Docs
~~~~~~~~~~~~~~~~~~~~~~~~~

If you do not have a connected account, you will need select **Import Manually**
and enter the information for your repository yourself. You will also need to
manually configure the webhook for your repository as well. When importing your
project, you will be asked for the repository URL, along with some other
information for you new project. The URL is normally the URL or path name you'd
use to checkout, clone, or branch your repository. Some examples:

* Git: ``http://github.com/ericholscher/django-kong.git``
* Subversion: ``http://varnish-cache.org/svn/trunk``
* Mercurial: ``https://bitbucket.org/ianb/pip``
* Subversion: ``http://varnish-cache.org/svn/trunk``
* Bazaar: ``lp:pasta``

Add an optional homepage URL and some tags, then click "Create".
Add an optional homepage URL and some tags, and then click **Next**.

Once your project is created, you'll need to manually configure the repository
webhook if you would like to have new changesets to trigger builds for your
project on Read the Docs. Go to your project's **Integrations** page to
configure a new webhook, or see :ref:`our steps for webhook creation
<webhook-creation>` for more information on this process.

Within a few seconds your code will automatically be fetched from your public repository,
and the documentation will be built.
Check out our :doc:`builds` page to learn more about how we build your docs,
and to troubleshoot any issues that arise.

If you want to keep your code updated as you commit,
configure your code repository to hit our `Post Commit Hooks`_.
This will rebuild your docs every time you push your code.

We support multiple versions of your code. You can read more about how to use this well on our :doc:`versions` page.
Read the Docs will host multiple versions of your code. You can read more about
how to use this well on our :doc:`versions` page.

If you have any more trouble, don't hesitate to reach out to us. The :doc:`support` page has more information on getting in touch.

Expand All @@ -126,4 +159,3 @@ If you have any more trouble, don't hesitate to reach out to us. The :doc:`suppo
.. _log in: http://readthedocs.org/accounts/login
.. _dashboard: http://readthedocs.org/dashboard
.. _Import: http://readthedocs.org/dashboard/import
.. _Post Commit Hooks: http://readthedocs.org/docs/read-the-docs/en/latest/webhooks.html
157 changes: 107 additions & 50 deletions docs/webhooks.rst
Original file line number Diff line number Diff line change
@@ -1,78 +1,135 @@
Webhooks
========

Webhooks are pretty amazing, and help to turn the web into a push instead of
pull platform. We have support for hitting a URL whenever you commit to your
project and we will try and rebuild your docs. This only rebuilds them if
something has changed, so it is cheap on the server side. As anyone who has
worked with push knows, pushing a doc update to your repo and watching it get
updated within seconds is an awesome feeling.
The primary method that Read the Docs uses to detect changes to your
documentation is through the use of *webhooks*. Webhooks are configured with
your repository provider, such as GitHub or Bitbucket, and with each commit,
merge, or other change to your repository, Read the Docs is notified. When we
receive a webhook notification, we determine if the change is related to an
active version for your project, and if it is, a build is triggered for that
version.

GitHub
---------
.. _integration-detail:

If your project is hosted on GitHub, you can easily add a hook that will rebuild
your docs whenever you push updates:
Webhook Integrations
--------------------

* Go to the "Settings" page for your project
* Click "Integrations & Services"
* In the "Services" section, click "Add service"
* In the list of available services, click "ReadTheDocs"
* Leave "Active" checked
* Click "Add service"
You'll find a list of configured webhook integrations on your project's admin
dashboard, under **Integrations**. You can select any of these integrations to
see the *integration detail page*. This page has additional configuration
details and a list of HTTP exchanges that have taken place for the integration.

.. note:: The GitHub URL in your Read the Docs project must match the URL on GitHub. The URL is case-sensitive.
.. _webhook-creation:

If you ever need to manually set the webhook on GitHub,
you can point it at ``https://readthedocs.org/github``.
Webhook creation
----------------

Bitbucket
-----------
If you import a project using a :ref:`connected account <connect-account>`, a
webhook will be set up automatically for your repository. However, if your
project was not imported through a connected account, you may need to
manually configure a webhook for your project.

To manually set up a webhook, click **Add integration** on your project's
**Integrations** admin dashboard page and select the integration type you'd like
to add. After you have added the integration, you'll see a URL for the
integration on the :ref:`integration detail page <integration-detail>`. Use this
URL when setting up a new webhook with your provider -- these steps vary
depending on the provider:

GitHub
~~~~~~

* Go to the **Settings** page for your project
* Click **Webhooks** and then **Add webhook**
* For **Payload URL**, use the URL of the integration on Read the Docs, found on
the :ref:`integration detail page <integration-detail>` page
* For **Content type**, both *application/json* and
*application/x-www-form-urlencoded* work
* Select **Just the push event**
* Finish by clicking **Add webhook**

If your project is hosted on Bitbucket, you can easily add a hook that will rebuild
your docs whenever you push updates:
.. note:: The webhook secret is not yet respected

* Go to the "admin" page for your project
* Click "Services"
* In the available service hooks, select "Read the Docs"
* Click "Add service"
Bitbucket
~~~~~~~~~

If you ever need to manually set the webhook on Bitbucket,
you can point it at ``https://readthedocs.org/bitbucket``.
* Go to the **Settings** page for your project
* Click **Webhooks** and then **Add webhook**
* For **URL**, use the URL of the integration on Read the Docs, found on the
:ref:`integration detail page <integration-detail>` page
* Under **Triggers**, **Repository push** should be selected
* Finish by clicking **Save**

GitLab
---------
~~~~~~

* Go to the **Settings** page for your project
* Click **Integrations**
* For **URL**, use the URL of the integration on Read the Docs, found on the
:ref:`integration detail page <integration-detail>` page
* Leave the default **Push events** selected
* Finish by clicking **Add Webhook**

If your project is hosted on GitLab, you can easily add a hook that will rebuild
your docs whenever you push updates.
Using the generic API integration
---------------------------------

* Go to the "Settings" page for your project
* Click "Integrations"
* In the "URL" section, enter ``https://readthedocs.org/gitlab``
* Leave the default "Push events" selected
* Click "Add Webhook"
For repositories that are not hosted with a supported provider, we also offer a
generic API endpoint for triggering project builds. Similar to webhook
integrations, this integration has a specific URL, found on the
:ref:`integration detail page <integration-detail>`.

Others
------
Token authentication is required to use the generic endpoint, you will find this
token on the integration details page. The token should be passed in as a
request parameter, either as form data or as part of JSON data input.

Your ReadTheDocs project detail page has your post-commit hook on it; it will
look something along the lines of ``http://readthedocs.org/build/<project_name>``.
Regardless of which revision control system you use, you can just hit this URL
to kick off a rebuild.
Parameters
~~~~~~~~~~

The following parameters available to customize the behavior of custom webhooks:
This endpoint accepts the following arguments during an HTTP POST:

* ``'version_slug'``: The build version to trigger build for (defaults to ``'latest'``)
branches
The names of the branches to trigger builds for. This can either be an array
of branch name strings, or just a single branch name string.

Example::
Default: **latest**

$ curl -X POST --data "version_slug=$VERSION" https://readthedocs.org/build/$PROJECT_NAME
token
The integration token. You'll find this value on the
:ref:`integration detail page <integration-detail>` page.

You could make this part of a hook using Git_, Subversion_, Mercurial_, or
Bazaar_, perhaps through a simple script that accesses the build URL using
``wget`` or ``curl``.
For example, the cURL command to build the ``dev`` branch, using the token
``1234``, would be::

curl -X POST -d "branches=dev" -d "token=1234" https://readthedocs.org/api/v2/webhook/example-project/1/

A command like the one above could be called from a cron job or from a hook
inside Git_, Subversion_, Mercurial_, or Bazaar_.

.. _Git: http://www.kernel.org/pub/software/scm/git/docs/githooks.html
.. _Subversion: http://mikewest.org/2006/06/subversion-post-commit-hooks-101
.. _Mercurial: http://hgbook.red-bean.com/read/handling-repository-events-with-hooks.html
.. _Bazaar: http://wiki.bazaar.canonical.com/BzrHooks

Authentication
~~~~~~~~~~~~~~

This endpoint requires authentication. If authenticating with an integration
token, a check will determine if the token is valid and matches the given
project. If instead an authenticated user is used to make this request, a check
will be performed to ensure the authenticated user is an owner of the project.

Debugging webhooks
------------------

If you are experiencing problems with an existing webhook, you may be able to
use the integration detail page to help debug the issue. Each project
integration, such as a webhook or the generic API endpoint, stores the HTTP
exchange that takes place between Read the Docs and the external source. You'll
find a list of these exchanges in any of the integration detail pages.

Resyncing webhooks
------------------

It might be necessary to re-establish a webhook if you are noticing problems.
To resync a webhook from Read the Docs, visit the integration detail page and
follow the directions for re-syncing your repository webhook.
60 changes: 47 additions & 13 deletions media/css/core.css
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,12 @@ h3 > span.link-help {
float: right;
}

form.form-wide input[type='text'],
form.form-wide select,
form.form-wide textarea {
width: 100%;
}

/* content */

#content { padding-top: 50px; }
Expand Down Expand Up @@ -925,6 +931,22 @@ body .edit-toggle { display: none; }

.navigable ul input[type=text] { width: 164px; }

div.button-bar ul {
list-style: none;
text-align: right;
}

div.button-bar ul li {
display: inline-block;
}

div.button-bar li a.button,
div.button-bar li input[type="submit"],
div.button-bar li input[type="button"],
div.button-bar li button {
margin-top: .5em;
margin-bottom: .5em;
}

select.dropdown { display: none; }
.dropdown > a { font-family: "ff-meta-web-pro", "ff-meta-web-pro-1", "ff-meta-web-pro-2", Arial, "Helvetica Neue", sans-serif; color: #666; font-weight: bold; padding: 8px 15px; border: none; background: #e6e6e6 url(../images/gradient.png) repeat-x bottom left; margin: 30px 5px 20px 0; text-shadow: 0 1px 0 rgba(255, 255, 255, 1); border: 1px solid #bfbfbf; display: block; text-decoration: none; box-shadow: 0 1px 0 rgba(0, 0, 0, 0.1), 0 1px 0 rgba(255, 255, 255, 0.5) inset; -moz-box-shadow: 0 1px 0 rgba(0, 0, 0, 0.1), 0 1px 0 rgba(255, 255, 255, 0.5) inset; -webkit-box-shadow: 0 1px 0 rgba(0, 0, 0, 0.1), 0 1px 0 rgba(255, 255, 255, 0.5) inset; }
Expand Down Expand Up @@ -1044,28 +1066,40 @@ div.module-list-wrapper.httpexchanges li span.status.status-fail {
background: #a55;
}

div.integration-details {
margin: 1em;
}

div.integration-details dl dt,
div.httpexchange dl dt {
display: inline-block;
font-weight: bold;
font-family: 'inconsolata', 'bitstream vera sans mono', 'andale mono', 'lucida console', monospace;
font-size: .9em;
display: inline-block;
font-weight: bold;
}
div.httpexchange dl dt {
font-family: 'inconsolata', 'bitstream vera sans mono', 'andale mono', 'lucida console', monospace;
font-size: .9em;
}

div.integration-details dl dd,
div.httpexchange dl dd {
display: inline;
font-family: 'inconsolata', 'bitstream vera sans mono', 'andale mono', 'lucida console', monospace;
}
div.httpexchange dl dd {
display: inline;
font-family: 'inconsolata', 'bitstream vera sans mono', 'andale mono', 'lucida console', monospace;
font-size: .9em;
font-size: .9em;
}

div.integration-details dl dd:after,
div.httpexchange dl dd:after {
display: block;
content: '';
display: block;
content: '';
}

div.httpexchange div.highlight pre {
padding: 1em;
background: #f4f4f4;
border: 1px solid #ccc;
font-size: .9em;
padding: 1em;
background: #f4f4f4;
border: 1px solid #ccc;
font-size: .9em;
}

/* Pygments */
Expand Down
1 change: 1 addition & 0 deletions readthedocs/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@
'lang_slug': LANGUAGES_REGEX,
'version_slug': VERSION_SLUG_REGEX,
'filename_slug': '(?:.*)',
'integer_pk': r'[\d]+',
}
9 changes: 9 additions & 0 deletions readthedocs/core/fields.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
"""Shared model fields and defaults"""

import binascii
import os


def default_token():
"""Generate default value for token field"""
return binascii.hexlify(os.urandom(20)).decode()
Loading

0 comments on commit a66bd56

Please sign in to comment.