Skip to content

Commit

Permalink
ARROW-17512: [Doc] Updates to crossbow documentation for clarity (#13993
Browse files Browse the repository at this point in the history
)

## Overview
While setting up a queue repository for testing changes to `crossbow`, we noticed some updates that can be made to help future developers set up their environment. 

## Implementation
1. Clarify Travis CI auto-cancellation default behavior.
2. Fix broken links referenced by instructions.
3. Minor typos.

## Testing
1. Qualified by performing a directory level sphinx build and visually verifying the changes.

## Notes
Thank you for your help on this pull request, @kevingurney!

Lead-authored-by: Fiona La <[email protected]>
Co-authored-by: Kevin Gurney <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>
  • Loading branch information
lafiona and kevingurney authored Sep 29, 2022
1 parent ec714db commit 60c9383
Show file tree
Hide file tree
Showing 2 changed files with 32 additions and 38 deletions.
64 changes: 29 additions & 35 deletions docs/source/developers/continuous_integration/crossbow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ The content of ``arrow/dev/tasks`` directory aims for automating the process of
Arrow packaging and integration testing.

Packages:
- C++ and Python `conda-forge packages`_ for Linux, Mac and Windows
- Python `Wheels`_ for Linux, Mac and Windows
- C++ and Python `conda-forge packages`_ for Linux, macOS and Windows
- Python `Wheels`_ for Linux, macOS and Windows
- C++ and GLib `Linux packages`_ for multiple distributions
- Java for Gandiva

Expand All @@ -46,21 +46,21 @@ Executors

Individual jobs are executed on public CI services, currently:

- Linux: TravisCI, CircleCI, Azure Pipelines
- Mac: TravisCI, Azure Pipelines
- Windows: AppVeyor, Azure Pipelines
- Linux: GitHub Actions, Travis CI, Azure Pipelines
- macOS: GitHub Actions, Travis CI, Azure Pipelines
- Windows: GitHub Actions, Azure Pipelines

Queue
~~~~~

Because of the nature of how the CI services work, the scheduling of
jobs happens through an additional git repository, which acts like a job
queue for the tasks. Anyone can host a ``queue`` repository which is usually
called as ``crossbow``.
queue for the tasks. Anyone can host a ``queue`` repository (usually
named ``<ghuser>/crossbow``).

A job is a git commit on a particular git branch, containing only the required
configuration file to run the requested build (like ``.travis.yml``,
``appveyor.yml`` or ``azure-pipelines.yml``).
A job is a git commit on a particular git branch, containing the required
configuration files to run the requested builds (like ``.travis.yml``,
``azure-pipelines.yml``, or ``crossbow.yml`` for `GitHub Actions`_ ).

Scheduler
~~~~~~~~~
Expand All @@ -74,22 +74,20 @@ Install
The following guide depends on GitHub, but theoretically any git
server can be used.

If you are not using the `ursacomputing/crossbow <https://github.com/ursacomputing/crossbow>`_
If you are not using the `ursacomputing/crossbow`_
repository, you will need to complete the first two steps, otherwise procede
to step 3:

1. `Create the queue repository`_

2. Enable `TravisCI`_, `Appveyor`_, `Azure Pipelines`_ and `CircleCI`_
integrations on for the newly created queue repository.
2. Enable `Travis CI`_ and `Azure Pipelines`_ integrations for the newly
created queue repository.

- turn off Travis’ `auto cancellation`_ feature on branches

3. Clone either ursacomputing/crossbow if you are using that, or the newly
3. Clone either `ursacomputing/crossbow`_ if you are using that, or the newly
created repository next to the arrow repository:

By default the scripts looks for ``crossbow`` next to arrow repository, but
this can configured through command line arguments.
By default the scripts looks for a ``crossbow`` clone next to the ``arrow``
directory, but this can configured through command line arguments.

.. code:: bash
Expand All @@ -110,22 +108,18 @@ to step 3:
or pass as an argument to the CLI script ``--github-token``

6. Export the previously created GitHub token on both CI services:
6. Add the previously created GitHub token to **Travis CI**:

Use ``CROSSBOW_GITHUB_TOKEN`` encrypted environment variable. You can
set them at the following URLs, where ``ghuser`` is the GitHub
set it at the following URL, where ``ghuser`` is the GitHub
username and ``ghrepo`` is the GitHub repository name (typically
``crossbow``):

- TravisCI: ``https://travis-ci.org/<ghuser>/<ghrepo>/settings``
- Appveyor:
``https://ci.appveyor.com/project/<ghuser>/<ghrepo>/settings/environment``
- CircleCI:
``https://circleci.com/gh/<ghuser>/<ghrepo>/edit#env-vars``

On Appveyor check the ``skip branches without appveyor.yml`` checkbox
on the web UI under crossbow repository’s settings.
``https://travis-ci.com/<ghuser>/<ghrepo>/settings``

- Confirm the `auto cancellation`_ feature is turned off for branch builds. This should be the default setting.

.. line-block::
7. Install Python (minimum supported version is 3.7):

Miniconda is preferred, see installation instructions:
Expand Down Expand Up @@ -176,8 +170,8 @@ The script does the following:
3. Reads and renders the required build configurations with the
parameters substituted.

4. Create a branch per task, prefixed with the job id. For example to
build conda recipes on linux it will create a new branch:
4. Create a branch per task, prefixed with the job id. For example, to
build conda recipes on linux, it will create a new branch:
``crossbow@build-<id>-conda-linux``.

5. Pushes the modified branches to GitHub which triggers the builds. For
Expand Down Expand Up @@ -251,11 +245,11 @@ see its help page:
.. _conda-forge packages: conda-recipes
.. _Wheels: python-wheels
.. _Linux packages: linux-packages
.. _Create the queue repository: https://help.github.com/articles/creating-a-new-repository
.. _TravisCI: https://travis-ci.org/getting_started
.. _Appveyor: https://www.appveyor.com/docs/
.. _CircleCI: https://circleci.com/docs/2.0/getting-started/
.. _Create the queue repository: https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-new-repository
.. _Github Actions: https://docs.github.com/en/actions/quickstart
.. _Travis CI: https://travis-ci.com/getting-started/
.. _Azure Pipelines: https://docs.microsoft.com/en-us/azure/devops/pipelines/get-started/pipelines-sign-up
.. _auto cancellation: https://docs.travis-ci.com/user/customizing-the-build/#Building-only-the-latest-commit
.. _auto cancellation: https://docs.travis-ci.com/user/customizing-the-build/#building-only-the-latest-commit
.. _Create a Personal Access Token: https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/
.. _setuptools_scm: https://pypi.python.org/pypi/setuptools_scm
.. _ursacomputing/crossbow: https://github.com/ursacomputing/crossbow
6 changes: 3 additions & 3 deletions docs/source/developers/continuous_integration/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Some files central to Arrow CI are:

We use :ref:`Docker<docker-builds>` in order to have portable and reproducible Linux builds, as well as running Windows builds in Windows containers. We use :ref:`Archery<Archery>` and :ref:`Crossbow<Crossbow>` to help co-ordinate the various CI tasks.

One thing to note is the some of the services defined in ``docker-compose.yml`` are interdependent. When running services locally, you must either manually build its dependencies first, or build it via the use of ``archery run ...`` which automatically finds and builds dependencies.
One thing to note is that some of the services defined in ``docker-compose.yml`` are interdependent. When running services locally, you must either manually build its dependencies first, or build it via the use of ``archery run ...`` which automatically finds and builds dependencies.

There are numerous important directories in the Arrow project which relate to CI:

Expand Down Expand Up @@ -69,8 +69,8 @@ Extended builds
Crossbow is a subcomponent of Archery and can be used to manually trigger builds. The tasks which can be run on Crossbow can be found in the ``dev/tasks`` directory. This directory contains:

- the file ``dev/tasks/tasks.yml`` containing the configuration for various tasks which can be run via Crossbow
- subdirectories containing different task templates (specified using `jinja2 syntax <https://jinja2docs.readthedocs.io/en/stable/>`_), divided roughly by language or package management system.
- subdirectories containing different task templates (specified using `jinja2 syntax <https://jinja.palletsprojects.com/>`_), divided roughly by language or package management system.

Most of these tasks are run as part of the nightly builds, though also can be triggered manually by add a comment to a PR which begins with ``@github-actions crossbow submit`` followed by the name of the task to be run.
Most of these tasks are run as part of the nightly builds, though they can also be triggered manually by add a comment to a PR which begins with ``@github-actions crossbow submit`` followed by the name of the task to be run.

For convenience purpose, the tasks in ``dev/tasks/tasks.yml`` are defined in groups, which makes it simpler for multiple tasks to be submitted to Crossbow at once. The task definitions here contain information about which service defined in ``docker-compose.yml`` to run, the CI service to run the task on, and which template file to use as the basis for that task.

0 comments on commit 60c9383

Please sign in to comment.