From 60c938333995f2ac7399085d2ae90d2f5f3e33cd Mon Sep 17 00:00:00 2001 From: lafiona Date: Thu, 29 Sep 2022 03:52:36 -0400 Subject: [PATCH] ARROW-17512: [Doc] Updates to crossbow documentation for clarity (#13993) ## Overview While setting up a queue repository for testing changes to `crossbow`, we noticed some updates that can be made to help future developers set up their environment. ## Implementation 1. Clarify Travis CI auto-cancellation default behavior. 2. Fix broken links referenced by instructions. 3. Minor typos. ## Testing 1. Qualified by performing a directory level sphinx build and visually verifying the changes. ## Notes Thank you for your help on this pull request, @kevingurney! Lead-authored-by: Fiona La Co-authored-by: Kevin Gurney Signed-off-by: Antoine Pitrou --- .../continuous_integration/crossbow.rst | 64 +++++++++---------- .../continuous_integration/overview.rst | 6 +- 2 files changed, 32 insertions(+), 38 deletions(-) diff --git a/docs/source/developers/continuous_integration/crossbow.rst b/docs/source/developers/continuous_integration/crossbow.rst index 2515903a28da1..4b587f014d3f1 100644 --- a/docs/source/developers/continuous_integration/crossbow.rst +++ b/docs/source/developers/continuous_integration/crossbow.rst @@ -25,8 +25,8 @@ The content of ``arrow/dev/tasks`` directory aims for automating the process of Arrow packaging and integration testing. Packages: - - C++ and Python `conda-forge packages`_ for Linux, Mac and Windows - - Python `Wheels`_ for Linux, Mac and Windows + - C++ and Python `conda-forge packages`_ for Linux, macOS and Windows + - Python `Wheels`_ for Linux, macOS and Windows - C++ and GLib `Linux packages`_ for multiple distributions - Java for Gandiva @@ -46,21 +46,21 @@ Executors Individual jobs are executed on public CI services, currently: -- Linux: TravisCI, CircleCI, Azure Pipelines -- Mac: TravisCI, Azure Pipelines -- Windows: AppVeyor, Azure Pipelines +- Linux: GitHub Actions, Travis CI, Azure Pipelines +- macOS: GitHub Actions, Travis CI, Azure Pipelines +- Windows: GitHub Actions, Azure Pipelines Queue ~~~~~ Because of the nature of how the CI services work, the scheduling of jobs happens through an additional git repository, which acts like a job -queue for the tasks. Anyone can host a ``queue`` repository which is usually -called as ``crossbow``. +queue for the tasks. Anyone can host a ``queue`` repository (usually +named ``/crossbow``). -A job is a git commit on a particular git branch, containing only the required -configuration file to run the requested build (like ``.travis.yml``, -``appveyor.yml`` or ``azure-pipelines.yml``). +A job is a git commit on a particular git branch, containing the required +configuration files to run the requested builds (like ``.travis.yml``, +``azure-pipelines.yml``, or ``crossbow.yml`` for `GitHub Actions`_ ). Scheduler ~~~~~~~~~ @@ -74,22 +74,20 @@ Install The following guide depends on GitHub, but theoretically any git server can be used. -If you are not using the `ursacomputing/crossbow `_ +If you are not using the `ursacomputing/crossbow`_ repository, you will need to complete the first two steps, otherwise procede to step 3: 1. `Create the queue repository`_ -2. Enable `TravisCI`_, `Appveyor`_, `Azure Pipelines`_ and `CircleCI`_ - integrations on for the newly created queue repository. +2. Enable `Travis CI`_ and `Azure Pipelines`_ integrations for the newly + created queue repository. - - turn off Travis’ `auto cancellation`_ feature on branches - -3. Clone either ursacomputing/crossbow if you are using that, or the newly +3. Clone either `ursacomputing/crossbow`_ if you are using that, or the newly created repository next to the arrow repository: - By default the scripts looks for ``crossbow`` next to arrow repository, but - this can configured through command line arguments. + By default the scripts looks for a ``crossbow`` clone next to the ``arrow`` + directory, but this can configured through command line arguments. .. code:: bash @@ -110,22 +108,18 @@ to step 3: or pass as an argument to the CLI script ``--github-token`` -6. Export the previously created GitHub token on both CI services: +6. Add the previously created GitHub token to **Travis CI**: Use ``CROSSBOW_GITHUB_TOKEN`` encrypted environment variable. You can - set them at the following URLs, where ``ghuser`` is the GitHub + set it at the following URL, where ``ghuser`` is the GitHub username and ``ghrepo`` is the GitHub repository name (typically ``crossbow``): - - TravisCI: ``https://travis-ci.org///settings`` - - Appveyor: - ``https://ci.appveyor.com/project///settings/environment`` - - CircleCI: - ``https://circleci.com/gh///edit#env-vars`` - - On Appveyor check the ``skip branches without appveyor.yml`` checkbox - on the web UI under crossbow repository’s settings. + ``https://travis-ci.com///settings`` + - Confirm the `auto cancellation`_ feature is turned off for branch builds. This should be the default setting. + +.. line-block:: 7. Install Python (minimum supported version is 3.7): Miniconda is preferred, see installation instructions: @@ -176,8 +170,8 @@ The script does the following: 3. Reads and renders the required build configurations with the parameters substituted. -4. Create a branch per task, prefixed with the job id. For example to - build conda recipes on linux it will create a new branch: +4. Create a branch per task, prefixed with the job id. For example, to + build conda recipes on linux, it will create a new branch: ``crossbow@build--conda-linux``. 5. Pushes the modified branches to GitHub which triggers the builds. For @@ -251,11 +245,11 @@ see its help page: .. _conda-forge packages: conda-recipes .. _Wheels: python-wheels .. _Linux packages: linux-packages -.. _Create the queue repository: https://help.github.com/articles/creating-a-new-repository -.. _TravisCI: https://travis-ci.org/getting_started -.. _Appveyor: https://www.appveyor.com/docs/ -.. _CircleCI: https://circleci.com/docs/2.0/getting-started/ +.. _Create the queue repository: https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-new-repository +.. _Github Actions: https://docs.github.com/en/actions/quickstart +.. _Travis CI: https://travis-ci.com/getting-started/ .. _Azure Pipelines: https://docs.microsoft.com/en-us/azure/devops/pipelines/get-started/pipelines-sign-up -.. _auto cancellation: https://docs.travis-ci.com/user/customizing-the-build/#Building-only-the-latest-commit +.. _auto cancellation: https://docs.travis-ci.com/user/customizing-the-build/#building-only-the-latest-commit .. _Create a Personal Access Token: https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/ .. _setuptools_scm: https://pypi.python.org/pypi/setuptools_scm +.. _ursacomputing/crossbow: https://github.com/ursacomputing/crossbow diff --git a/docs/source/developers/continuous_integration/overview.rst b/docs/source/developers/continuous_integration/overview.rst index 3c21c1706317b..4a61120a0b3b2 100644 --- a/docs/source/developers/continuous_integration/overview.rst +++ b/docs/source/developers/continuous_integration/overview.rst @@ -31,7 +31,7 @@ Some files central to Arrow CI are: We use :ref:`Docker` in order to have portable and reproducible Linux builds, as well as running Windows builds in Windows containers. We use :ref:`Archery` and :ref:`Crossbow` to help co-ordinate the various CI tasks. -One thing to note is the some of the services defined in ``docker-compose.yml`` are interdependent. When running services locally, you must either manually build its dependencies first, or build it via the use of ``archery run ...`` which automatically finds and builds dependencies. +One thing to note is that some of the services defined in ``docker-compose.yml`` are interdependent. When running services locally, you must either manually build its dependencies first, or build it via the use of ``archery run ...`` which automatically finds and builds dependencies. There are numerous important directories in the Arrow project which relate to CI: @@ -69,8 +69,8 @@ Extended builds Crossbow is a subcomponent of Archery and can be used to manually trigger builds. The tasks which can be run on Crossbow can be found in the ``dev/tasks`` directory. This directory contains: - the file ``dev/tasks/tasks.yml`` containing the configuration for various tasks which can be run via Crossbow -- subdirectories containing different task templates (specified using `jinja2 syntax `_), divided roughly by language or package management system. +- subdirectories containing different task templates (specified using `jinja2 syntax `_), divided roughly by language or package management system. -Most of these tasks are run as part of the nightly builds, though also can be triggered manually by add a comment to a PR which begins with ``@github-actions crossbow submit`` followed by the name of the task to be run. +Most of these tasks are run as part of the nightly builds, though they can also be triggered manually by add a comment to a PR which begins with ``@github-actions crossbow submit`` followed by the name of the task to be run. For convenience purpose, the tasks in ``dev/tasks/tasks.yml`` are defined in groups, which makes it simpler for multiple tasks to be submitted to Crossbow at once. The task definitions here contain information about which service defined in ``docker-compose.yml`` to run, the CI service to run the task on, and which template file to use as the basis for that task.