doc: Repository structure and modules

doc/development_process/index.rst

@@ -7,9 +7,9 @@ Development Model
 .. toctree::
    :maxdepth: 1
 
+   repo_structure
    release_process
    proposals
-   code_flow
    dev_env_and_tools
    issues
    api_lifecycle

doc/development_process/repo_structure.rst

@@ -0,0 +1,100 @@
+.. _repo-structure:
+
+Repository structure
+####################
+
+Introduction
+************
+
+The Zephyr Project codebase was, for a long time, contained in a single `Git`_
+repository. As the project evolved, it became clear that, :ref:`due to multiple
+reasons <west-why-multi-repo>`, the project's code and supporting files needed
+to be split out into a set of repositories instead.
+
+To that end a new tool, :ref:`west`, was developed and introduced to ease the
+burden of managing multiple Git repositories.
+
+Structure
+*********
+
+The Zephyr Project codebase is composed of several `Git`_ repositories,
+organized in a :ref:`star topology <west-topologies>` as depicted below.
+
+.. figure:: zephyr-repos.png
+    :align: center
+    :alt: Zephyr repository structure
+
+    Zephyr repository structure
+
+There are two types of repositories as shown by the figure:
+
+  * The `manifest repository`_ or also commonly named the "zephyr" repository
+    contains the `west.yml`_ :ref:`manifest file <west-manifests>` that
+    lists all other repositories that the manifest repository depends on.
+  * The project repositories are the ones containing :ref:`west projects
+    <west-installation>`. Most (but not all) of those west projects are also
+    :ref:`modules <modules>`.
+
+All repositories are hosted under the `zephyrproject-rtos`_ GitHub organization,
+since only repositories hosted under it are allowed to be featured in the
+official Zephyr manifest file. This ensures that the repositories and all
+referenced revisions will always be available to Zephyr users.
+
+.. _branches:
+
+Branches
+********
+
+Both the `manifest repository`_ and the project repositories may include the
+following branches:
+
+``master``
+  Which contains the latest state of development. In the case of :ref:`modules
+  <modules>` that fork an external project this may not reflect the latest state
+  of the upstream project's development, since projects are upmerged manually
+  when required by Zephyr itself.
+
+``topic-\*``
+  Topic branches that are used for shared development of a new feature
+
+``vx.y-branch``
+  Branches which track maintenance releases based on a major
+  release
+
+``backport-*``
+  Branches that are created automatically when a Pull Request is labeled to be
+  backported to a ``vx.y-branch``
+
+Development in topic branches before features go to mainline allows teams to
+work independently on a subsystem or a feature, improves efficiency and
+turnaround time, and encourages collaboration and streamlines communication
+between developers.
+
+Changes submitted to a development topic branch can evolve and improve
+incrementally in a branch, before they are submitted to the mainline tree for
+final integration.
+
+By dedicating an isolated branch to complex features, it's
+possible to initiate in-depth discussions around new additions before
+integrating them into the official project.
+
+Topic branches
+==============
+
+Development topic branch owners have the following responsibilities:
+
+- Use the infrastructure and tools provided by the project (GitHub, Git)
+- Review changes coming from team members and request review from branch owners
+  when submitting changes.
+- Keep the branch in sync with upstream and update on a regular basis.
+- Push changes frequently to upstream using the following methods:
+
+  - GitHub pull requests: for example, when reviews have not been done in the local
+    branch (one-man branch).
+  - Merge requests: When a set of changes has been done in a local branch and
+    has been reviewed and tested in a topic branch.
+
+.. _Git: https://git-scm.com/
+.. _zephyrproject-rtos: https://github.com/zephyrproject-rtos
+.. _manifest repository: https://github.com/zephyrproject-rtos/zephyr
+.. _west.yml: https://github.com/zephyrproject-rtos/zephyr/blob/master/west.yml

doc/guides/modules.rst

@@ -26,25 +26,32 @@ build and configure them, respectively. Module :file:`CMakeLists.txt` files are
 added to the build using CMake's `add_subdirectory()`_ command, and the
 :file:`Kconfig` files are included in the build's Kconfig menu tree.
 
-If you have :ref:`west <west>` installed, you don't need to worry about how
-this variable is defined unless you are adding a new module. The build system
-knows how to use west to set :makevar:`ZEPHYR_MODULES`. You can add additional
-modules to this list by setting the :makevar:`ZEPHYR_EXTRA_MODULES` CMake
-variable or by adding a :makevar:`ZEPHYR_EXTRA_MODULES` line to ``.zephyrrc``
-(See the section on :ref:`env_vars` for more details). This can be
-useful if you want to keep the list of modules found with west and also add
-your own.
-
-See the section about :ref:`west-multi-repo` for more details.
-
-Finally, you can also specify the list of modules yourself in various ways, or
-not use modules at all if your application doesn't need them.
-
-
+Each module used by upstream Zephyr is contained in its own Git repository, and
+the set of Git repositories is managed by :ref:`west <west>`. See
+:ref:`repo-structure` for more information on the Zephyr repository topology,
+and :ref:`west-multi-repo` for more details on how west manages multiple
+repositories.
+
+In that way, if you use upstream Zephyr and therefore west, you don't need to
+worry about how this variable is defined at all. The build system knows how to
+use west to set :makevar:`ZEPHYR_MODULES`. You can add additional modules to
+this list by setting the :makevar:`ZEPHYR_EXTRA_MODULES` CMake variable or by
+setting the :makevar:`ZEPHYR_EXTRA_MODULES` environment varialbe in
+``.zephyrrc`` (See the section on :ref:`env_vars` for more details). This can be
+useful if you want to extend the list of modules found with west with your own
+local ones without having to modify the `manifest file <west.yml_>`_.
+
+It is also possible to avoid using west or modules at all, see the section below
+for more information.
 
 Module Inclusion
 ****************
 
+.. note::
+   In all cases below the :makevar:`ZEPHYR_EXTRA_MODULES` environment and CMake
+   variables are taken into account only if the :makevar:`ZEPHYR_MODULES` CMake
+   variable is set, since the former is an extension to the latter.
+
 .. _modules_using_west:
 
 Using West
@@ -179,13 +186,15 @@ Zephyr project issue tracking system with details about the module and how it
 integrates into the project. If the request is approved, a new repository will
 created by the project team and initialized with basic information that would
 allow submitting code to the module project following the project contribution
-guidelines.
+guidelines. The request may be approved by a maintainer of a particular
+subsystem or may have to be escalated to the TSC.
 
-All modules should be hosted in repositories under the Zephyr organization. The
-manifest should only point to repositories maintained under the Zephyr project.
-If a module is maintained as a fork of another project on Github, the Zephyr module
-related files and changes in relation to upstream need to be maintained in a
-special branch named ``zephyr``.
+All modules must be hosted in repositories under the `Zephyr organization
+<zephyrproject-rtos_>`_, and thus the manifest must only point to repositories
+maintained under the Zephyr project. If a module is maintained as a fork of an
+externally maintained project on Github or elsehwere, the Zephyr module related
+files and changes in relation to upstream must be committed to the module
+repository's ``master`` branch. See :ref:`branches` for additional information.
 
 Process
 -------
@@ -196,11 +205,12 @@ Follow the following steps to request a new module:
    created
 #. Propose a name for the repository to be created under the Zephyr project
    organization on Github.
-#. Maintainers from the Zephyr project will create the repository and initialize
-   it. You will be added as a collaborator in the new repository.
+#. If the request is approved, maintainers from the Zephyr project will create
+   the repository and initialize it. You will be added as a collaborator in the
+   new repository.
 #. Submit the module content (code) to the new repository following the
    guidelines described :ref:`here <modules_using_west>`.
-#. Add a new entry to the :zephyr_file:`west.yml` with the following
+#. Add a new entry to the `west.yml`_ manifest file with the following
    information:
 
    .. code-block:: console
@@ -231,7 +241,7 @@ Changes to existing modules
 #. Submit the changes using a pull request to an existing repository following
    the :ref:`contribution guidelines <contribute_guidelines>`.
 #. Submit a pull request changing the entry referencing the module into the
-   :zephyr_file:`west.yml` of the main Zephyr tree with the following
+   `west.yml`_ of the main Zephyr tree with the following
    information:
 
    .. code-block:: console
@@ -253,9 +263,43 @@ Where 23 in the example above indicated the pull request number submitted to the
 *my_module* repository. Once the module changes are reviewed and merged, the
 revision needs to be changed to the commit hash from the module repository.
 
-
+Upmerging forks
+===============
+
+When a module is in fact a fork of an existing, external project that is hosted
+on GitHub or elsewhere, it is often necessary to upmerge it: update the copy
+that is hosted under the `zephyrproject-rtos`_ GitHub organization with the
+latest changes from the external upstream repository.
+
+The procedure for upmerging module repositories that are forks of external
+projects is constrained by the following two factors:
+
+* History must **never** be rewritten in any repository that is referenced in
+  the `west.yml`_ manifest file, since that could break older revisions of the
+  repository set as a whole. This therefore excludes rebasing the module
+  repository and keeping the zephyr-specific changes always on top.
+* GitHub does not deal correctly with merge commits contained inside Pull
+  Requests, which prevents us from using regular ``git merge`` operations
+  without having to resort to pushing directly to the repository.
+
+Due to the above, this is the procedure that you must use in order to upmerge a
+module repository with the latest changes from the external upstream
+repository:
+
+#. Make a note of the current upstream revision that the module repository is
+   currently upmerged to.
+#. Take the changes from the current to the latest upstream revision and apply
+   them to the current ``master`` branch of the module repository. There are
+   several ways of achieving this  It is important to note that the
+   module repository may not share a common Git history with the original
+   upstream repository, so using ``git diff`` and ``git apply`` might be the
+   simplest course of action.
+#. Submit a pull request against the module and zephyr (manifest) repository as
+   described in :ref:`changes_to_existing_module`.
 
 .. _CMake list: https://cmake.org/cmake/help/latest/manual/cmake-language.7.html#lists
 .. _add_subdirectory(): https://cmake.org/cmake/help/latest/command/add_subdirectory.html
 
 .. _GitHub issues: https://github.com/zephyrproject-rtos/zephyr/issues
+.. _zephyrproject-rtos: https://github.com/zephyrproject-rtos
+.. _west.yml: https://github.com/zephyrproject-rtos/zephyr/blob/master/west.yml

doc/guides/west/repo-tool.rst

@@ -75,6 +75,8 @@ Finally, any repository managed by a west installation can contain
 provided by that project. This includes the manifest repository and any project
 repository.
 
+.. _west-topologies:
+
 Topologies supported
 ********************
 

doc/guides/west/why.rst

@@ -57,6 +57,8 @@ examined in detail:
     repositories (**R4**)
   - Requires hardcoding of the paths/locations of the external repositories
 
+.. _west-why-multi-repo:
+
 Multiple Git Repositories
 *************************