doc: Repository structure and modules #20067

carlescufi · 2019-10-23T14:47:18Z

Extend the doc for repo structure and modules.

carlescufi · 2019-10-23T14:51:24Z

doc/guides/modules.rst

+
+#. Make a note of the current upstream revision that the module repository is
+   currently upmerged to.
+#. Take the changes from the current to the latest upstream revision and apply


This might be controversial, but it's the only way I can think of to avoid the usual upmerge pitfalls.
Perhaps we should add a templated commit message here that says:

upmerge: Update from SHA1 to SHA2 Origin: project@SHA2

The other option is to allow multiple upmerge strategies

To say publicly what I've said before in slack/telecons: When I'm working on forks like those Zephyr requires I usually use a workspace with access to every upstream source. I find it extremely inconvenient to be guessing what upstream commit a particular fork commit is supposed to be equivalent to. A comment in a commit would at least provide a clue, but allowing real merge commits where you can see the upstream tag in its canonical form associated with the real SHA1 is the only sane way to be doing cross-repository comparisons.

I agree that real merge commits (if merging) is the sane way to handle it.

I still think we should consider allowing sporadic rebasing, but that's a different question.

zephyrbot · 2019-10-23T15:11:29Z

All checks are passing now.

Tip: The bot edits this comment instead of posting a new one, so you can check the comment's history to see earlier messages.

Replace the former "code flow and branches" section with a new one that covers the whole repository structure and the branches that each repository may have. Signed-off-by: Carles Cufi <[email protected]>

Rework some of the existing sections for clarity and extend the guide to cover upmerges. Signed-off-by: Carles Cufi <[email protected]>

mbolivar · 2019-10-23T18:14:37Z

doc/development_process/repo_structure.rst

+  * The `manifest repository`_ or also commonly named the "zephyr" repository
+    contains the `west.yml`_ :ref:`manifest file <west-manifests>` that
+    lists all other repositories that the manifest repository depends on.
+  * The project repositories are the ones containing :ref:`west projects


The project repositories are set in the `projects:` part of :file:`west.yml`.

They aren't contained there as much as they are the same thing, no?

In @mbolivar suggested edit, use double back-ticks around projects: (not single back ticks)

mbolivar · 2019-10-23T18:14:57Z

doc/development_process/repo_structure.rst

+    lists all other repositories that the manifest repository depends on.
+  * The project repositories are the ones containing :ref:`west projects
+    <west-installation>`. Most (but not all) of those west projects are also
+    :ref:`modules <modules>`.


I'd say Zephyr modules just to disambiguate.

mbolivar · 2019-10-23T18:16:59Z

doc/development_process/repo_structure.rst

+- Push changes frequently to upstream using the following methods:
+
+  - GitHub pull requests: for example, when reviews have not been done in the local
+    branch (one-man branch).


(only one person worked on the branch)

mbolivar · 2019-10-23T18:17:20Z

doc/development_process/repo_structure.rst

+
+  - GitHub pull requests: for example, when reviews have not been done in the local
+    branch (one-man branch).
+  - Merge requests: When a set of changes has been done in a local branch and


What is a merge request?

mbolivar · 2019-10-23T18:18:12Z

doc/guides/modules.rst

+worry about how this variable is defined at all. The build system knows how to
+use west to set :makevar:`ZEPHYR_MODULES`. You can add additional modules to
+this list by setting the :makevar:`ZEPHYR_EXTRA_MODULES` CMake variable or by
+setting the :makevar:`ZEPHYR_EXTRA_MODULES` environment varialbe in


mbolivar · 2019-10-23T18:19:50Z

doc/guides/modules.rst


 .. _CMake list: https://cmake.org/cmake/help/latest/manual/cmake-language.7.html#lists
 .. _add_subdirectory(): https://cmake.org/cmake/help/latest/command/add_subdirectory.html

 .. _GitHub issues: https://github.com/zephyrproject-rtos/zephyr/issues
+.. _zephyrproject-rtos: https://github.com/zephyrproject-rtos
+.. _west.yml: https://github.com/zephyrproject-rtos/zephyr/blob/master/west.yml


Why are you removing the :zephyr_file: uses?

That custom role has a hope of being extended to not go stale by emitting links for release documentation that don't point at master. This doesn't.

See the discussion here for details on my concern: #19123 (comment)

mbolivar · 2019-10-23T18:20:34Z

doc/guides/modules.rst

+local ones without having to modify the `manifest file <west.yml_>`_.
+
+It is also possible to avoid using west or modules at all, see the section below
+for more information.


Why not just

please see :ref:`no-west` for details.

?

mbolivar · 2019-10-23T18:23:12Z

doc/guides/modules.rst

@@ -253,9 +263,43 @@ Where 23 in the example above indicated the pull request number submitted to the
 *my_module* repository. Once the module changes are reviewed and merged, the
 revision needs to be changed to the commit hash from the module repository.

-
+Upmerging forks


"upmerging" is an NCS term I would like to avoid in official upstream docs.

"Synchronizing forks" perhaps? Here and elsewhere in this section.

agree, I do not really know what upmerging means in this context.

mbolivar · 2019-10-23T18:24:07Z

doc/guides/modules.rst

+projects is constrained by the following two factors:
+
+* History must **never** be rewritten in any repository that is referenced in
+  the `west.yml`_ manifest file, since that could break older revisions of the


Why? We could prevent the old commits being garbage collected using tags, as we do downstream.

dbkinder · 2019-10-23T23:17:05Z

doc/development_process/repo_structure.rst

+reasons <west-why-multi-repo>`, the project's code and supporting files needed
+to be split out into a set of repositories instead.
+
+To that end a new tool, :ref:`west`, was developed and introduced to ease the


Better would be simply, A new tool, ... (delete To that end)

dbkinder · 2019-10-23T23:19:45Z

doc/development_process/repo_structure.rst

+  * The `manifest repository`_ or also commonly named the "zephyr" repository
+    contains the `west.yml`_ :ref:`manifest file <west-manifests>` that
+    lists all other repositories that the manifest repository depends on.
+  * The project repositories are the ones containing :ref:`west projects


In @mbolivar suggested edit, use double back-ticks around projects: (not single back ticks)

dbkinder · 2019-10-23T23:24:33Z

doc/guides/modules.rst

+   currently upmerged to.
+#. Take the changes from the current to the latest upstream revision and apply
+   them to the current ``master`` branch of the module repository. There are
+   several ways of achieving this  It is important to note that the


missing a period at the end of the sentence this. It is

dbkinder · 2019-10-23T23:49:36Z

doc/guides/modules.rst

 #. Submit the module content (code) to the new repository following the
   guidelines described :ref:`here <modules_using_west>`.
-#. Add a new entry to the :zephyr_file:`west.yml` with the following
+#. Add a new entry to the `west.yml`_ manifest file with the following


I'd recommend leaving this as :zephyr_file:`west.yml` (as noted below, you shouldn't lock it to master)

dbkinder · 2019-10-23T23:50:26Z

doc/guides/modules.rst

@@ -231,7 +241,7 @@ Changes to existing modules
 #. Submit the changes using a pull request to an existing repository following
   the :ref:`contribution guidelines <contribute_guidelines>`.
 #. Submit a pull request changing the entry referencing the module into the
-   :zephyr_file:`west.yml` of the main Zephyr tree with the following
+   `west.yml`_ of the main Zephyr tree with the following


I'd recommend leaving this as :zephyr_file:`west.yml` (as noted below, you shouldn't lock it to master)

aescolar

One minor suggestion below.
I guess the only really important comment I'd have is that you state now very categorically all repos linked from the manifest must be hosted in the zephyr-rtos org. Anyhow, I guess that if at some point it is necessary to do otherwise, then the paragraph can be just changed.

aescolar · 2019-10-24T14:03:03Z

doc/development_process/repo_structure.rst

+  Which contains the latest state of development. In the case of :ref:`modules
+  <modules>` that fork an external project this may not reflect the latest state
+  of the upstream project's development, since projects are upmerged manually
+  when required by Zephyr itself.


Bikeshedding a bit: Maybe you want to comment that they may contain patches needed by Zephyr? (meaning they are not just behind, but are actually forked)

nashif · 2019-11-06T14:06:12Z

doc/guides/modules.rst

-not use modules at all if your application doesn't need them.
-
-
+Each module used by upstream Zephyr is contained in its own Git repository, and


s/upstream/mainline

nashif · 2019-11-06T14:06:38Z

doc/guides/modules.rst

+and :ref:`west-multi-repo` for more details on how west manages multiple
+repositories.
+
+In that way, if you use upstream Zephyr and therefore west, you don't need to


s/upstream/mainline

nashif · 2019-11-06T14:08:42Z

doc/guides/modules.rst

@@ -253,9 +263,43 @@ Where 23 in the example above indicated the pull request number submitted to the
 *my_module* repository. Once the module changes are reviewed and merged, the
 revision needs to be changed to the commit hash from the module repository.

-
+Upmerging forks


agree, I do not really know what upmerging means in this context.

nashif · 2019-11-06T14:10:05Z

doc/guides/modules.rst

+===============
+
+When a module is in fact a fork of an existing, external project that is hosted
+on GitHub or elsewhere, it is often necessary to upmerge it: update the copy


avoid upmerge, maybe synchronize?

carlescufi requested a review from dbkinder as a code owner October 23, 2019 14:47

carlescufi requested review from nashif, mbolivar, MaureenHelm, pabigot, galak and aescolar October 23, 2019 14:47

carlescufi commented Oct 23, 2019

View reviewed changes

zephyrbot added the area: Documentation label Oct 23, 2019

carlescufi force-pushed the org-repos branch from 738d687 to 0a13408 Compare October 23, 2019 15:17

carlescufi added 2 commits October 23, 2019 17:46

doc: Add a repository structure section

d1f61e3

Replace the former "code flow and branches" section with a new one that covers the whole repository structure and the branches that each repository may have. Signed-off-by: Carles Cufi <[email protected]>

doc: modules: Rework and extend the documentation

fd7064a

Rework some of the existing sections for clarity and extend the guide to cover upmerges. Signed-off-by: Carles Cufi <[email protected]>

carlescufi force-pushed the org-repos branch from 0a13408 to fd7064a Compare October 23, 2019 15:46

mbolivar suggested changes Oct 23, 2019

View reviewed changes

dbkinder suggested changes Oct 23, 2019

View reviewed changes

aescolar reviewed Oct 24, 2019

View reviewed changes

nashif requested changes Nov 6, 2019

View reviewed changes

nashif added the Stale label Sep 4, 2020

github-actions bot closed this Sep 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

doc: Repository structure and modules #20067

doc: Repository structure and modules #20067

carlescufi commented Oct 23, 2019

carlescufi Oct 23, 2019

carlescufi Oct 23, 2019

pabigot Oct 24, 2019

mbolivar Oct 24, 2019

zephyrbot commented Oct 23, 2019 •

edited

Loading

mbolivar Oct 23, 2019

dbkinder Oct 23, 2019 •

edited

Loading

mbolivar Oct 23, 2019

mbolivar Oct 23, 2019

mbolivar Oct 23, 2019

mbolivar Oct 23, 2019

mbolivar Oct 23, 2019

mbolivar Oct 23, 2019

mbolivar Oct 23, 2019

nashif Nov 6, 2019

mbolivar Oct 23, 2019

dbkinder Oct 23, 2019

dbkinder Oct 23, 2019 •

edited

Loading

dbkinder Oct 23, 2019

dbkinder Oct 23, 2019

dbkinder Oct 23, 2019

aescolar left a comment

aescolar Oct 24, 2019

nashif Nov 6, 2019

nashif Nov 6, 2019

nashif Nov 6, 2019

nashif Nov 6, 2019

		not use modules at all if your application doesn't need them.


		Each module used by upstream Zephyr is contained in its own Git repository, and

doc: Repository structure and modules #20067

doc: Repository structure and modules #20067

Conversation

carlescufi commented Oct 23, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zephyrbot commented Oct 23, 2019 • edited Loading

Choose a reason for hiding this comment

dbkinder Oct 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbkinder Oct 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aescolar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zephyrbot commented Oct 23, 2019 •

edited

Loading

dbkinder Oct 23, 2019 •

edited

Loading

dbkinder Oct 23, 2019 •

edited

Loading