Merge publishing assistant PRs into master

doc/publishing.md

@@ -1,10 +1,9 @@
 # Publishing your projects in Alire
 
-The process for submitting a new crate or release is currently manual, although
-automated support through the `alr publish` command is in the roadmap. The
-sequence of steps to take is described after some introductory concepts; you
-can also ask for help in the [gitter channel](https://gitter.im/ada-lang/Alire)
-of the project.
+Publishing a project in Alire is done with the help of the `alr publish`
+command. The steps to take are described after some introductory concepts (jump
+to these steps directly [here](#detailed-steps); you can also ask for help on
+the [gitter channel](https://gitter.im/ada-lang/Alire) of the project.
 
 ## General concepts
 
@@ -13,7 +12,7 @@ The community index is a collection of
 [alire-index](https://github.com/alire-project/alire-index) repository, under
 the [index](https://github.com/alire-project/alire-index/blob/master/index)
 directory. Each file contains a release for a crate and is named after the
-crate and version it contains. A crate contains the description of a release,
+crate and version it contains. A file contains the description of a release,
 with other metadata.
 
 The complete specification of such TOML files is available in this
@@ -33,12 +32,15 @@ The community index is supported through two kinds of branches:
   format, during the development of `alr`.
 
 Your `alr` version knows which branch to use, so you do not need to manually
-select one. However, when submitting crates/releases as pull requests, you must
-decide to which branch they will be added: selecting the latest stable branch
-results in the release becoming immediately available to the latest stable
-`alr`. Conversely, using the latest development branch will make the releases
-available for testing by unstable clients, and will become generally available
-with the next stable release of `alr`.
+select one. When using `alr publish` to assist on creating a release, you will
+be provided with an upload link for the branch your `alr` is using.
+
+However, when submitting releases manually, you must decide to which branch
+they will be added: selecting the latest stable branch results in the release
+becoming immediately available to the latest stable `alr`. Conversely, using
+the latest development branch will make the releases available for testing by
+unstable clients, and will become generally available with the next stable
+release of `alr`.
 
 Note that, as of this writing, only development branches exist, until the first
 stable release of `alr` is made.
@@ -49,50 +51,153 @@ Each crate is "owned" by a list of maintainers, provided with the
 `maintainers-logins` property of the crate file. After the initial submission,
 which will be manually approved (see the [policies](policies.md) for details),
 the maintainers of a crate are the only people allowed to submit new releases
-or metadata modifications to the corresponding crate file.
+or metadata modifications to the corresponding crate.
+
+Other checks your submission will go through are:
+
+- It contains all required metadata.
+- It builds on all of our CI configurations.
+    - You can disable an unsupported target with the `available` property.
 
 ## Detailed steps
 
-These steps describe how to submit a new release. 
-
-1. Prepare a source archive (zip, tar) with the release sources and project
-   files.
-    1. This archive **must not** contain the `alire` directory generated by
-       `alr` in working directories. The `alire` directory is designed to be
-       put in your VCS ignore list.
-    1. The archive **must** contain a single directory (name not important)
-       containing, in turn, the sources. This is the kind of archives
-       automatically generated by GitHub, GitLab, Sourceforge... or through
-       `git archive`.
-1. Compute an integrity hash for the archive with `alr publish --hash <URL>`
-    1. The URL can be remote or local (`file://`), although using the final
-       online file is recommended.
-    1. The final online file should be immutable. Note that online
-       autogenerated archives can be found currently in the index, but should
-       be avoided whenever possible (because of lack of immutability
-       guarantees, see [this
-       conversation](https://github.community/t5/How-to-use-Git-and-GitHub/Ensuring-identical-tarballs-from-releases/m-p/34196/highlight/true#M8997)).
-       In the future, `alr publish` will offer support to generate and upload
-       such files more conveniently.
-1. Fill in the release manifest providing all the necessary information, 
-   dependencies (`depends-on`) and possibly excluded platforms
-   (`available`). See, for example, the [hello
-   world](https://github.com/alire-project/alire-index/blob/master/index/he/hello/hello-1.0.1.toml)
-   crate, or the [empty
-   skeleton](https://github.com/alire-project/alire-index/blob/master/templates/skeleton.toml)
-   template.
-1. Place the manifest file in the appropriate directory of your locally checked
-   out index repository, and create a new pull request. The detailed substeps
-   are the usual for submitting contributions in GitHub:
-    1. Fork the community index to your GitHub account.
-    1. Clone your fork locally and make the changes locally.
-    1. Commit and push the changes to your fork.
-    1. Create the pull request from your fork against the community repository
-       through the GitHub web interface (or the
-       [`hub`](https://github.com/github/hub) tool).
-        1. The base branch you select for the pull request will determine where
-           your changes will become available; see the section on [index
-           branches](#index-branches) for details.
+Depending on how you develop your project, you can use one of the following
+methods to prepare your release submission:
+
+### Starting from a git repository that contains an Alire workspace
+
+For this common use case, you need:
+
+- A git repository that is clean an up-to-date with its remote.
+   - The repository already contains the release you want to publish.
+   - The commit with the release must exist both locally and at the remote.
+- The repository must also be an Alire-enabled workspace:
+   - It contains a top-level `alire.toml` manifest describing the release.
+- The remote host must be one of a few trusted major open-source sites.
+   - This requirement is motivated by vulnerabilities identified with SHA1,
+     whose migration to a stronger hash is [not yet complete]
+     (https://git-scm.com/docs/hash-function-transition/) in `git`.
+   - `alr` will inform you if your host is not supported. Please contact us if
+     you think a site should be allowed. The complete list can be consulted by
+running `alr publish --trusted-sites`.
+   - This is a temporary measure until more sophisticated publishing automation
+     is supported. See the [Remote Source Archive](#remote-source-archive) case
+for alternatives to this scenario (you are not forced to change your code
+hosting, or even have an online repository).
+
+By default, the last commit is used for the release. You can alternatively
+provide another commit, tag, or branch. In any case, the git revision will be
+used to obtain a final commit. That is, a release cannot evolve with a branch,
+or be updated by moving a tag.
+
+- Within the repository, issue
+
+`alr publish`
+
+to use the last commit. You can, alternatively, issue:
+
+`alr publish . <commit|tag|branch>`
+
+Note the path between `publish` and your non-commit revision. Likewise, you can
+run this command from outside your repository, as long as you supply the proper
+path to it.
+
+At this point, `alr publish` will carry out a few tests and, if everything
+checks out, it will create a `${repo_root}/alire/releases/crate-version.toml`
+file. This file must be submitted to the community index via a PR. A link for
+conveniently creating this PR will also be provided by `alr`:
+
+- Upload the generated index manifest file (`crate-version.toml`) to the
+  supplied page link on github and create a pull-request.
+
+### Starting with a remote repository, without local clone
+
+This case is analogous to the previous one, but you don't need the local
+repository. The same considerations about allowed hosts discussed in the
+previous scenario apply:
+
+- The repository already contains the commit with release you want to publish.
+- The repository must also be an Alire-enabled workspace:
+   - It contains a top-level `alire.toml` manifest describing the release.
+- The remote host must be one of a few trusted major open-source sites.
+   - This requirement is motivated by vulnerabilities identified with SHA1,
+     whose migration to a stronger hash is [not yet complete]
+     (https://git-scm.com/docs/hash-function-transition/) in `git`.
+   - `alr` will inform you if your host is not supported. Please contact us if
+     you think a site should be allowed. The complete list can be consulted by
+running `alr publish --trusted-sites`.
+
+The only difference when invoking `alr` is that you must supply the remote URL
+and commit (not a tag or branch). The commit must exist in the repository:
+
+`alr publish <URL> <commit>`
+
+The checks will be carried out and the outcome will be the same as in the
+previous scenario.
+
+### Starting with a remote source archive
+
+This case can be used when you use another VCS other than `git`, or do not work
+with an online repository.
+
+In this use case, you start from an already prepared final remote tarball/zipball:
+
+- The archive **must** contain a single directory (name not important)
+   containing, in turn, the sources. This is the kind of archives
+   automatically generated by GitHub, GitLab, Sourceforge... or through
+   `git archive`.
+- The `alire.toml` manifest must be placed at the top-level with the rest of
+   your sources (inside the same single directory just described), containing
+   all required information except for the `[origin]` table, which will be created
+   by `alr`.
+- This archive **must not** contain the `alire` directory generated by
+   `alr` in working directories. The `alire` directory is designed to be
+    put in your VCS ignore list.
+
+With the source archive already uploaded to the online host where it is going
+to be served (there are no restrictions on this host), you should issue
+
+`alr publish <URL>`
+
+and the publishing process will carry on as in the previous cases, performing
+the checks and providing you with a file to submit to the index, and an upload
+URL to do so.
+
+### Starting with a local source folder
+
+Assisting with the creation and upload of a source archive from local sources, as
+a pre-step to the case just presented, is in the roadmap.
+
+### Support for complex projects whose sources become multiple Alire crates
+
+In case your project does not easily map to a single Alire crate (e.g., because
+you manage multiple project files with different dependencies, or there are
+other reasons to keep the sources together even if they generate several
+crates), you will need to prepare individual online source archives (or
+repositories) and proceed from there.
+
+### Starting from other configurations
+
+If your case does not fit well into any of the provided guidance we will be
+sure interested in hearing about it, and seeing how it can be brought into
+existing or new Alire workflows.
+
+### Creating the PR via cloning.
+
+Instead of uploading the generated index manifest file via the github upload
+link, you can follow the usual procedure to submit a PR to a github repository:
+
+1. Fork the community index to your GitHub account.
+1. Clone your fork locally and place generated manifest at the intended folder.
+1. Commit and push the changes to your fork.
+1. Create the pull request from your fork against the community repository
+   through the GitHub web interface (or the
+   [`hub`](https://github.com/github/hub) tool).
+    1. The base branch you select for the pull request will determine where
+       your changes will become available; see the section on [index
+       branches](#index-branches) for details.
+
+## Publishing outcome
 
 Once the pull request is verified and merged, the new release will become
 available for normal use. The open source Ada ecosystem needs all the help it

doc/user-changes.md

@@ -4,6 +4,35 @@ This document is a development diary summarizing changes in `alr` that notably
 affect the user experience. It is intended as a one-stop point for users to
 stay on top of `alr` new features.
 
+### First publishing assistant
+
+PR [#527](https://github.com/alire-project/alire/pull/527).
+
+A new publishing assistant can be invoked with `alr publish [URL [commit]]`. At
+this time, the assistant requires that all necessary metadata for a release,
+excepting the `[origin]` table, is informed in the `alire.toml` file.
+
+The assistant has local and remote modes of operation. In the local mode, the
+user invokes the assistant from within a repository on its computer that is
+up-to-date with its remote, and that contains an Alire workspace. In this case,
+it is enough to run `alr publish`.
+
+In the remote mode, the user must prepare a source file or repository in their
+final online locations, and use `alr publish <URL> [<commit>]`, with the commit
+being mandatory for repositories and not needed for source archives.
+
+In all cases, `alr` will fetch the sources, perform a few checks on the
+completeness of the information, and generate a final metadata file, intended
+to be submitted to the community index via pull request. An upload link is
+provided for convenience that can be used to create this pull request.
+
+Complete information about this feature is available in the updated
+[Publishing](publishing.md) page.
+
+Other features of the assistant are that, in the local mode, a branch or tag
+can be specified to pinpoint a commit, and that the test build of the crate can
+be skipped with `--skip-build`.
+
 ### Move manifest and lock files to top-level folder
 
 PR [#501](https://github.com/alire-project/alire/pull/501).

src/alire/alire-directories.adb

@@ -353,6 +353,24 @@ package body Alire.Directories is
             Delete_Tree (This.Filename);
          end if;
       end if;
+   exception
+      when E : Use_Error =>
+         --  For unclear reasons, Windows has trouble removing a folder that
+         --  contains a git repository. Warn and continue until we know more.
+
+         --  The actual error, as reported by Python3:
+         --  PermissionError: [WinError 5] Access is denied:
+         --  'alr-fwrt.tmp\\.git\\objects\\1d\\'
+         --  '696bed4ef917b1adbdef18723016987ed62e41'
+
+         --  Since we are running tests as root, this might be that the file is
+         --  still open, but by whom?
+
+         Log_Exception (E);
+         Trace.Warning ("Could not delete temporary: " & This.Filename);
+      when E : others =>
+         Log_Exception (E);
+         raise;
    end Finalize;
 
    -------------------

src/alire/alire-index.ads

@@ -44,6 +44,11 @@ package Alire.Index is
    --  The index version understood by alire must match the one in the indexes
    --  being loaded.
 
+   Community_Upload_URL : constant URL :=
+                            Utils.Tail (Community_Repo, '+')
+                            & "/upload/" & Community_Branch;
+   --  Base upload URL into our community index repository
+
    subtype Release is Alire.Releases.Release;
 
    ------------------------

src/alire/alire-origins-deployers-source_archive.adb

@@ -5,6 +5,7 @@ with Alire.Directories;
 with Alire.OS_Lib.Subprocess;
 with Alire.OS_Lib.Download;
 with Alire.VFS;
+with Alire.URI;
 with Alire.Utils;             use Alire.Utils;
 with Alire.Utils.Tools;
 
@@ -116,7 +117,8 @@ package body Alire.Origins.Deployers.Source_Archive is
    function Compute_Hash (This   : Deployer;
                           Folder : String;
                           Kind   : Hashes.Kinds) return Hashes.Any_Digest is
-      Archive_Name : constant String := This.Base.Archive_Name;
+      Archive_Name : constant String := Dirs.Simple_Name
+        (This.Base.Archive_Name);
       Archive_File : constant String := Dirs.Compose (Folder, Archive_Name);
    begin
       return Hashes.Digest (Hashes.Hash_File (Kind, Archive_File));
@@ -129,9 +131,26 @@ package body Alire.Origins.Deployers.Source_Archive is
    overriding
    function Fetch (This : Deployer; Folder : String) return Outcome is
    begin
-      return OS_Lib.Download.File (URL      => This.Base.Archive_URL,
-                                   Filename => This.Base.Archive_Name,
-                                   Folder   => Folder);
+
+      --  If the file is local, do a simple copy. While curl seems to work in
+      --  linux also for local files, something funny is going on Windows which
+      --  is difficult to pinpoint.
+
+      if URI.Scheme (This.Base.Archive_URL) in URI.File_Schemes then
+         if not Dirs.Exists (Folder) then
+            Dirs.Create_Directory (Folder);
+         end if;
+
+         Ada.Directories.Copy_File
+           (URI.Local_Path (This.Base.Archive_URL),
+            Dirs.Compose (Folder, Dirs.Simple_Name (This.Base.Archive_Name)));
+
+         return Outcome_Success;
+      else
+         return OS_Lib.Download.File (URL      => This.Base.Archive_URL,
+                                      Filename => This.Base.Archive_Name,
+                                      Folder   => Folder);
+      end if;
    end Fetch;
 
    ------------