alr install redux: Installation of binary crates

.gitattributes

@@ -10,3 +10,4 @@
 # Mark some misidentified files as always binaries
 *.pdf -text
 *.png -text
+*.tgz -text

doc/AEPs/aep-0003.md

@@ -0,0 +1,176 @@
+    AEP: 3
+    Title: Command `alr install` information
+    Author: Alejandro R. Mosteo <[email protected]>
+    Status: Draft
+    Created: 2022-01-19
+
+Abstract
+========
+
+Information gathered about the operation of `gprinstall`, relevant to our
+purposes of how to work with static/dynamic libraries and with different
+versions of the same library/compilers.
+
+Information
+===========
+
+By using different names during installation, several versions of the same lib
+can be made to coexist, with some caveats.
+
+For example, after having installed libhello 1.0.0 (static) and libhello 1.0.1
+(dynamic) and hello 1.0.2 (dynamic), we obtain this tree:
+
+```
+prefix/
+├── bin
+│   └── hello
+├── include
+│   ├── hello=1.0.2
+│   │   └── hello
+│   │       ├── hello.adb
+│   │       └── hello_config.ads
+│   ├── libhello=1.0.0
+│   │   └── libhello
+│   │       ├── libhello.adb
+│   │       └── libhello.ads
+│   └── libhello=1.0.1
+│       └── libhello
+│           ├── libhello.adb
+│           ├── libhello.ads
+│           └── libhello_config.ads
+├── lib
+│   ├── libhello=1.0.0
+│   │   └── libhello
+│   │       ├── libhello.a
+│   │       └── libhello.ali
+│   ├── libhello=1.0.1
+│   │   └── libhello
+│   │       ├── libhello.ali
+│   │       ├── libhello_config.ali
+│   │       ├── Libhello.so.1.0.1
+│   │       └── libLibhello.so -> ../libhello/Libhello.so.1.0.1
+│   ├── Libhello.so.1.0.1 -> ../lib/libhello=1.0.1/libhello/Libhello.so.1.0.1
+│   └── libLibhello.so -> ../lib/libhello=1.0.1/libhello/libLibhello.so
+└── share
+    └── gpr
+        ├── hello.gpr
+        ├── libhello.gpr
+        └── manifests
+            ├── hello=1.0.2
+            ├── libhello=1.0.0
+            └── libhello=1.0.1
+```
+
+First caveat is that `gprinstall` clobbers `prefix/share/gpr/libhello.gpr`
+without warning, even if `-f` was not used. This means that an installation
+with the purpose of development cannot have several versions installed.
+
+Still, it seems we could rely on such a prefix for executables depending on
+different versions of the same dynamic library, as ldd shows the proper
+dependency:
+
+```
+$ ldd prefix/bin/hello
+	linux-vdso.so.1 (0x00007fffa772f000)
+	Libhello.so.1.0.1 => not found
+	libgnat-12.so => /path/to/compiler/.../libgnat-12.so (0x00007fda1ca0e000)
+	libgcc_s.so.1 => /path/to/compiler/.../libgcc_s.so.1 (0x00007fda1c7ef000)
+	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fda1c5c5000)
+	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fda1c5c0000)
+	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fda1c4d9000)
+	/lib64/ld-linux-x86-64.so.2 (0x00007fda1d0b4000)
+```
+
+By using `--mode=usage`, there is no ali, gpr or source files installed. So
+going with this mode, unneeded conflicting files are not even installed.
+
+Uninstalling two versions will remove the gpr file after the first uninstall.
+However, the second uninstall will not fail, silently removing the rest of
+files.
+
+For uninstallation we need to supply the project file, which can come from the
+original build folder or from the installed share/gpr location.
+
+However, when installing in usage mode, there will be no gpr file installed,
+forcing to preserve the original project file.
+
+In usage mode, static libraries are not installed, and no manifest is created
+if nothing gets installed. Uninstalling will then complain about lack of manifest.
+
+It seems that, at least for basic code, an executable build with a compiler and
+a dynamic library build with another are compatible.
+
+Executables depend on properly versioned `.so.x.x.x` files, so the extra `.so`
+file clobbered by several installs is not important.
+
+Summary of findings
+===================
+
+- Several dynamic versions of a library are possible, for executables.
+  - Development with several versions is not possible.
+- Dependencies on libraries from the compiler also appear.
+  - It would then be better to use a compiler from within the prefix.
+- Uninstall of several versions requires preserving the original project file.
+  - When uninstalling only one version, the installed gpr file suffices.
+- Compiler consistency between dynamic libraries is not mandatory.
+- Executables link against the properly versioned dynamic library.
+
+Proposal
+========
+
+Given these findings, and the primary need of using `alr install` to make binaries
+available, and not to make development libraries available, we can abandon this
+latter notion for good. Sharing of large dependencies, if ever implemented will use
+a different mechanism.
+
+For installing executables, each installation can be performed on its own: we need
+not track crates (as we can check the gprinstall manifests) nor consider
+incompatibilities, as there are none. Worst case, two releases from the same crate
+would be detected by their manifests, and two different crates clobbering each other
+would be detected during `gprinstall`.
+
+For uninstallation, we may redeploy sources to have access to the original project
+files.
+
+For (un)installation of local crates, there's also no issue: they will use the local
+version, and we have the gpr file available too for both.
+
+We could consider caching build directories for faster installs of related crates
+sharing several dependencies, and faster uninstallation. This would incur some disk
+usage penalty so we may want to make this optional.
+
+We may want to track dependencies to prevent uninstallation of libraries which are
+depended upon. This would be a final enhancement. Uninstallation doesn't seem to be
+a pressing matter, as prefixes will be fast to recreate.
+
+Tracking of dependencies isn't trivial as we can't use a single root (there will be
+"incompatible" (in solution sense) crates installed quite often, if only because of
+forced compilers).
+
+Roadmap
+=======
+
+Wanted and simple to implement:
+
+- Installation of binary releases
+   - Un-uninstallable if there is no project file
+- Installation of local releases
+- Installation of indexed releases
+
+Optional or low priority:
+
+- Uninstallation relying on project file (local/indexed releases)
+- Uninstallation of binary releases without a project file
+   - Using our own-created manifest during installation
+- Tracking of dependencies
+   - To prevent uninstalls with dependents
+   - Or to also uninstall dependents
+
+Out of scope:
+
+- Installation for development (users can manually do this via alr exec -- gprinstall)
+
+Copyright
+=========
+
+This document has been placed in the public domain.

doc/user-changes.md

@@ -6,6 +6,51 @@ stay on top of `alr` new features.
 
 ## Release 1.3-dev
 
+### New subcommand `alr install
+
+PR [#1302](https://github.com/alire-project/alire/pull/1302)
+
+A new subcommand `alr install` allows the installation of binaries to a location
+intended to be added to the user's PATH. The default install location is
+`$HOME/.alire`, with binaries going into `$HOME/.alire/bin`.
+
+This is a experimental feature that will see improvements and tweaks in further
+PRs and as we gather feedback on its usage.
+
+At present, only binary releases can be installed (e.g., compilers, `gprbuild`,
+`gnatprove`, `gnatstudio`). There is no ability to uninstall releases either
+(but reinstallation can be forced).
+
+Only one version per executable can be installed, meaning that, for example,
+only one toolchain version can exist in an installation prefix. So, this
+feature is intended to make the user's preferred version of a crate generally
+available in the system for use outside of Alire, but not to replace e.g. the
+ability of Alire to use several compilers, or to reuse compiled libraries as
+dependencies in several workspaces.
+
+Examples:
+
+```
+$ alr install gnatprove
+ⓘ Installing gnatprove=12.1.1...
+ⓘ Installation complete.
+
+$ alr install
+Installation prefix found at /home/jano/.alire
+Contents:
+   gnatprove=12.1.1
+
+$ PATH+=:$HOME/.alire/bin gnatprove --version
+Usage: gnatprove -Pproj [switches] [-cargs switches]
+...
+
+$ alr install gnatprove^11
+error: Requested release gnatprove=11.2.3 has another version already
+installed: gnatprove=12.1.1. (This error can be overridden with --force.)
+
+$ alr --force install gnatprove^11  # Downgrade installation
+```
+
 ### Find dependents of a release with `alr show --dependents
 
 PR [#1170](https://github.com/alire-project/alire/pull/1170)

scripts/alr-completion.bash

@@ -46,7 +46,7 @@ function _alr_completion() {
     # Command-specific completions
     $found &&\
     case $cmd in
-        get | show | toolchain)
+        get | install | show | toolchain)
             # Suggest crate names
             COMPREPLY+=($(compgen -W "$_alr_crates" -- $curr))
             ;;

src/alire/alire-directories.adb

@@ -8,6 +8,7 @@ with Alire.Errors;
 with Alire.OS_Lib.Subprocess;
 with Alire.Paths;
 with Alire.Platforms.Current;
+with Alire.VFS;
 
 with GNATCOLL.VFS;
 
@@ -419,6 +420,13 @@ package body Alire.Directories is
               Exception_Information (E));
    end Finalize;
 
+   ------------------
+   -- Is_Directory --
+   ------------------
+
+   function Is_Directory (Path : Any_Path) return Boolean
+   is (Adirs.Exists (Path) and then Adirs.Kind (Path) in Adirs.Directory);
+
    ----------------
    -- TEMP FILES --
    ----------------
@@ -542,11 +550,117 @@ package body Alire.Directories is
          raise;
    end Finalize;
 
+   --------------------
+   -- Merge_Contents --
+   --------------------
+
+   procedure Merge_Contents (Src, Dst              : Any_Path;
+                             Skip_Top_Level_Files  : Boolean;
+                             Fail_On_Existing_File : Boolean)
+   is
+
+      Base   : constant Absolute_Path := Adirs.Full_Name (Src);
+      Target : constant Absolute_Path := Adirs.Full_Name (Dst);
+
+      -----------
+      -- Merge --
+      -----------
+
+      procedure Merge
+        (Item : Ada.Directories.Directory_Entry_Type;
+         Stop : in out Boolean)
+      is
+         use all type Adirs.File_Kind;
+
+         Rel_Path : constant Relative_Path :=
+                      Find_Relative_Path (Base, Adirs.Full_Name (Item));
+         --  If this proves to be too slow, we should do our own recursion,
+         --  building the relative path along the way, as this is recomputing
+         --  it for every file needlessly.
+
+         Dst : constant Absolute_Path := Target / Rel_Path;
+         Src : constant Absolute_Path := Adirs.Full_Name (Item);
+      begin
+         Stop := False;
+
+         --  Check if we must skip (we delete source file)
+
+         if Adirs.Kind (Item) = Ordinary_File
+           and then Skip_Top_Level_Files
+           and then Base = Parent (Src)
+         then
+            Trace.Debug ("   Merge: Not merging top-level file " & Src);
+            Adirs.Delete_File (Src);
+            return;
+         end if;
+
+         --  Create a new dir if necessary
+
+         if Adirs.Kind (Item) = Directory then
+            if not Is_Directory (Dst) then
+               Trace.Debug ("   Merge: Creating destination dir " & Dst);
+               Adirs.Create_Directory (Dst);
+            end if;
+
+            return;
+            --  Nothing else to do for a directory. If we controlled the
+            --  recursion we could more efficiently rename now into place.
+         end if;
+
+         --  Move a file into place
+
+         Trace.Debug ("   Merge: Moving " & Adirs.Full_Name (Item)
+                    & " into " & Dst);
+         if Adirs.Exists (Dst) then
+            if Fail_On_Existing_File then
+               Recoverable_Error ("Cannot move " & TTY.URL (Src)
+                                  & " into place, file already exists: "
+                                  & TTY.URL (Dst));
+            elsif Adirs.Kind (Dst) /= Ordinary_File then
+               Raise_Checked_Error ("Cannot replace " & TTY.URL (Dst)
+                                    & " as it is not a regular file");
+            else
+               Trace.Debug ("   Merge: Deleting in preparation to replace: "
+                            & Dst);
+               Adirs.Delete_File (Dst);
+            end if;
+         end if;
+
+         --  We use GNATCOLL.VFS here as some binary packages contain softlinks
+         --  to .so libs that we must copy too, and these are troublesome
+         --  with regular Ada.Directories (that has no concept of softlink).
+         --  Also, some of these softlinks are broken and although they are
+         --  presumably safe to discard, let's just go for an identical copy.
+
+         declare
+            VF : constant VFS.Virtual_File :=
+                   VFS.New_Virtual_File (VFS.From_FS (Src));
+            OK : Boolean := False;
+         begin
+            if VF.Is_Symbolic_Link then
+               VF.Rename (VFS.New_Virtual_File (Dst), OK);
+               if not OK then
+                  Raise_Checked_Error ("Failed to move softlink: "
+                                       & TTY.URL (Src));
+               end if;
+            else
+               Adirs.Rename (Old_Name => Src,
+                             New_Name => Dst);
+            end if;
+         end;
+      end Merge;
+
+   begin
+      Traverse_Tree (Start   => Src,
+                     Doing   => Merge'Access,
+                     Recurse => True);
+   end Merge_Contents;
+
    -------------------
    -- Traverse_Tree --
    -------------------
 
-   procedure Traverse_Tree (Start   : Relative_Path;
+   procedure Traverse_Tree (Start   : Any_Path;
                             Doing   : access procedure
                               (Item : Ada.Directories.Directory_Entry_Type;
                                Stop : in out Boolean);