Pins in the manifest #743

mosteo · 2021-05-28T08:38:47Z

Pins are now to be defined in the manifest:

[[pins]]
foo = { version = "1.3.2+bugfix" }
bar = { path = "../my/bar" }
baz = { url = "https://github.com/baz.git" } # No commit, will use current HEAD, will update on `alr update`
gru = { url = "https://gitlab.com/gru.git" commit="123456890abcdef..." } # Explicit commit, won't update

This PR is still incomplete, but is testable: the new pins work, and the old pins also work, but such old pins will be forgotten after any update (explicitly requested, or by editing the manifest), as the update process syncs with the pins in the manifest.

To do:

Remove old pins
Tests specific to the new format

Fixes #721

Fabien-Chouteau · 2021-05-28T08:42:52Z

[[pins]]
foo = { version = "1.3.2+bugfix" }
bar = { path = "../my/bar" }
baz = { url = "https://github.com/baz.git" } # No commit, will use current HEAD, will update on alr update
gru = { url = "https://gitlab.com/gru.git" commit="123456890abcdef..." } # Explicit commit, won't update

This looks great!

src/alire/alire-solutions-diffs.adb

onox · 2021-05-28T20:49:13Z

config/alr_config.ads

@@ -0,0 +1,4 @@
+--  Configuration for alr generated by Alire


Should this folder not be added to git?

Fixed by #744

Fixed in Alire, but they still should not be checked in here.

onox · 2021-05-28T21:15:05Z

I did some testing with orka_simd, added the following to its alire.toml file:

[[pins]]
orka_types = {path = "../orka_types"}

and removing alire.lock and then running:

alr build -XORKA_SIMD_EXT="AVX2" (says solution is incomplete)
alr update (asks me to confirm new solution)

It has found the folder of orka_types 👍 (and gives me the original alire.lock again)

but if I remove alire.lock and then immediately run alr update, I get:

stderr: SYSTEM.ASSERTIONS.ASSERT_FAILURE
stderr: failed precondition from alire-solutions.ads:124
stderr: raised SYSTEM.ASSERTIONS.ASSERT_FAILURE : failed precondition from alire-solutions.ads:124

mosteo · 2021-05-29T09:22:52Z

Thanks, @onox. The PR is still undercooked so I'll keep an eye out for these issues.

Minor testsuite tweak for a change in logging format

We had two confusing Update_Dependencies and Update_And_Deploy_Dependencies that were in practice doing almost the same. There is now a single Sync_Dependencies.

For now, these cannot work as we are going to remove the ability to edit pins via `alr with`/`alr pin`. This functionality could be reintroduced at a later time.

Most of those should be reimplemented in their manual edition alternative

This is purely for user comfort and will probably result in dependencies having to be added at publish time. However, if we manage to restore command-line pinning, we can remove that pain by adding missing dependencies at that time.

Some tests are not easily portable without support from `alr with --use`. Since that should be easy to implement later, they are disabled for now and will be enabled in a subsequent patch.

alire.lock

doc/catalog-format-spec.md

src/alire/alire-roots.adb

Also the required code changes to pass the test

mosteo · 2021-06-16T16:58:42Z

This one is ready for review. There are quite some disabled tests and some portions of unreachable code. I intend to tidy that up in a follow-up patch (since this one has grown already too large), in which at least alr with --use works again, since that would allows us to keep lots of tests as-is that otherwise are hard to adapt. As this will require only to append to the manifest, as is already done for with, it should not be too complicated (fingers crossed).

Other than that, the new pins should be ready. @onox , as you have an use case, your feedback on this one would be great.

Fabien-Chouteau · 2021-06-17T09:04:48Z

config/alr_config.ads

@@ -0,0 +1,4 @@
+--  Configuration for alr generated by Alire


Fixed in Alire, but they still should not be checked in here.

doc/catalog-format-spec.md

src/alire/alire-dependencies-states.ads

raph-amiard · 2021-06-17T10:24:05Z

Hello @mosteo, as I said live to Fabien, I think being able to specify a branch name in updatable pins would be a useful, or even necessary functionality in complex use cases.

Thanks for tackling this! I think it will be the tipping point for a lot of us at AdaCore to start using Alire.

Fabien-Chouteau · 2021-06-17T10:45:15Z

@mosteo I tried to create a test crate for one of my project a found an issue. Here's how to reproduce:

$ git clone --branch new_alr_pins https://github.com/Fabien-Chouteau/bbqueue-spark
$ cd cd bbqueue-spark/tests
$ alr run main_framed
[works fine]
$ touch alire.toml # to simulate modification of the manifest
$ alr run main_framed
# Detected changes in manifest, synchronizing workspace...
Dependencies automatically updated as follows:                                                

   New solution is incomplete.
   #  atomic  ~0.3  (missing)                                    
   ## bbqueue 0.2.0 (pin=../../bbqueue-spark,upgraded from 0.1.0)
warn: Generating possibly incomplete environment because of missing dependencies
warn: Generating possibly incomplete configuration because of missing dependencies

bbqueue.gpr:3:06: unknown project file: "atomic.gpr"
bbqueue.gpr:3:06: imported by "/tmp/bbqueue-spark/bbqueue.gpr"
bbqueue.gpr:3:06: imported by "/tmp/bbqueue-spark/tests/config/tests_config.gpr"
bbqueue.gpr:3:06: imported by "/tmp/bbqueue-spark/tests/tests.gpr"
gprbuild: "tests.gpr" processing failed
error: Command ["gprbuild", "-gnatwU", "-j0", "-p", "-P", "tests.gpr"] exited with code 4
error: Build failed

mosteo · 2021-06-17T13:34:05Z

Hello @mosteo, as I said live to Fabien, I think being able to specify a branch name in updatable pins would be a useful, or even necessary functionality in complex use cases.

Right, this is at my top of follow-up improvements. I'll tackle it in a follow-up PR.

mosteo · 2021-06-17T14:55:23Z

@mosteo I tried to create a test crate for one of my project a found an issue. Here's how to reproduce:

I cannot reproduce it (it runs properly both times) but there's definitely something wrong because the first time around the pin is not detected. Will fix that and we'll see if it's related.

We were creating an empty lockfile, which was newer than the manifest, and thus not triggering the expected automatic update.

This was a bug detected and corrected in the previous commit

mosteo · 2021-06-17T15:47:28Z

Ready for another round, @Fabien-Chouteau . Let me know if this improved things for your test.

onox · 2021-06-17T19:07:17Z

@mosteo Pins in wayland-ada repo seem to work, but Alire could try to simplify the relative paths of indirect dependencies, it currently just concatenates the paths. For example, in wayland_ada_info crate I get changes like this:

-path = "file:../wayland_ada_scanner"
+path = "file:../wayland_protocols_ada/../wayland_client_ada/../wayland_ada_scanner"

Not a big problem though.

onox · 2021-06-17T20:05:05Z

I can run make just fine in wayland-ada and orka repositories, but in awt I still get some errors and exceptions.

I get a few errors when running alr build in the awt repository:

alr build -XORKA_SIMD_EXT="AVX2"
orka.gpr:1:06: unknown project file: "orka_types"
orka.gpr:1:06: imported by "/path/to/repo/orka/orka/orka.gpr"
orka.gpr:1:06: imported by "/path/to/repo/orka/orka/orka-egl.gpr"
orka.gpr:1:06: imported by "/path/to/repo/awt/awt-linux.gpr"
orka.gpr:2:06: unknown project file: "orka_simd"
orka.gpr:2:06: imported by "/path/to/repo/orka/orka/orka.gpr"
orka.gpr:2:06: imported by "/path/to/repo/orka/orka/orka-egl.gpr"
orka.gpr:2:06: imported by "/path/to/repo/awt/awt-linux.gpr"
orka.gpr:3:06: unknown project file: "orka_transforms"
orka.gpr:3:06: imported by "/path/to/repo/orka/orka/orka.gpr"
orka.gpr:3:06: imported by "/path/to/repo/orka/orka/orka-egl.gpr"
orka.gpr:3:06: imported by "/path/to/repo/awt/awt-linux.gpr"
gprbuild: "awt-linux.gpr" processing failed

If I go to ../orka/orka (orka crate in orka repo), and then run alr build -XORKA_SIMD_EXT="AVX2". It builds everything just fine.

(If I run alr update in awt repo I get a warning: warn: Missing variable in environment: os; 'other' expressions discarded.)

And I get some exceptions when running alr build -vvd in awt repo:

stderr: Pin path is not a valid directory: ../orka_egl
debug: ---8<--- Exception dump begin ---8<---
debug: ALIRE.CHECKED_ERROR
debug: /path/to/repo/awt/alire.lock:
solution
states
state  3
Loading release from manifest: ../orka/orka/alire.toml
pins
orka_egl
Pin path is not a valid directory: ../orka_egl
debug: raised ALIRE.CHECKED_ERROR : alire-stored-error:1

debug: Loading release orka_egl=0.0.0
debug: Valid root found at ../orka/orka_egl
debug: Loading release wayland_client_ada=0.0.0
stderr: Pin path is not a valid directory: ../wayland_ada_scanner
debug: ---8<--- Exception dump begin ---8<---
debug: ALIRE.CHECKED_ERROR
debug: /path/to/repo/awt/alire.lock:
solution
states
state  5
Loading release from manifest: ../wayland-ada/wayland_client_ada/alire.toml
pins
wayland_ada_scanner
Pin path is not a valid directory: ../wayland_ada_scanner
debug: raised ALIRE.CHECKED_ERROR : alire-stored-error:3

debug: Loading release wayland_cursor_ada=0.0.0
stderr: Pin path is not a valid directory: ../wayland_client_ada
debug: ---8<--- Exception dump begin ---8<---
debug: ALIRE.CHECKED_ERROR
debug: /path/to/repo/awt/alire.lock:
solution
states
state  6
Loading release from manifest: ../wayland-ada/wayland_cursor_ada/alire.toml
pins
wayland_client_ada
Pin path is not a valid directory: ../wayland_client_ada
debug: raised ALIRE.CHECKED_ERROR : alire-stored-error:5

debug: Loading release wayland_egl_ada=0.0.0
stderr: Pin path is not a valid directory: ../wayland_client_ada
debug: ---8<--- Exception dump begin ---8<---
debug: ALIRE.CHECKED_ERROR
debug: /path/to/repo/awt/alire.lock:
solution
states
state  7
Loading release from manifest: ../wayland-ada/wayland_egl_ada/alire.toml
pins
wayland_client_ada
Pin path is not a valid directory: ../wayland_client_ada
debug: raised ALIRE.CHECKED_ERROR : alire-stored-error:17

There are many more similar and duplicate exceptions, these are just a few of them. I can commit the changes to the alire.toml files if that helps you with debugging.

onox · 2021-06-17T20:32:38Z

I have pushed some commits with pins added to the various alire.toml files.

Also get this error in awt:

orka.gpr:19:04: no value defined for "os"
orka.gpr:19:20: undefined external reference "ALR_OS"
gprbuild: "awt-linux.gpr" processing failed

even though I have the following in orka/alire.toml:

[gpr-set-externals.'case(os)']
linux = { ALR_OS = "linux" }

mosteo · 2021-06-18T07:12:30Z

@mosteo Pins in wayland-ada repo seem to work, but Alire could try to simplify the relative paths of indirect dependencies, it currently just concatenates the paths. For example, in wayland_ada_info crate I get changes like this:
-path = "file:../wayland_ada_scanner"
+path = "file:../wayland_protocols_ada/../wayland_client_ada/../wayland_ada_scanner"

Actually I think this is caused by the relative path simplification functions. This is in the lockfile?

mosteo · 2021-06-18T07:13:45Z

I have pushed some commits with pins added to the various alire.toml files.

Thanks, this will help me for sure.

mosteo · 2021-06-18T09:33:56Z

@onox, thanks to your repos I have a clear picture of what's going wrong. I will submit the fix in a separate PR not to make this one even more complex, as it touches on recursive pin loading and the solver. I'll ping you there when ready.

onox reviewed May 28, 2021

View reviewed changes

mosteo added 16 commits June 14, 2021 14:07

Added data structures

3c07450

Pins may appear in manifest, being ignored

05a7614

Loading of user pins complete

b58d2fa

Pins are downloaded or skipped as needed

34c1a40

Pins are properly pruned, and info displayed

e01ff27

Minor testsuite tweak for a change in logging format

Added new-format pins to alire.toml for self-build

41c2cc5

Allow selective update of pins as for regular deps

6f6627b

Local pins work with new manifest syntax

b7d6605

Fixed bug in which version pins were not used

575e795

Make version explicit key in user pin

09ca589

Fix bug about confirming empty updates

254b5a3

Roots: Conflated dep updating into single Sync

5298c1a

We had two confusing Update_Dependencies and Update_And_Deploy_Dependencies that were in practice doing almost the same. There is now a single Sync_Dependencies.

Disable tests that rely on alr pin

d622f05

For now, these cannot work as we are going to remove the ability to edit pins via `alr with`/`alr pin`. This functionality could be reintroduced at a later time.

More tests temporarily disabled

aeb52cd

Most of those should be reimplemented in their manual edition alternative

Disable code for pin edition in command-line

d71b7aa

Changes to allow pins to non-dependencies

f293ec1

This is purely for user comfort and will probably result in dependencies having to be added at publish time. However, if we manage to restore command-line pinning, we can remove that pain by adding missing dependencies at that time.

mosteo force-pushed the feat/newpins branch from c96fdb4 to f293ec1 Compare June 14, 2021 15:59

Tests in tests/pin using new pins

00f1a34

Some tests are not easily portable without support from `alr with --use`. Since that should be easy to implement later, they are disabled for now and will be enabled in a subsequent patch.

mosteo force-pushed the feat/newpins branch from 793c4ff to 00f1a34 Compare June 15, 2021 20:38

Documentation on new pins

6d1a2e2

github-actions bot reviewed Jun 16, 2021

View reviewed changes

alire.lock Show resolved Hide resolved

doc/catalog-format-spec.md Outdated Show resolved Hide resolved

src/alire/alire-roots.adb Outdated Show resolved Hide resolved

src/alire/alire-roots.adb Outdated Show resolved Hide resolved

src/alire/alire-roots.adb Outdated Show resolved Hide resolved

mosteo added 4 commits June 16, 2021 10:07

Fix for testcase on Windows using path separators

674582e

Spelling fixes, and exclude lockfiles from check

0215202

New test for various invalid pin entries in manifest

ebaba3a

Also the required code changes to pass the test

Fixes from code self-review

b90150c

mosteo marked this pull request as ready for review June 16, 2021 20:26

mosteo requested a review from Fabien-Chouteau June 16, 2021 20:27

Fabien-Chouteau requested changes Jun 17, 2021

View reviewed changes

Fixes suggested during code review

9a3174e

mosteo added 2 commits June 17, 2021 17:09

Fix for missed update when there is no lockfile

472a849

We were creating an empty lockfile, which was newer than the manifest, and thus not triggering the expected automatic update.

Test to check pins are applied with no prior lockfile

e3fb579

This was a bug detected and corrected in the previous commit

mosteo force-pushed the feat/newpins branch from 2aca953 to e3fb579 Compare June 17, 2021 15:33

Fabien-Chouteau approved these changes Jun 18, 2021

View reviewed changes

mosteo mentioned this pull request Jun 21, 2021

Pins to remote branches #754

Merged

mosteo changed the base branch from master to merging/pins June 22, 2021 15:54

mosteo merged commit 13acd70 into merging/pins Jun 22, 2021

mosteo deleted the feat/newpins branch June 22, 2021 15:55

mosteo mentioned this pull request Aug 18, 2021

Move the lockfile inside $crate/alire #789

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pins in the manifest #743

Pins in the manifest #743

mosteo commented May 28, 2021 •

edited

Loading

Fabien-Chouteau commented May 28, 2021

onox May 28, 2021

mosteo May 29, 2021

Fabien-Chouteau Jun 17, 2021

onox commented May 28, 2021

mosteo commented May 29, 2021

mosteo commented Jun 16, 2021

Fabien-Chouteau Jun 17, 2021

raph-amiard commented Jun 17, 2021

Fabien-Chouteau commented Jun 17, 2021

mosteo commented Jun 17, 2021

mosteo commented Jun 17, 2021

mosteo commented Jun 17, 2021

onox commented Jun 17, 2021

onox commented Jun 17, 2021

onox commented Jun 17, 2021

mosteo commented Jun 18, 2021

mosteo commented Jun 18, 2021

mosteo commented Jun 18, 2021

Pins in the manifest #743

Pins in the manifest #743

Conversation

mosteo commented May 28, 2021 • edited Loading

Fabien-Chouteau commented May 28, 2021

onox May 28, 2021

Choose a reason for hiding this comment

mosteo May 29, 2021

Choose a reason for hiding this comment

Fabien-Chouteau Jun 17, 2021

Choose a reason for hiding this comment

onox commented May 28, 2021

mosteo commented May 29, 2021

mosteo commented Jun 16, 2021

Fabien-Chouteau Jun 17, 2021

Choose a reason for hiding this comment

raph-amiard commented Jun 17, 2021

Fabien-Chouteau commented Jun 17, 2021

mosteo commented Jun 17, 2021

mosteo commented Jun 17, 2021

mosteo commented Jun 17, 2021

onox commented Jun 17, 2021

onox commented Jun 17, 2021

onox commented Jun 17, 2021

mosteo commented Jun 18, 2021

mosteo commented Jun 18, 2021

mosteo commented Jun 18, 2021

mosteo commented May 28, 2021 •

edited

Loading