feat(gnovm): packages loader #2932

n0izn0iz · 2024-10-10T11:30:05Z

WIP

Continuation of #2201
Taking #2904 into account

This implements a Load function with a similar interface to the packages.Load go function

Contributors' checklist...

Added new tests, or not needed, or not feasible
Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory
Updated the official documentation or not needed
No breaking changes were made, or a BREAKING CHANGE: xxx message was included in the description
Added references to related issues and PRs
Provided any useful hints for running manual tests
Added new benchmarks to generated graphs, if any. More info here.

…esolver

Signed-off-by: Norman Meier <[email protected]>

…no-resolver

codecov · 2024-10-10T11:32:47Z

Codecov Report

All modified and coverable lines are covered by tests ✅

📢 Thoughts on this report? Let us know!

Signed-off-by: Norman Meier <[email protected]>

…ve download to own file Signed-off-by: Norman Meier <[email protected]>

Signed-off-by: Norman Meier <[email protected]>

moul · 2024-10-11T15:30:03Z

examples/gno.land/gno.mod

@@ -0,0 +1 @@
+module gno.land


do we need this file?

if we need it, please explain in a comment.

please see #2904 (comment) and tell me what you think

I think the best would be to have a gno.work file at repo root to support multiple modules in a monorepo
But in that case we need to implement the gno work commands

moul · 2024-10-11T15:32:03Z

gnovm/tests/machine_test.go

@@ -40,7 +40,7 @@ func TestMachineTestMemPackage(t *testing.T) {
 					Name: tt.name,
 					F: func(t2 *testing.T) { //nolint:thelper
 						rootDir := filepath.Join("..", "..")
-						store := TestStore(rootDir, "test", os.Stdin, os.Stdout, os.Stderr, ImportModeStdlibsOnly)
+						store := TestStore(rootDir, "test", nil, os.Stdin, os.Stdout, os.Stderr, ImportModeStdlibsOnly)


create a variable that you set to nil, so we now what is this parameter.

done in 0123512

gnovm/pkg/packages/download.go

moul · 2024-10-11T15:35:17Z

gnovm/pkg/packages/download.go

+func DownloadModule(pkgPath string, dst string) error {
+	modFilePath := filepath.Join(dst, ModfileName)
+	if _, err := os.Stat(modFilePath); os.IsNotExist(err) {
+		fmt.Fprintln(os.Stderr, "gno: downloading", pkgPath)


we usually don't use os.StdXXX, but stdouterr from a struct (io something) which is passed when needed.

done in e648305

moul · 2024-10-11T15:36:53Z

gnovm/pkg/packages/download.go

+			}
+		}
+
+		// write gno.mod


do we need this?

it depends, but to keep sanity, I think we do, see: #2904 (comment)

If you first download a sub-module, for example gno.land/p/demo/json/eisel_lemire
it will create the $GNOHOME/pkg/mod/gno.land/p/demo/json directory since it's a parent directory.
To know that the gno.land/p/demo/json package needs to be downloaded, I'm checking for the presence of $GNOHOME/pkg/mod/gno.land/p/demo/json/gno.mod

moul · 2024-10-11T15:42:10Z

gnovm/cmd/gno/test.go

@@ -617,3 +592,21 @@ func shouldRun(filter filterMatch, path string) bool {
 	ok, _ := filter.matches(elem, matchString)
 	return ok
 }
+
+// Adapted from https://yourbasic.org/golang/formatting-byte-size-to-human-readable-format/
+func prettySize(nb int64) string {


why moving this from util?

conflict resolution fail, reverted in 53c5770

moul · 2024-10-11T15:43:46Z

gnovm/pkg/packages/resolve.go

+	Dir              string   `json:",omitempty"`
+	ImportPath       string   `json:",omitempty"`
+	Name             string   `json:",omitempty"`
+	Root             string   `json:",omitempty"`


add comments to explain the fields.

this one for instance, is it the "location of the package that can be either specified locally or automatically cached on the disk"? or something else?

done in 457b0c9

Co-authored-by: Manfred Touron <[email protected]>

Signed-off-by: Norman Meier <[email protected]>

thehowl · 2024-10-17T19:15:41Z

Thanks <3

Signed-off-by: Norman Meier <[email protected]>

A step towards the importer package (#2932) and future of `gno.mod` (#2904) - BREAKING CHANGE: remove `require` statement support from `gno.mod` - BREAKING CHANGE: remove `-v` (verbose) and `--remote` flags in `gno mod download` - Don't require version specification in `gno.mod`'s `replace` statements - Use `.gno` files `import` statements to find dependencies - Extract and refacto imports gathering utils in `gnovm/pkg/packages` - Add `gnovm/cmd/gno/internal/pkgdownload.PackageFetcher` interface - Implement `PackageFetcher` using `vm/qfile` queries in `gnovm/cmd/gno/internal/pkgdownload/rpcpackagefetcher` - Rewrite single package download routine in `gnovm/cmd/gno/internal/pkgdownload` - Move and refacto dependencies download routine in `gnovm/cmd/gno/download_deps.go` - Add a `--remote-overrides` flag for `gno mod download` that takes `chain-domain=rpc-url` comma-separated pairs to override endpoints used to fetch packages - Add and use a testing implementation of `PackageFetcher` called `examplesPackageFetcher` that serves package from the `examples` directory for testing purposes (download tests before this PR use the portal loop public endpoint) - Make `ReadMemPackage` and it's dependencies error-out instead of panicking - Create panicking `MemPackage` utils that wrap the erroring ones and use them at existing callsites I decided to do this first to avoid having multiple ways to resolve dependencies lying around in the codebase and causing confusion in subsequent steps <details><summary>Contributors' checklist...</summary> - [x] Added new tests, or not needed, or not feasible - [ ] Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory - [ ] Updated the official documentation or not needed - [x] No breaking changes were made, or a `BREAKING CHANGE: xxx` message was included in the description - [x] Added references to related issues and PRs - [ ] Provided any useful hints for running manual tests </details> --------- Signed-off-by: Norman Meier <[email protected]> Co-authored-by: Morgan Bazalgette <[email protected]>

…ng#3123) A step towards the importer package (gnolang#2932) and future of `gno.mod` (gnolang#2904) - BREAKING CHANGE: remove `require` statement support from `gno.mod` - BREAKING CHANGE: remove `-v` (verbose) and `--remote` flags in `gno mod download` - Don't require version specification in `gno.mod`'s `replace` statements - Use `.gno` files `import` statements to find dependencies - Extract and refacto imports gathering utils in `gnovm/pkg/packages` - Add `gnovm/cmd/gno/internal/pkgdownload.PackageFetcher` interface - Implement `PackageFetcher` using `vm/qfile` queries in `gnovm/cmd/gno/internal/pkgdownload/rpcpackagefetcher` - Rewrite single package download routine in `gnovm/cmd/gno/internal/pkgdownload` - Move and refacto dependencies download routine in `gnovm/cmd/gno/download_deps.go` - Add a `--remote-overrides` flag for `gno mod download` that takes `chain-domain=rpc-url` comma-separated pairs to override endpoints used to fetch packages - Add and use a testing implementation of `PackageFetcher` called `examplesPackageFetcher` that serves package from the `examples` directory for testing purposes (download tests before this PR use the portal loop public endpoint) - Make `ReadMemPackage` and it's dependencies error-out instead of panicking - Create panicking `MemPackage` utils that wrap the erroring ones and use them at existing callsites I decided to do this first to avoid having multiple ways to resolve dependencies lying around in the codebase and causing confusion in subsequent steps <details><summary>Contributors' checklist...</summary> - [x] Added new tests, or not needed, or not feasible - [ ] Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory - [ ] Updated the official documentation or not needed - [x] No breaking changes were made, or a `BREAKING CHANGE: xxx` message was included in the description - [x] Added references to related issues and PRs - [ ] Provided any useful hints for running manual tests </details> --------- Signed-off-by: Norman Meier <[email protected]> Co-authored-by: Morgan Bazalgette <[email protected]>

thehowl · 2024-12-08T09:32:39Z

@n0izn0iz what are other dependencies awaiting before tackling merging this one?

Gno2D2 · 2024-12-08T09:33:28Z

I'm a bot that assists the Gno Core team in maintaining this repository. My role is to ensure that contributors understand and follow our guidelines, helping to streamline the development process.

The following requirements must be fulfilled before a pull request can be merged.
Some requirement checks are automated and can be verified by the CI, while others need manual verification by a staff member.

These requirements are defined in this configuration file.

Automated Checks

🔴 Maintainers must be able to edit this pull request (more info)
🔴 The pull request head branch must be up-to-date with its base (more info)

Manual Checks

No manual checks match this pull request.

Debug

Automated Checks
Maintainers must be able to edit this pull request (more info)
If
🟢 Condition met
└── 🟢 On every pull request
Then
🔴 Requirement not satisfied
└── 🔴 Maintainer can modify this pull request
The pull request head branch must be up-to-date with its base (more info)
If
🟢 Condition met
└── 🟢 On every pull request
Then
🔴 Requirement not satisfied
└── 🔴 Head branch (TERITORI:gno-resolver) is up to date with base (master): behind by 115 / ahead by 33

n0izn0iz · 2024-12-08T10:06:30Z

I wonder if I should do the genesis stdlibs first, I'm not sure how these two would interact since we need to return the source dir of packages for tooling like gnopls and I'm not sure if we should download them from the remote and how we should store them if we do. Also not sure how to proceed with the embedded ones if we don't download them

…ng#3123) A step towards the importer package (gnolang#2932) and future of `gno.mod` (gnolang#2904) - BREAKING CHANGE: remove `require` statement support from `gno.mod` - BREAKING CHANGE: remove `-v` (verbose) and `--remote` flags in `gno mod download` - Don't require version specification in `gno.mod`'s `replace` statements - Use `.gno` files `import` statements to find dependencies - Extract and refacto imports gathering utils in `gnovm/pkg/packages` - Add `gnovm/cmd/gno/internal/pkgdownload.PackageFetcher` interface - Implement `PackageFetcher` using `vm/qfile` queries in `gnovm/cmd/gno/internal/pkgdownload/rpcpackagefetcher` - Rewrite single package download routine in `gnovm/cmd/gno/internal/pkgdownload` - Move and refacto dependencies download routine in `gnovm/cmd/gno/download_deps.go` - Add a `--remote-overrides` flag for `gno mod download` that takes `chain-domain=rpc-url` comma-separated pairs to override endpoints used to fetch packages - Add and use a testing implementation of `PackageFetcher` called `examplesPackageFetcher` that serves package from the `examples` directory for testing purposes (download tests before this PR use the portal loop public endpoint) - Make `ReadMemPackage` and it's dependencies error-out instead of panicking - Create panicking `MemPackage` utils that wrap the erroring ones and use them at existing callsites I decided to do this first to avoid having multiple ways to resolve dependencies lying around in the codebase and causing confusion in subsequent steps <details><summary>Contributors' checklist...</summary> - [x] Added new tests, or not needed, or not feasible - [ ] Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory - [ ] Updated the official documentation or not needed - [x] No breaking changes were made, or a `BREAKING CHANGE: xxx` message was included in the description - [x] Added references to related issues and PRs - [ ] Provided any useful hints for running manual tests </details> --------- Signed-off-by: Norman Meier <[email protected]> Co-authored-by: Morgan Bazalgette <[email protected]>

thehowl and others added 8 commits May 26, 2024 13:41

copy over from gnoutil pr

01b0719

remove util_test

232834c

Merge branch 'master' of github.com:gnolang/gno into dev/morgan/gno-r…

f378217

…esolver

linter fixes

8afeacf

tmp: gno list

808061d

Signed-off-by: Norman Meier <[email protected]>

tmp

2c5d604

Signed-off-by: Norman Meier <[email protected]>

tmp: working

dfc0881

Signed-off-by: Norman Meier <[email protected]>

chore: merge remote-tracking branch 'origin/master' into dev/morgan/g…

a35050c

…no-resolver

github-actions bot assigned n0izn0iz Oct 10, 2024

github-actions bot added 🧾 package/realm Tag used for new Realms or Packages. 📦 🤖 gnovm Issues or PRs gnovm related 📦 ⛰️ gno.land Issues or PRs gno.land package related labels Oct 10, 2024

n0izn0iz added 4 commits October 10, 2024 14:47

feat: match mapping

7c7165f

Signed-off-by: Norman Meier <[email protected]>

fix: test and lint

1c3a830

Signed-off-by: Norman Meier <[email protected]>

fix: partially fix transpile

f95e4bb

Signed-off-by: Norman Meier <[email protected]>

tmp: support loaded pkgs in test store

2d990e5

Signed-off-by: Norman Meier <[email protected]>

n0izn0iz mentioned this pull request Oct 11, 2024

rfc: the future of gno.mod #2904

Open

n0izn0iz added 6 commits October 11, 2024 12:59

tmp: better Module info

6995485

Signed-off-by: Norman Meier <[email protected]>

tmp: follow go resolver logic

fb8d518

Signed-off-by: Norman Meier <[email protected]>

fix: resolver usage in test

1c51509

Signed-off-by: Norman Meier <[email protected]>

chore: mod tidy

da65417

Signed-off-by: Norman Meier <[email protected]>

chore: ignore draft

4a8401a

Signed-off-by: Norman Meier <[email protected]>

chore: rename importer -> packages like go, break import cycle and mo…

e792771

…ve download to own file Signed-off-by: Norman Meier <[email protected]>

n0izn0iz changed the title ~~feat(gnovm): package resolver lib~~ feat(gnovm): packages loader Oct 11, 2024

chore: lint and fmt

08fbd84

Signed-off-by: Norman Meier <[email protected]>

moul reviewed Oct 11, 2024

View reviewed changes

gnovm/pkg/packages/download.go Show resolved Hide resolved

moul reviewed Oct 11, 2024

View reviewed changes

n0izn0iz and others added 7 commits October 11, 2024 17:46

Update gnovm/pkg/packages/download.go

f93a221

Co-authored-by: Manfred Touron <[email protected]>

chore: revert merge fails

53c5770

Signed-off-by: Norman Meier <[email protected]>

chore: add package and module struct fields comments

457b0c9

Signed-off-by: Norman Meier <[email protected]>

chore: go -> gno

59a752a

Signed-off-by: Norman Meier <[email protected]>

fix: use io instead of os

e648305

Signed-off-by: Norman Meier <[email protected]>

chore: explicit args

0123512

Signed-off-by: Norman Meier <[email protected]>

chore: improve comment

d266b8e

Signed-off-by: Norman Meier <[email protected]>

n0izn0iz added 7 commits October 21, 2024 18:19

fix: break import cycle

ac435c2

Signed-off-by: Norman Meier <[email protected]>

chore: merge remote-tracking branch 'origin/master' into gno-resolver

178706f

chore: linter

9fbe83c

Signed-off-by: Norman Meier <[email protected]>

chore: reuse helper

d78fb90

Signed-off-by: Norman Meier <[email protected]>

chore: merge remote-tracking branch 'origin/master' into gno-resolver

f7b6df7

tmp: use gno.work and add back mod files

6f2330e

Signed-off-by: Norman Meier <[email protected]>

chore: merge remote-tracking branch 'origin/master' into gno-resolver

2d9fc96

n0izn0iz mentioned this pull request Nov 14, 2024

feat(gnomod)!: forbid require and find dependencies without it #3123

Merged

6 tasks

gfanton mentioned this pull request Nov 30, 2024

wip(gnodev): lazy loading & staging support #3237

Draft

9 tasks

thehowl mentioned this pull request Dec 8, 2024

feat(gnovm): add importer package #2201

Closed

thehowl mentioned this pull request Dec 9, 2024

gno doc command + rpc + gnoweb view #522

Open

n0izn0iz mentioned this pull request Dec 11, 2024

feat(gnovm): categorize imports #3323

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(gnovm): packages loader #2932

feat(gnovm): packages loader #2932

n0izn0iz commented Oct 10, 2024 •

edited

Loading

codecov bot commented Oct 10, 2024 •

edited

Loading

moul Oct 11, 2024

n0izn0iz Oct 11, 2024

n0izn0iz Oct 11, 2024 •

edited

Loading

moul Oct 11, 2024

n0izn0iz Oct 11, 2024

moul Oct 11, 2024

n0izn0iz Oct 11, 2024

moul Oct 11, 2024

n0izn0iz Oct 11, 2024

n0izn0iz Oct 11, 2024 •

edited

Loading

moul Oct 11, 2024

n0izn0iz Oct 11, 2024 •

edited

Loading

moul Oct 11, 2024

n0izn0iz Oct 11, 2024

thehowl commented Oct 17, 2024

thehowl commented Dec 8, 2024

Gno2D2 commented Dec 8, 2024 •

edited

Loading

If

Then

If

Then

n0izn0iz commented Dec 8, 2024

feat(gnovm): packages loader #2932

Are you sure you want to change the base?

feat(gnovm): packages loader #2932

Conversation

n0izn0iz commented Oct 10, 2024 • edited Loading

codecov bot commented Oct 10, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

n0izn0iz Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

n0izn0iz Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

n0izn0iz Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thehowl commented Oct 17, 2024

thehowl commented Dec 8, 2024

Gno2D2 commented Dec 8, 2024 • edited Loading

Automated Checks

Manual Checks

If

Then

If

Then

n0izn0iz commented Dec 8, 2024

n0izn0iz commented Oct 10, 2024 •

edited

Loading

codecov bot commented Oct 10, 2024 •

edited

Loading

n0izn0iz Oct 11, 2024 •

edited

Loading

n0izn0iz Oct 11, 2024 •

edited

Loading

n0izn0iz Oct 11, 2024 •

edited

Loading

Gno2D2 commented Dec 8, 2024 •

edited

Loading