Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: use go 1.20 and latest go-waku #4522

Merged
merged 1 commit into from
Jan 18, 2024
Merged

chore: use go 1.20 and latest go-waku #4522

merged 1 commit into from
Jan 18, 2024

Conversation

richard-ramos
Copy link
Member

@richard-ramos richard-ramos commented Jan 3, 2024

See title

@ghost
Copy link

ghost commented Jan 3, 2024

vendor/github.com/pion/webrtc/v3/yarn.lock changed. Pinging @corpetty

@status-im-auto
Copy link
Member

status-im-auto commented Jan 3, 2024

Jenkins Builds

Click to see older builds (93)
Commit #️⃣ Finished (UTC) Duration Platform Result
8521a2a #1 2024-01-03 21:59:19 ~1 min android 📄log
8521a2a #1 2024-01-03 21:59:25 ~1 min linux 📄log
8521a2a #1 2024-01-03 21:59:26 ~1 min ios 📄log
✖️ 8521a2a #1 2024-01-03 22:00:26 ~2 min tests 📄log
827a5b1 #2 2024-01-03 22:11:29 ~18 sec android 📄log
827a5b1 #2 2024-01-03 22:12:20 ~1 min linux 📄log
827a5b1 #2 2024-01-03 22:12:21 ~1 min ios 📄log
✖️ 827a5b1 #2 2024-01-03 22:13:16 ~2 min tests 📄log
ca471bb #3 2024-01-03 22:21:07 ~11 sec android 📄log
ca471bb #3 2024-01-03 22:21:07 ~11 sec linux 📄log
✖️ ca471bb #3 2024-01-03 22:21:07 ~11 sec tests 📄log
ca471bb #3 2024-01-03 22:21:11 ~14 sec ios 📄log
a42e454 #4 2024-01-10 14:42:06 ~2 min ios 📄log
a42e454 #4 2024-01-10 14:43:24 ~3 min linux 📄log
✖️ a42e454 #4 2024-01-10 14:43:31 ~3 min tests 📄log
a42e454 #4 2024-01-10 14:44:13 ~4 min android 📄log
a1bdfd5 #5 2024-01-10 14:44:59 ~11 sec linux 📄log
✖️ a1bdfd5 #5 2024-01-10 14:45:07 ~18 sec tests 📄log
a1bdfd5 #5 2024-01-10 14:45:36 ~47 sec ios 📄log
a1bdfd5 #5 2024-01-10 14:48:33 ~3 min android 📄log
27bf5a0 #6 2024-01-10 15:10:22 ~23 sec android 📄log
27bf5a0 #6 2024-01-10 15:10:23 ~24 sec ios 📄log
27bf5a0 #6 2024-01-10 15:10:29 ~24 sec linux 📄log
✖️ 27bf5a0 #6 2024-01-10 15:12:10 ~2 min tests 📄log
8f44c51 #7 2024-01-10 15:13:30 ~16 sec linux 📄log
8f44c51 #7 2024-01-10 15:13:37 ~23 sec ios 📄log
8f44c51 #7 2024-01-10 15:14:05 ~52 sec android 📄log
✖️ 8f44c51 #7 2024-01-10 15:14:11 ~52 sec tests 📄log
a9a2d5b #8 2024-01-10 15:21:37 ~20 sec ios 📄log
a9a2d5b #8 2024-01-10 15:21:51 ~37 sec android 📄log
a9a2d5b #8 2024-01-10 15:22:17 ~1 min linux 📄log
✖️ a9a2d5b #8 2024-01-10 15:23:16 ~1 min tests 📄log
633e54c #9 2024-01-10 15:27:23 ~19 sec android 📄log
633e54c #9 2024-01-10 15:27:50 ~46 sec ios 📄log
633e54c #9 2024-01-10 15:28:15 ~1 min linux 📄log
✖️ 633e54c #9 2024-01-10 15:29:08 ~2 min tests 📄log
ada8eb3 #10 2024-01-10 15:53:39 ~10 sec ios 📄log
ada8eb3 #10 2024-01-10 15:53:39 ~11 sec linux 📄log
ada8eb3 #10 2024-01-10 15:53:41 ~13 sec android 📄log
✖️ ada8eb3 #10 2024-01-10 15:53:43 ~11 sec tests 📄log
f358aaf #11 2024-01-10 19:26:50 ~1 min ios 📄log
f358aaf #11 2024-01-10 19:26:50 ~1 min linux 📄log
✖️ f358aaf #11 2024-01-10 19:27:10 ~1 min tests 📄log
f358aaf #11 2024-01-10 19:27:15 ~1 min android 📄log
df8b955 #12 2024-01-10 19:27:26 ~13 sec linux 📄log
df8b955 #12 2024-01-10 19:27:30 ~20 sec ios 📄log
✖️ df8b955 #12 2024-01-10 19:28:10 ~53 sec tests 📄log
df8b955 #12 2024-01-10 19:29:06 ~1 min android 📄log
aec4c80 #13 2024-01-10 19:39:31 ~11 sec android 📄log
aec4c80 #13 2024-01-10 19:39:35 ~9.4 sec linux 📄log
aec4c80 #13 2024-01-10 19:39:36 ~9.9 sec ios 📄log
✖️ aec4c80 #13 2024-01-10 19:39:36 ~10 sec tests 📄log
12c93c8 #14 2024-01-10 19:42:10 ~20 sec ios 📄log
12c93c8 #14 2024-01-10 19:42:45 ~55 sec linux 📄log
✖️ 12c93c8 #14 2024-01-10 19:42:51 ~1 min tests 📄log
12c93c8 #14 2024-01-10 19:43:29 ~1 min android 📄log
✖️ 99db6fc #15 2024-01-10 19:44:22 ~1 min tests 📄log
99db6fc #15 2024-01-10 19:44:39 ~1 min linux 📄log
99db6fc #15 2024-01-10 19:44:48 ~1 min android 📄log
✔️ 99db6fc #15 2024-01-10 19:46:08 ~2 min ios 📦zip
716c0da #16 2024-01-10 19:51:54 ~27 sec android 📄log
716c0da #16 2024-01-10 19:52:03 ~28 sec linux 📄log
✖️ 716c0da #16 2024-01-10 19:52:37 ~1 min tests 📄log
✔️ 716c0da #16 2024-01-10 19:54:12 ~2 min ios 📦zip
7cc05c6 #17 2024-01-10 20:05:38 ~1 min android 📄log
✖️ 7cc05c6 #17 2024-01-10 20:06:05 ~1 min tests 📄log
7cc05c6 #17 2024-01-10 20:06:22 ~1 min linux 📄log
✔️ 7cc05c6 #17 2024-01-10 20:08:00 ~3 min ios 📦zip
98efd25 #18 2024-01-10 20:14:53 ~28 sec android 📄log
98efd25 #18 2024-01-10 20:15:01 ~31 sec linux 📄log
✖️ 98efd25 #18 2024-01-10 20:15:25 ~51 sec tests 📄log
✔️ 98efd25 #18 2024-01-10 20:17:10 ~2 min ios 📦zip
92a168a #19 2024-01-11 13:51:54 ~1 min linux 📄log
92a168a #19 2024-01-11 13:52:24 ~2 min android 📄log
✖️ 92a168a #19 2024-01-11 13:52:55 ~2 min tests 📄log
✖️ 2e91d7b #20 2024-01-11 14:18:05 ~1 min tests 📄log
✔️ 2e91d7b #20 2024-01-11 14:18:13 ~1 min android 📦aar
✔️ 2e91d7b #20 2024-01-11 14:18:54 ~2 min linux 📦zip
✔️ 5da3b57 #21 2024-01-11 16:04:38 ~2 min linux 📦zip
✔️ 5da3b57 #21 2024-01-11 16:06:46 ~5 min android 📦aar
✖️ 5da3b57 #21 2024-01-11 16:36:17 ~34 min tests 📄log
✔️ f245a83 #22 2024-01-12 17:45:55 ~2 min linux 📦zip
✔️ f245a83 #22 2024-01-12 17:46:47 ~3 min android 📦aar
✔️ f245a83 #21 2024-01-12 17:49:20 ~5 min ios 📦zip
✖️ f245a83 #22 2024-01-12 18:12:59 ~29 min tests 📄log
✔️ b47b206 #23 2024-01-15 08:35:01 ~1 min android 📦aar
✔️ b47b206 #22 2024-01-15 08:36:37 ~3 min ios 📦zip
✔️ b47b206 #23 2024-01-15 08:36:46 ~3 min linux 📦zip
✖️ b47b206 #23 2024-01-15 08:56:01 ~22 min tests 📄log
95b9867 #24 2024-01-15 08:43:25 ~10 sec android 📄log
95b9867 #24 2024-01-15 08:43:25 ~10 sec linux 📄log
95b9867 #23 2024-01-15 08:43:28 ~11 sec ios 📄log
✖️ 95b9867 #24 2024-01-15 08:56:20 ~11 sec tests 📄log
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 86e2b53 #25 2024-01-15 10:34:53 ~3 min android 📦aar
✔️ 86e2b53 #25 2024-01-15 10:35:38 ~4 min linux 📦zip
✔️ 86e2b53 #24 2024-01-15 10:36:13 ~4 min ios 📦zip
✖️ 86e2b53 #25 2024-01-15 10:53:37 ~21 min tests 📄log
✖️ 86e2b53 #26 2024-01-15 11:16:25 ~22 min tests 📄log
✖️ 86e2b53 #28 2024-01-15 12:16:41 ~19 min tests 📄log
✖️ 86e2b53 #30 2024-01-16 07:22:15 ~3 min tests 📄log
✖️ 86e2b53 #31 2024-01-16 07:49:31 ~20 min tests 📄log
✖️ 86e2b53 #32 2024-01-16 08:07:28 ~14 min tests 📄log
✖️ 86e2b53 #33 2024-01-16 08:34:36 ~19 min tests 📄log
✖️ 86e2b53 #34 2024-01-16 09:34:21 ~19 min tests 📄log
✔️ 06d04f6 #25 2024-01-18 14:32:44 ~4 min ios 📦zip

@ghost
Copy link

ghost commented Jan 3, 2024

vendor/github.com/pion/webrtc/v3/yarn.lock changed. Pinging @corpetty

1 similar comment
@ghost
Copy link

ghost commented Jan 3, 2024

vendor/github.com/pion/webrtc/v3/yarn.lock changed. Pinging @corpetty

@richard-ramos
Copy link
Member Author

@jakubgs , while i was attempting to upgrade to go1.21, i ended up with this error in nix:

[2024-01-03T22:21:05.635Z]        error: undefined variable 'go_1_21'

According to https://discourse.nixos.org/t/how-to-get-go-1-21-in-nix-shell/33433/3 it seems that the default go version is now 1.21 and there is no go_1_21 package, but seems weird to not specify the go version. Your advice is appreciated!

@jakubgs
Copy link
Member

jakubgs commented Jan 5, 2024

@richard-ramos remember that the packages available form Nix are controlled by the version of nixpkgs repo used, which in our case is controlled in the shell.nix by specifying a nixpkgs commit:

status-go/shell.nix

Lines 2 to 7 in 846a4e2

/* This should match Nixpkgs commit in status-mobile. */
source ? builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/e7603eba51f2c7820c0a182c6bbb351181caa8e7.tar.gz";
sha256 = "sha256:0mwck8jyr74wh1b7g6nac1mxy6a0rkppz8n12andsffybsipz5jw";
},
pkgs ? import (source){

If you look at the e7603eba51f2c7820c0a182c6bbb351181caa8e7 commit you'll see it doesn't contain Go 1.21:
https://github.com/NixOS/nixpkgs/tree/e7603eba51f2c7820c0a182c6bbb351181caa8e7/pkgs/development/compilers/go

Now as the comment states, this should be the same version as used in status-mobile to avoid diverging behaviors due to different tooling and libraries, but a temporary divergence probably wouldn't be a big issue.

As for what to upgrade to, I would argue that using master might not be the best option, and I'd switch it to nixos-23.11 to use a stable branch. Though I would prefer to start the upgrade with status-mobile as that will have more issues most probably.

@jakubgs
Copy link
Member

jakubgs commented Jan 5, 2024

And actually, @siddarthkay is attempting a Nixpkgs upgrade in here:

So you might want to sync with him as to what version to use.

@richard-ramos
Copy link
Member Author

richard-ramos commented Jan 10, 2024

@jakubgs: remember that the packages available form Nix are controlled by the version of nixpkgs repo used, which in our case is controlled in the shell.nix by specifying a nixpkgs commit .... So you might want to sync with @siddarthkay as to what version to use.

Just checked 23.11, (the one used in mobile PR) and I see it contains the go version I need. I'll try changing it in this PR and see how it goes.
image

Once status-im/status-mobile#18321 is merged we can proceed to upgrade nix in status-go

@richard-ramos richard-ramos force-pushed the test-bump-libp2p branch 5 times, most recently from 8f44c51 to a9a2d5b Compare January 10, 2024 15:20
@richard-ramos
Copy link
Member Author

Looks like upgrading to go1.21 ends up causing issues in the torrent lib. (cc: @siddarthkay @chaitanyaprem), maybe there's a new version compatible with it.

[2024-01-10T15:21:41.590Z] vendor/github.com/anacrolix/go-libutp/callbacks.go:16:10: cannot define new methods on non-local type *C.utp_callback_arguments
[2024-01-10T15:21:41.590Z] vendor/github.com/anacrolix/go-libutp/callbacks.go:24:10: cannot define new methods on non-local type *C.utp_callback_arguments
[2024-01-10T15:21:41.590Z] vendor/github.com/anacrolix/go-libutp/callbacks.go:28:10: cannot define new methods on non-local type *C.utp_callback_arguments
[2024-01-10T15:21:41.590Z] vendor/github.com/anacrolix/go-libutp/callbacks.go:32:10: cannot define new methods on non-local type *C.utp_callback_arguments
[2024-01-10T15:21:41.590Z] vendor/github.com/anacrolix/go-libutp/callbacks.go:36:10: cannot define new methods on non-local type *C.utp_callback_arguments
[2024-01-10T15:21:41.590Z] vendor/github.com/anacrolix/go-libutp/utp.go:29:12: cannot define new methods on non-local type *C.utp_context
[2024-01-10T15:21:41.590Z] vendor/github.com/anacrolix/go-libutp/utp.go:40:12: cannot define new methods on non-local type *C.utp_context

@richard-ramos
Copy link
Member Author

richard-ramos commented Jan 10, 2024

After changing the version of anacrolix/torrent, now gomobile has a different complain:

 unable to import bind/java: [-: # golang.org/x/mobile/bind/java
[2024-01-10T15:27:21.903Z] /nix/store/nb4a2r49nfnpfqhixfmzkxh38iighjk0-gomobile-unstable-2022-05-18/src/golang.org/x/mobile/bind/java/context_android.go:9:10: fatal error: jni.h: No such file or directory
[2024-01-10T15:27:21.903Z]     9 | //#include <jni.h>
[2024-01-10T15:27:21.903Z]       |          ^~~~~~~
[2024-01-10T15:27:21.903Z] compilation terminated.]

I did run into this before, so it seems we need to upgrade gomobile somehow?

@jakubgs
Copy link
Member

jakubgs commented Jan 10, 2024

To upgrade gomobile we need to override the buildGoModule attribute since the use of a builder makes this a nested derivation, so just using overrideAttrs on gomobile wouldn't be enough:

status-go/shell.nix

Lines 27 to 39 in df8b955

# https://github.com/golang/go/issues/58426
gomobile = prev.gomobile.override {
buildGoModule = args: prev.buildGo121Module ( args // rec {
version = "unstable-2023-11-27";
src = prev.fetchgit {
rev = "76ac6878050a2eef81867f2c6c21108e59919e8f";
name = "gomobile";
url = "https://go.googlesource.com/mobile";
sha256 = "sha256-mq7gKccvI7VCBEiQTueWxMPOCgg/MGE8y2+BlwWx5pw=";
};
vendorHash = "sha256-8OBLVd4zs89hoJXzC8BPRgrYjjR7DiA39+7tTaSYUFI=";
});
};

For more details see:

@richard-ramos richard-ramos changed the title test: bump go-libp2p test: bump go-libp2p and go1.21 Jan 10, 2024
@richard-ramos richard-ramos force-pushed the test-bump-libp2p branch 3 times, most recently from 99db6fc to 716c0da Compare January 10, 2024 19:51
@richard-ramos
Copy link
Member Author

looks like there's a dependency that also includes sqlite and clashes with sqlcipher. I'm trying to figure out where this is happening

cc: @siddarthkay @chaitanyaprem

[2024-01-10T19:51:53.371Z] ld: error: duplicate symbol: sqlite3_aggregate_context
[2024-01-10T19:51:53.371Z] >>> defined at sqlite3.c:89125
[2024-01-10T19:51:53.371Z] >>>            ../../go-link-3238948430/000084.o:(sqlite3_aggregate_context)
[2024-01-10T19:51:53.371Z] >>> defined at sqlite3.c:89261 (/home/jenkins/workspace/status-go_prs_android_PR-4522@tmp/go/src/github.com/status-im/status-go/vendor/github.com/go-llsqlite/crawshaw/./c/sqlite3.c:89261)
[2024-01-10T19:51:53.371Z] >>>            ../../go-link-3238948430/000111.o:(.text+0x103A0)
[2024-01-10T19:51:53.371Z] 
[2024-01-10T19:51:53.371Z] ld: error: duplicate symbol: sqlite3_auto_extension
[2024-01-10T19:51:53.371Z] >>> defined at sqlite3.c:138759
[2024-01-10T19:51:53.371Z] >>>            ../../go-link-3238948430/000084.o:(sqlite3_auto_extension)
[2024-01-10T19:51:53.371Z] >>> defined at sqlite3.c:134211 (/home/jenkins/workspace/status-go_prs_android_PR-4522@tmp/go/src/github.com/status-im/status-go/vendor/github.com/go-llsqlite/crawshaw/./c/sqlite3.c:134211)
[2024-01-10T19:51:53.371Z] >>>            ../../go-link-3238948430/000111.o:(.text+0x180E0)
...

@richard-ramos
Copy link
Member Author

Oh. It's anacrolix/torrent

@status-im status-im deleted a comment Jan 10, 2024
@status-im status-im deleted a comment Jan 10, 2024
Copy link
Collaborator

@igor-sirotin igor-sirotin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💪

appDbPath + "-shm",
appDbPath + "-wal",
appDbPath + shm,
appDbPath + "-wal", // nolint: goconst
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see where you got tired of fixing goconst 😄

shell.nix Outdated
@@ -1,17 +1,17 @@
{
/* This should match Nixpkgs commit in status-mobile. */
source ? builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/e7603eba51f2c7820c0a182c6bbb351181caa8e7.tar.gz";
sha256 = "sha256:0mwck8jyr74wh1b7g6nac1mxy6a0rkppz8n12andsffybsipz5jw";
url = "https://github.com/NixOS/nixpkgs/archive/ddf0003c57fb5cbb4a9754f2f6d5ebe9cdae5151.tar.gz";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this commit going to be used in status-mobile as well?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be! i copied it from status-im/status-mobile#18321 so it probably makes sense to merge both this PR and that one once approved

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm
we cant use this url -> https://github.com/NixOS/nixpkgs/archive/ddf0003c57fb5cbb4a9754f2f6d5ebe9cdae5151.tar.gz since its from nixpkgs master branch, what I had in the status-mobile PR back then was an experiment :D

Here is what I'll do next week :

  • open PR with this status-go version in status-mobile to see what breaks and what doesn't.
  • we'll verify if any nixpkgs stable version 23.11 got the fixes we want related to GraalVM or not ( else we just fork and move on )

and then what url we decide on at status-mobile we can use that here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually for now you could just use : e7603eba51f2c7820c0a182c6bbb351181caa8e7
since we do have go 1.20 over here -> https://github.com/NixOS/nixpkgs/blob/e7603eba51f2c7820c0a182c6bbb351181caa8e7/pkgs/development/compilers/go/1.20.nix

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright! will do that

@ghost
Copy link

ghost commented Jan 12, 2024

vendor/github.com/pion/webrtc/v3/yarn.lock changed. Pinging @corpetty

@siddarthkay
Copy link
Contributor

but work will be needed to get status-go to work with go1.21 due to dependencies included by anacrolix/torrent

Hey @richard-ramos : can you create an issue with the error you faced and I can take a look for what we can do to get the torrent package to work with go 1.21

@ghost
Copy link

ghost commented Jan 15, 2024

vendor/github.com/pion/webrtc/v3/yarn.lock changed. Pinging @corpetty

@ghost
Copy link

ghost commented Jan 15, 2024

vendor/github.com/pion/webrtc/v3/yarn.lock changed. Pinging @corpetty

@ghost
Copy link

ghost commented Jan 15, 2024

vendor/github.com/pion/webrtc/v3/yarn.lock changed. Pinging @corpetty

Copy link
Member

@jakubgs jakubgs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm approving the shell.nix changes.

@ghost
Copy link

ghost commented Jan 18, 2024

vendor/github.com/pion/webrtc/v3/yarn.lock changed. Pinging @corpetty

@cammellos cammellos merged commit d0ca444 into develop Jan 18, 2024
2 of 3 checks passed
@cammellos cammellos deleted the test-bump-libp2p branch January 18, 2024 14:28
@ghost
Copy link

ghost commented Jan 18, 2024

vendor/github.com/pion/webrtc/v3/yarn.lock changed. Pinging @corpetty

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants