Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: update obfs4, goptlib, snowflake #1347

Merged
merged 1 commit into from
Oct 6, 2023
Merged

chore: update obfs4, goptlib, snowflake #1347

merged 1 commit into from
Oct 6, 2023

Conversation

bassosimone
Copy link
Contributor

This diff upgrades obfs4, goptlib, and snowflake. I tried to import lyrebird instead of obfs4, but that did not work as intended. It seems tags aren't compatible with the format expected by Go (vX.Y.Z) and I did not want to pin to the latest commit. I will report this issue to lyrebird developers later today.

While there, also upgrade x/net and x/exp.

This work is part of ooni/probe#2524.

Here's what we could not upgrade (based on the content of go.mod after we committed this diff and ran go get -u -v -d ./...):

diff --git a/go.mod b/go.mod
index 67e0af02..18175d73 100644
--- a/go.mod
+++ b/go.mod
@@ -80,7 +80,7 @@ require (
 	golang.org/x/exp/typeparams v0.0.0-20230522175609-2e198f4a06a1 // indirect
 	golang.org/x/time v0.3.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
-	gvisor.dev/gvisor v0.0.0-20230603040744-5c9219dedd33 // indirect
+	gvisor.dev/gvisor v0.0.0-20231006032704-15cc3fcbbd77 // indirect
 )

 require (
@@ -89,7 +89,7 @@ require (
 	github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 // indirect
 	github.com/armon/go-proxyproto v0.0.0-20210323213023-7e956b284f0a // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
-	github.com/bifurcation/mint v0.0.0-20180306135233-198357931e61 // indirect
+	github.com/bifurcation/mint v0.0.0-20210616192047-fd18df995463 // indirect
 	github.com/cespare/xxhash/v2 v2.2.0 // indirect
 	github.com/cheekybits/genny v1.0.0 // indirect
 	github.com/cognusion/go-cache-lru v0.0.0-20170419142635-f73e2280ecea // indirect
@@ -132,7 +132,7 @@ require (
 	github.com/prometheus/common v0.44.0 // indirect
 	github.com/prometheus/procfs v0.12.0 // indirect
 	github.com/refraction-networking/gotapdance v1.7.4 // indirect
-	github.com/refraction-networking/utls v1.3.3 // indirect
+	github.com/refraction-networking/utls v1.5.3 // indirect
 	github.com/sergeyfrolov/bsbuffer v0.0.0-20180903213811-94e85abb8507 // indirect
 	github.com/sirupsen/logrus v1.9.3 // indirect
 	github.com/spf13/cobra v1.7.0
@@ -142,10 +142,10 @@ require (
 	github.com/templexxx/xorsimd v0.4.2 // indirect
 	github.com/tjfoc/gmsm v1.4.1 // indirect
 	github.com/wader/filtertransport v0.0.0-20200316221534-bdd9e61eee78 // indirect
-	github.com/xtaci/kcp-go/v5 v5.6.2 // indirect
+	github.com/xtaci/kcp-go/v5 v5.6.3 // indirect
 	github.com/xtaci/smux v1.5.24 // indirect
 	gitlab.com/yawning/bsaes.git v0.0.0-20190805113838-0a714cd429ec // indirect
-	golang.org/x/mod v0.12.0 // indirect
+	golang.org/x/mod v0.13.0 // indirect
 	golang.org/x/term v0.13.0 // indirect
 	golang.org/x/text v0.13.0 // indirect
 	golang.org/x/tools v0.13.0 // indirect

We cannot upgrade:

  • gvisor because (a) we're fine with our pinned version used by netem and (b) the main branch of gvistor is not suitable for go-getting anyway

  • mint because this is the version Psiphon requires

  • utls because this is the version Psiphon requires

  • kcp-go because v.5.6.3 requires go1.21

  • x/mod which seems to be needed by quic-go to generate mocks, so not upgrading it isn't a big deal.

This diff upgrades obfs4, goptlib, and snowflake. I tried to import
lyrebird instead of obfs4, but that did not work as intended. It seems
tags aren't compatible with the format expected by Go (vX.Y.Z) and I
did not want to pin to the latest commit. I will report this issue
to lyrebird developers later today.

While there, also upgrade x/net and x/exp.

This work is part of ooni/probe#2524.

Here's what we could not upgrade (based on the content of `go.mod`
after we committed this diff and ran `go get -u -v -d ./...`):

```diff
diff --git a/go.mod b/go.mod
index 67e0af02..18175d73 100644
--- a/go.mod
+++ b/go.mod
@@ -80,7 +80,7 @@ require (
 	golang.org/x/exp/typeparams v0.0.0-20230522175609-2e198f4a06a1 // indirect
 	golang.org/x/time v0.3.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
-	gvisor.dev/gvisor v0.0.0-20230603040744-5c9219dedd33 // indirect
+	gvisor.dev/gvisor v0.0.0-20231006032704-15cc3fcbbd77 // indirect
 )

 require (
@@ -89,7 +89,7 @@ require (
 	github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 // indirect
 	github.com/armon/go-proxyproto v0.0.0-20210323213023-7e956b284f0a // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
-	github.com/bifurcation/mint v0.0.0-20180306135233-198357931e61 // indirect
+	github.com/bifurcation/mint v0.0.0-20210616192047-fd18df995463 // indirect
 	github.com/cespare/xxhash/v2 v2.2.0 // indirect
 	github.com/cheekybits/genny v1.0.0 // indirect
 	github.com/cognusion/go-cache-lru v0.0.0-20170419142635-f73e2280ecea // indirect
@@ -132,7 +132,7 @@ require (
 	github.com/prometheus/common v0.44.0 // indirect
 	github.com/prometheus/procfs v0.12.0 // indirect
 	github.com/refraction-networking/gotapdance v1.7.4 // indirect
-	github.com/refraction-networking/utls v1.3.3 // indirect
+	github.com/refraction-networking/utls v1.5.3 // indirect
 	github.com/sergeyfrolov/bsbuffer v0.0.0-20180903213811-94e85abb8507 // indirect
 	github.com/sirupsen/logrus v1.9.3 // indirect
 	github.com/spf13/cobra v1.7.0
@@ -142,10 +142,10 @@ require (
 	github.com/templexxx/xorsimd v0.4.2 // indirect
 	github.com/tjfoc/gmsm v1.4.1 // indirect
 	github.com/wader/filtertransport v0.0.0-20200316221534-bdd9e61eee78 // indirect
-	github.com/xtaci/kcp-go/v5 v5.6.2 // indirect
+	github.com/xtaci/kcp-go/v5 v5.6.3 // indirect
 	github.com/xtaci/smux v1.5.24 // indirect
 	gitlab.com/yawning/bsaes.git v0.0.0-20190805113838-0a714cd429ec // indirect
-	golang.org/x/mod v0.12.0 // indirect
+	golang.org/x/mod v0.13.0 // indirect
 	golang.org/x/term v0.13.0 // indirect
 	golang.org/x/text v0.13.0 // indirect
 	golang.org/x/tools v0.13.0 // indirect
```

We cannot upgrade:

- gvisor because (a) we're fine with our pinned version used by netem
and (b) the main branch of gvistor is not suitable for go-getting anyway

- mint because this is the version Psiphon requires

- utls because this is the version Psiphon requires

- kcp-go because v.5.6.3 requires go1.21

- x/mod which seems to be needed by quic-go to generate mocks, so
not upgrading it isn't a big deal.
@bassosimone bassosimone requested a review from hellais as a code owner October 6, 2023 06:44
@bassosimone bassosimone changed the title chore: upgrade obfs4, goptlib, snowflake chore: update obfs4, goptlib, snowflake Oct 6, 2023
@bassosimone bassosimone mentioned this pull request Oct 6, 2023
26 tasks
@bassosimone bassosimone merged commit ac1563b into master Oct 6, 2023
6 checks passed
@bassosimone bassosimone deleted the issue/2524 branch October 6, 2023 06:52
@bassosimone
Copy link
Contributor Author

bassosimone commented Oct 6, 2023

Uhm, before I wrote:

I tried to import lyrebird instead of obfs4, but that did not work as intended. It seems tags aren't compatible with the format expected by Go (vX.Y.Z) and I did not want to pin to the latest commit.

However, this sentence is inaccurate.

Here's what happens. First, we need to create a new project:

$ mkdir example
$ cd example
$ go mod init example.com/lyrebird

Then, we create the following main.go file:

package main

import "gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird"

func main() {
	txp := &lyrebird.Transport{}
	_ = txp
}

Then we run:

$ go mod tidy -x
go: finding module for package gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird
# get https://proxy.golang.org/gitlab.torproject.org/@v/list
# get https://proxy.golang.org/gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/@v/list
# get https://proxy.golang.org/gitlab.torproject.org/tpo/anti-censorship/@v/list
# get https://proxy.golang.org/gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird/@v/list
# get https://proxy.golang.org/gitlab.torproject.org/tpo/@v/list
# get https://proxy.golang.org/gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/@v/list: 404 Not Found (0.151s)
# get https://proxy.golang.org/gitlab.torproject.org/tpo/@v/list: 404 Not Found (0.188s)
# get https://proxy.golang.org/gitlab.torproject.org/@v/list: 404 Not Found (0.188s)
# get https://proxy.golang.org/gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird/@v/list: 200 OK (0.190s)
# get https://proxy.golang.org/gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird/@latest
# get https://proxy.golang.org/gitlab.torproject.org/tpo/anti-censorship/@v/list: 404 Not Found (0.191s)
# get https://proxy.golang.org/gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird/@latest: 200 OK (0.046s)
example.com/lyrebird imports
	gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird: module gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird@latest found (v0.0.0-20231005141435-20cf093f50ae), but does not contain package gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird

The commit in there is the latest commit as of today. Hence the issue is not related with tags.

Running go build -v ./... from the source tree of lyrebird is WAI.

What am I missing?

Ah, so the issue seems to be the package name that I am importing.

This main.go file:

package main

import "gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird/transports/obfs4"

func main() {
	txp := &obfs4.Transport{}
	_ = txp
}

Leads to:

$ go mod tidy
go: finding module for package gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird/transports/obfs4
go: found gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird/transports/obfs4 in gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird v0.0.0-20231005141435-20cf093f50ae

Hence, it turns out we had a case of PEBCAK 🤦

I'll make a new PR to switch to lyrebird!

bassosimone added a commit that referenced this pull request Oct 6, 2023
This diff replaces obfs4 and uses lyrebird instead. See
#1347 (comment)
to understand why my previous assessment that we could not import
lyrebird was actually caused by PEBCAK.

The build currently fails because the set of dependencies required by
Psiphon conflict with lyrebird ones, so upgrading isn't possible.

Here is how the build fails:

```console
$ go build -v ./...
github.com/Psiphon-Labs/psiphon-tunnel-core/psiphon/common/protocol
/home/sbs/go/pkg/mod/github.com/!psiphon-!labs/[email protected]/psiphon/common/protocol/customTLSProfiles.go:180:16: undefined: utls.UtlsExtendedMasterSecretExtension
```

So, I think the underlying issue here may be that utls has removed
some symbols without bumping their major version number.

If it was possible to merge this work, this work would be part of
the ooni/probe#2524 issue.
Murphy-OrangeMud pushed a commit to Murphy-OrangeMud/probe-cli that referenced this pull request Feb 13, 2024
This diff upgrades obfs4, goptlib, and snowflake. I tried to import
lyrebird instead of obfs4, but that did not work as intended. It seems
tags aren't compatible with the format expected by Go (vX.Y.Z) and I did
not want to pin to the latest commit. I will report this issue to
lyrebird developers later today.

While there, also upgrade x/net and x/exp.

This work is part of ooni/probe#2524.

Here's what we could not upgrade (based on the content of `go.mod` after
we committed this diff and ran `go get -u -v -d ./...`):

```diff
diff --git a/go.mod b/go.mod
index 67e0af02..18175d73 100644
--- a/go.mod
+++ b/go.mod
@@ -80,7 +80,7 @@ require (
 	golang.org/x/exp/typeparams v0.0.0-20230522175609-2e198f4a06a1 // indirect
 	golang.org/x/time v0.3.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
-	gvisor.dev/gvisor v0.0.0-20230603040744-5c9219dedd33 // indirect
+	gvisor.dev/gvisor v0.0.0-20231006032704-15cc3fcbbd77 // indirect
 )

 require (
@@ -89,7 +89,7 @@ require (
 	github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 // indirect
 	github.com/armon/go-proxyproto v0.0.0-20210323213023-7e956b284f0a // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
-	github.com/bifurcation/mint v0.0.0-20180306135233-198357931e61 // indirect
+	github.com/bifurcation/mint v0.0.0-20210616192047-fd18df995463 // indirect
 	github.com/cespare/xxhash/v2 v2.2.0 // indirect
 	github.com/cheekybits/genny v1.0.0 // indirect
 	github.com/cognusion/go-cache-lru v0.0.0-20170419142635-f73e2280ecea // indirect
@@ -132,7 +132,7 @@ require (
 	github.com/prometheus/common v0.44.0 // indirect
 	github.com/prometheus/procfs v0.12.0 // indirect
 	github.com/refraction-networking/gotapdance v1.7.4 // indirect
-	github.com/refraction-networking/utls v1.3.3 // indirect
+	github.com/refraction-networking/utls v1.5.3 // indirect
 	github.com/sergeyfrolov/bsbuffer v0.0.0-20180903213811-94e85abb8507 // indirect
 	github.com/sirupsen/logrus v1.9.3 // indirect
 	github.com/spf13/cobra v1.7.0
@@ -142,10 +142,10 @@ require (
 	github.com/templexxx/xorsimd v0.4.2 // indirect
 	github.com/tjfoc/gmsm v1.4.1 // indirect
 	github.com/wader/filtertransport v0.0.0-20200316221534-bdd9e61eee78 // indirect
-	github.com/xtaci/kcp-go/v5 v5.6.2 // indirect
+	github.com/xtaci/kcp-go/v5 v5.6.3 // indirect
 	github.com/xtaci/smux v1.5.24 // indirect
 	gitlab.com/yawning/bsaes.git v0.0.0-20190805113838-0a714cd429ec // indirect
-	golang.org/x/mod v0.12.0 // indirect
+	golang.org/x/mod v0.13.0 // indirect
 	golang.org/x/term v0.13.0 // indirect
 	golang.org/x/text v0.13.0 // indirect
 	golang.org/x/tools v0.13.0 // indirect
```

We cannot upgrade:

- gvisor because (a) we're fine with our pinned version used by netem
and (b) the main branch of gvistor is not suitable for go-getting anyway

- mint because this is the version Psiphon requires

- utls because this is the version Psiphon requires

- kcp-go because v.5.6.3 requires go1.21

- x/mod which seems to be needed by quic-go to generate mocks, so not
upgrading it isn't a big deal.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant