git.apache.org down, use github for mods

[mikecook]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" comments, they generate extra noise for pull request followers and do not help prioritize the request

Relates hashicorp/terraform#22664

[bflad]

Hi @mikecook 👋 Can you confirm that setting the GOPROXY=https://proxy.golang.org environment variable also works around this issue?

[gcase555]

Its worth mentioning that the official documentation points people to pull the dependency from github, not the apache mirror https://thrift.apache.org/lib/go#using-thrift-with-go

@bflad I think this approach is more aligned with what is officially recommended

[mikecook]

@bflad If you're using go modules then there's no good solution without this patch.

go1.12.9 GO111MODULE=on GOPROXY="https://proxy.golang.org" make build doesn't work. (410)
go1.12.9 GO111MODULE=on GOPROXY="https://proxy.golang.org,direct" make build doesn't work. (cannot have comma)
go1.13.0 GO111MODULE=on GOPROXY="https://proxy.golang.org,direct" make build doesn't work. (golangci-lint deps do not match version-control timestamp)

Deepdive below

Note: Output is dockerized for repeatability, scroll right to see the full command lines to duplicate. All use GO111MODULE=on

go1.12.9 (the latest 1.12.x) does not (and will not) support a goproxy comma delimited list.
go1.13.0 (the latest 1.13.x) does support a goproxy comma delimited list.
golang/go#26334 (where they add goproxy list support for 1.13.x and confirm they will not add goproxy list support for 1.12.x)

Without a comma delimited list that contains direct, as soon as https://proxy.golang.org/ responds 404/410 for any dep that it doesn't have yet, a build will fail.

go1.12.9, aws provider v2.26.0 without patch, GOPROXY="https://proxy.golang.org"

$ docker run --rm -it -v `pwd`:/go/src/github.com/terraform-providers/terraform-provider-aws -w=/go/src/github.com/terraform-providers/terraform-provider-aws golang:1.12.9-alpine3.10 sh -c "apk add --no-cache make git bash; set -x ; git clean -fx ; git checkout -f v2.26.0 ; export GO111MODULE=on ; export GOPROXY="https://proxy.golang.org" ; go env | grep "GOPROXY" ; make build"
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
<snip>
OK: 30 MiB in 26 packages
+ git clean -fx
+ git checkout -f v2.26.0
HEAD is now at f3ad9853b v2.26.0
+ export 'GO111MODULE=on'
+ export 'GOPROXY=https://proxy.golang.org'
+ grep GOPROXY
+ go env
GOPROXY="https://proxy.golang.org"
+ make build
==> Checking that code complies with gofmt requirements...
go install
go: finding github.com/kubernetes-sigs/aws-iam-authenticator v0.3.1-0.20181019024009-82544ec86140
go: finding github.com/golangci/gosec v0.0.0-20180901114220-8afd9cbb6cfb
<snip>
go: finding golang.org/x/net v0.0.0-20190213061140-3a22650c66bd
go: github.com/terraform-providers/[email protected]+incompatible: unexpected status (https://proxy.golang.org/github.com/terraform-providers/terraform-provider-template/@v/v2.1.2+incompatible.info): 410 Gone
go: error loading module requirements
make: *** [GNUmakefile:10: build] Error 1

With a comma delimited list (just to see what happens), on go1.12.x, the build will fail: cannot have comma.

go1.12.9, aws provider v2.26.0 without patch, GOPROXY="https://proxy.golang.org,direct"

$ docker run --rm -it -v `pwd`:/go/src/github.com/terraform-providers/terraform-provider-aws -w=/go/src/github.com/terraform-providers/terraform-provider-aws golang:1.12.9-alpine3.10 sh -c "apk add --no-cache make git bash; set -x ; git clean -fx ; git checkout -f v2.26.0 ; export GO111MODULE=on ; export GOPROXY="https://proxy.golang.org,direct" ; go env | grep "GOPROXY" ; make build"
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
<snip>
OK: 30 MiB in 26 packages
+ git clean -fx
+ git checkout -f v2.26.0
HEAD is now at f3ad9853b v2.26.0
+ export 'GO111MODULE=on'
+ export 'GOPROXY=https://proxy.golang.org,direct'
+ go env
+ grep GOPROXY
GOPROXY="https://proxy.golang.org,direct"
+ make build
==> Checking that code complies with gofmt requirements...
go install
go: github.com/hashicorp/[email protected]: invalid $GOPROXY setting: cannot have comma
<snip>
go: github.com/terraform-providers/[email protected]+incompatible: invalid $GOPROXY setting: cannot have comma
go: error loading module requirements
make: *** [GNUmakefile:10: build] Error 1

In go1.13.0 "The default setting for GOPROXY is "https://proxy.golang.org,direct", which means to try the Go module mirror run by Google and fall back to a direct connection if the proxy reports that it does not have the module (HTTP error 404 or 410)." (quoted from: https://tip.golang.org/cmd/go/#hdr-Module_downloading_and_verification )

But go1.13.x goproxy can't seem to manage to work for golangci-lint golangci/golangci-lint#595 (Issue has pivoted over the months from 410 to does not match version-control timestamp)

go1.13.0, aws provider v2.26.0 without patch, GOPROXY="https://proxy.golang.org,direct"

$ docker run --rm -it -v `pwd`:/go/src/github.com/terraform-providers/terraform-provider-aws -w=/go/src/github.com/terraform-providers/terraform-provider-aws golang:1.13.0-alpine3.10 sh -c "apk add --no-cache make git bash; set -x ; git clean -fx ; git checkout -f v2.26.0 ; export GO111MODULE=on  ; go env | grep GOPROXY ; make build"
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
<snip>
OK: 30 MiB in 26 packages
+ git clean -fx
+ git checkout -f v2.26.0
HEAD is now at f3ad9853b v2.26.0
+ export 'GO111MODULE=on'
+ go env
+ grep GOPROXY
GOPROXY="https://proxy.golang.org,direct"
+ make build
==> Checking that code complies with gofmt requirements...
go install
go: github.com/golangci/[email protected] requires
	github.com/go-critic/[email protected]: invalid pseudo-version: does not match version-control timestamp (2019-05-26T07:48:19Z)
make: *** [GNUmakefile:10: build] Error 1

Which leaves no good option besides this patch to replace git.apache.org with apache's github.

[bflad]

Regarding Go 1.13, we do not currently support that version yet for development and testing with this project, especially due to the the golangci-lint situation. See #9992 for updates there. 👍 We are hoping they fix their Go module and cut a bug fix release within a reasonable timeframe before attempting any workarounds in our Go modules setup.

One important note here is that our Makefile does not currently force usage of -mod=vendor even though we have everything vendored in the repository. Given that, setting GOFLAGS=-mod=vendor may also help in your build environment so it uses the local vendored files instead of reaching out at all.

[bflad]

This situation should be resolved through a chain of dependency updates starting with hashicorp/go-getter#204 then hashicorp/terraform then here. 😄

[bflad]

Hi folks 👋 Apologies for the delayed response here -- the transitive dependency on thrift (git.apache.org/thrift and github.com/apache/thrift) was removed in #10367. Thank you for contributing this when it was really problematic before.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!