doc, x/tools/cmd/godoc, x/website: move golang.org content into its own repository #29206

andybons · 2018-12-13T02:41:01Z

With the website slated for a refresh, we should move the static content out of the main go source tree (https://go.googlesource.com/go/+/master/doc/) and into its own repository (web? website?).

This will reduce binary and source code download sizes (#27151) and allow us to apply changes unrelated to our release schedule without having to cherry-pick them to a release branch. Hopefully we can also implement continuous delivery as part of this as well.

We also won’t have to worry about bloating the main go source tree as we add more assets and content for the refresh.

We still only want to display package documentation and source files from a GOROOT pinned to the current stable release (and not show content only relevant for future releases like release notes). So there are elements that still require logic that ties content to a specific Go version. What we consider fluid vs tied to a release is to be determined and can be fleshed out here.

/cc @bradfitz @ianlancetaylor @julieqiu @katiehockman @dmitshur @FiloSottile @cnoellekb

The text was updated successfully, but these errors were encountered:

bradfitz · 2018-12-13T03:15:43Z

"website" SGTM. I like the repo name golang/website.

Inside that I'd like something like:

README.md
codereview.cfg
content/ directory
cmd/ directory
cmd/goweb or cmd/golangweb or cmd/golang-org-server (name TBD)

We could start with mirroring stuff over from the existing golang/go repo (and any necessary code from x/tools/cmd/godoc) but without modifying either for now. And then serve the new one on something like https://new.golang.org for a bit and when we're happy then switch over and start deleting code & assets from the old locations.

We also need to figure out whether this new cmd/golang-org-server is the new name of the recently-web-only x/tools/cmd/godoc (the tool for users to browse their local $GOPATH/etc documentation in the web on localhost on airplanes/etc).

If they're the same, then do we develop it in x/tools or in x/website? I kinda like having the code next to the assets but I could go either way and haven't thought about it much.

andybons · 2018-12-13T03:36:17Z

We also need to figure out whether this new cmd/golang-org-server is the new name of the recently-web-only x/tools/cmd/godoc (the tool for users to browse their local $GOPATH/etc documentation in the web on localhost on airplanes/etc).

My preference is for x/tools/cmd/godoc to become a binary for viewing local $GOPATH/etc documentation but not anything else (which would be more in line with its name anyhow). A common go-source-to-html rendering library could be pulled out (if it’s not already general enough in x/tools/godoc) and used in places like godoc.org, x/tools/cmd/godoc, and the new website. This way most of the logic specific to new.golang.org is right next to the content.

dmitshur · 2018-12-13T03:36:41Z

We also need to figure out whether this new cmd/golang-org-server is the new name of the recently-web-only x/tools/cmd/godoc (the tool for users to browse their local $GOPATH/etc documentation in the web on localhost on airplanes/etc).

Yes, we need to figure that out. At this time, I strongly suspect that we’ll want to make the new cmd/golang-org-server be strictly the website and not a tool users run locally, other than to contribute to the golang.org website development.

bradfitz · 2018-12-13T03:42:48Z

Pulling the render-godoc-as-HTML handler out and using it on both places SGTM. That can happen first probably.

And making x/tools/cmd/godoc be super minimal also SGTM.

cnoellekb · 2018-12-20T21:02:07Z

We discussed 4 main milestones for moving the website:

Separate out rendering logic (converting code -> HTML) from x/tools/cmd/godoc. There should be no behavior change at this point.
Creating website binary and serving templates (the static content) from go/doc to x/website
Using the common rendering library in x/website.
Reduce x/tools/cmd/godoc to be the minimal version (since the website can now be run out of x/website).

Right now I'm focusing on the second step because I was able to get a prototype up and running. I copied tools/cmd/godoc, tools/godoc, and go/doc into a local directory that will eventually become the root for the main website. The plan is to just copy everything for now to maintain x/tools functionality. Can someone help me set up an x/website repo?

Also, I was looking into how the rest of the tools directory would be affected by removing any godoc files and I found that x/tools/cmd/present/play.go uses the tools/godoc/static package. Does anyone think this may be an issue?

Lastly, is the final version of godoc going to need an app engine configuration? If not, then tools/internal/memcache can be moved to the x/website repository.

bradfitz · 2018-12-21T17:49:48Z

@dmitshur, @andybons, where are the notes from last time we created a new repo? I'd rather not do it myself. It'd be nice if we can have somebody else run through the steps from the docs. (gerrit, github, maintner, gitmirror updating, etc)

dmitshur · 2018-12-21T21:46:38Z

where are the notes from last time we created a new repo?

@bradfitz The steps are documented at golang.org/wiki/CreatingSubRepository.

It'd be nice if we can have somebody else run through the steps from the docs.

Yep, that's a good idea. If anyone on the team wants to go through the process to learn it, I'm happy to assist and fill in any gaps in docs. Note that most of the bottlenecks will be having the right permissions to all components involved. I can also do it myself.

bradfitz · 2019-01-02T22:03:25Z

I created https://github.com/golang/website and https://go.googlesource.com/website (Gerrit).

@cnoellekb, can you send CLs for the maintner and dashboard and gitmirror bits? And gopherbot if needed.

cnoellekb · 2019-01-03T16:31:56Z

Sure, I'll take care of it!

gopherbot · 2019-01-03T18:18:20Z

Change https://golang.org/cl/156217 mentions this issue: maintner/maintnerd, cmd/gerritbot, cmd/gitmirror: add golang/website to

…to whitelist The website now has its own subrepo with gerrit mirroring set up. Updates golang/go#29206 Change-Id: I6fb6fb62dfd50b48d2f78db2503641c521600ae7 Reviewed-on: https://go-review.googlesource.com/c/156217 Reviewed-by: Brad Fitzpatrick <[email protected]> Reviewed-by: Katie Hockman <[email protected]>

gopherbot · 2019-01-04T16:58:45Z

Change https://golang.org/cl/156337 mentions this issue: cmd/godoc: add x/website redirect

Add a redirect for the recently created x/website subrepository. It's not yet included at https://golang.org/pkg/#subrepo because it's in development. Once development reaches the point that x/website is the canonical location of the golang.org server, we can consider including x/website at https://golang.org/pkg/#subrepo (just like x/blog, x/tour, x/build are already included there). Updates golang/go#29206 Change-Id: I6889c1f5e40f11abca944b217a7354f76c08c8eb Reviewed-on: https://go-review.googlesource.com/c/156337 Reviewed-by: Brad Fitzpatrick <[email protected]> Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Gobot Gobot <[email protected]>

gopherbot · 2019-01-04T19:13:26Z

Change https://golang.org/cl/156321 mentions this issue: x/website: add code and static files for website

gopherbot · 2019-01-04T21:55:51Z

Change https://golang.org/cl/156326 mentions this issue: x/website: add code and static files for website

All of the code and static assets that the website uses to run have been copied to this repo. There was also a few lines of code added telling the website where the doc directory, favicon.ico and robots.txt are. go repo change-id: Ife6443c32673b38000b90dd2efb2985db37ab773 x/tools repo change-id: Ia979a8b06d1b4db47d25ffdfdf925ba8a0ac67de Real new code additions: - main.go * lines 89-95 added getFullPath method * lines 217-222 mapped paths to doc/, favicon.ico, robots.txt in vfs - appinit.go * lines 147-153 added getFullPath method * lines 80-84 mapped paths to doc/, favicon.ico in vfs Several files were copied from x/tools and go so paths (and corresponding import paths) were changed as follows: "x/tools/cmd/godoc/" --> "x/website/cmd/golangorg/" "x/tools/godoc/static/" --> "x/website/content/static/" "x/tools/godoc/" (without godoc/static/) --> "x/website/cmd/golangorg/godoc/" "x/tools/internal/memcache" --> "x/website/internal/memcache" "go/doc/" --> "x/website/content/doc/" "go/favicon.ico" --> "x/website/favicon.ico" "go/robots.txt" --> "x/website/robots.txt" Updates golang/go#29206 Change-Id: I53985fc027f73e60c6946038f85133acf1ecb08c Reviewed-on: https://go-review.googlesource.com/c/156321 Reviewed-by: Brad Fitzpatrick <[email protected]>

cnoellekb · 2019-01-09T16:16:17Z

Some decisions that have been made over the last few days:

I am currently prioritizing turning shared documentation-rendering logic between x/website and the godoc binary into a package, maybe it will live in a directory called "unstable" in x/tools?
Deployment of the website will come later when godoc has been properly stripped down with all non-documentation stuff ready to be deleted from x/tools
In the long term @dmitshur plans to expand the package to support rendering documentation for modules

katiehockman · 2019-01-09T18:56:38Z

I am currently prioritizing turning shared documentation-rendering logic between x/website and the godoc binary into a package, maybe it will live in a directory called "unstable" in x/tools?

This sounds good. Perhaps under x/tools/godoc/unstable since it will be for go documentation.

bcmills · 2019-01-09T20:22:12Z

I wouldn't bother with the /unstable suffix: just put a prominent note in the package comment explaining that it's unstable.

Actually, better still, could we start a new repo with the path golang.org/x/internal? Then the Go tool's visibility rules would enforce that it is only used within other golang.org/x modules.

dmitshur · 2019-01-09T22:14:30Z

We already have the golang.org/x/tools/godoc package, which is documented as:

Package godoc is a work-in-progress (2013-07-17) package to begin splitting up the godoc binary into multiple pieces.

This package comment will evolve over time as this package splits into smaller pieces.

I think this existing package is a good candidate to move code that we want to share between x/tools/cmd/godoc (for viewing documentation) and x/website/cmd/golangorg (for serving golang.org) in the short term.

I like the idea of x/internal in general, because if it existed, we could use it in other instances to do similar refactors between subrepos. However, I think that should be a separate proposal. Also, golang.org/x/internal would preclude golang.org/dl from being able to use it, so maybe it should be golang.org/internal.

gopherbot · 2019-01-25T22:23:38Z

Change https://golang.org/cl/159737 mentions this issue: x/website: remove godoc package/make module aware

gopherbot · 2019-01-28T21:05:48Z

Change https://golang.org/cl/159897 mentions this issue: website: use modules

gopherbot · 2019-12-13T16:22:31Z

Change https://golang.org/cl/211300 mentions this issue: content/static/doc: copy Go Security Policy page from Go repo

gopherbot · 2019-12-13T16:29:45Z

Change https://golang.org/cl/211301 mentions this issue: doc: remove Go Security Policy page (moved to x/website)

The Go Security Policy page is copied from Go master as of 2019/12/13 (commit golang/go@0497f91, CL 211177). This page not closely tied to specific Go releases, and thus is better suited to live in the x/website repo. Updates golang/go#29206 Change-Id: Ib23fa6ccd11218d5da231cca685f62e610d5c6c9 Reviewed-on: https://go-review.googlesource.com/c/website/+/211300 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Filippo Valsorda <[email protected]>

This page has moved to the x/website repo in CL 211300 (commit golang/website@3c8b7f9). Remove the old copy in this repo since it's no longer used. Updates #29206 Change-Id: I8b3396d9e42d1e7262a8cde9577962d33b215836 Reviewed-on: https://go-review.googlesource.com/c/go/+/211301 Reviewed-by: Filippo Valsorda <[email protected]>

gopherbot · 2020-01-02T23:09:29Z

Change https://golang.org/cl/213157 mentions this issue: doc: remove root.html and conduct.html (moved to x/website)

root.html has been moved to the x/website repo in CL 180959 (commit golang/website@d83058c). conduct.html has been moved to the x/website repo in CL 207437 (commit golang/website@99763cb). There should be only one copy, otherwise it may lead to confusion, or changes made in the wrong place. This CL removes the old copies from this repo since they're no longer used. Updates #29206 Change-Id: I41adfb2c34ed3d870fb7a671f48ccc8f90863feb Reviewed-on: https://go-review.googlesource.com/c/go/+/213157 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Jay Conrod <[email protected]>

gopherbot · 2020-01-08T23:23:23Z

Change https://golang.org/cl/213937 mentions this issue: content/static: add gopath_code.html file and regenerate

CL 199417 added a new doc/gopath_code.html file. This change adds it to the list of files that are embedded in package static, and regenerates the package. A change described in https://golang.org/issue/29206#issuecomment-536099768 will make it no longer neccessary to do this step in the future. Updates golang/go#28215 Updates golang/go#29206 Change-Id: Ia1bd125488e04b34f9b8e36de4397de32d77bcef Reviewed-on: https://go-review.googlesource.com/c/website/+/213937 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Jay Conrod <[email protected]>

gopherbot · 2020-04-20T21:03:45Z

Change https://golang.org/cl/229081 mentions this issue: cmd/golangorg: generate release history page from internal/history

Previously, the release history page was a raw HTML file that was manually edited whenever new Go releases were made. This change converts release history entries into a structured format in the new internal/history package, and generates release history entries from that format. For now, only Go 1.9 and newer releases are converted, but the structured format is flexible enough to represent all releases going back to the original Go 1 release. Various English grammar rules and special cases are preserved, so that the release history entries appear in a consistent way. New release history entries need only to be added to the internal/ history package, making it so that English grammar rules and HTML tags don't need to go through human code review for each release. Future work may involve constructing that list from data already available in the Go issue tracker. This change makes minimal contributions to reducing the dependence of x/website on the x/tools/godoc rendering engine for displaying pages other than Go package documentation. The x/tools/godoc code is in another module and does not provide flexibility desired for the general purpose website needs of x/website. Fixes golang/go#38488. For golang/go#37090. For golang/go#29206. Change-Id: I80864e4f218782e6e3b5fcd5a1d63f3699314c81 Reviewed-on: https://go-review.googlesource.com/c/website/+/229081 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Alexander Rakoczy <[email protected]>

gopherbot · 2020-04-22T22:23:13Z

Change https://golang.org/cl/229483 mentions this issue: cmd/golangorg: generate major version list on Go project page

gopherbot · 2020-04-22T22:23:15Z

Change https://golang.org/cl/229482 mentions this issue: content/static/doc: copy The Go Project page (contrib.html) from Go repo

gopherbot · 2020-04-22T23:33:30Z

Change https://golang.org/cl/229485 mentions this issue: doc: remove The Go Project page (moved to x/website)

The Go Project page is copied from Go master as of 2020/04/24 (commit golang/go@ee8972c, CL 209679). This page not closely tied to specific Go releases, and thus is better suited to live in the x/website repo. It will be modified in CL 229483 to use the release history data from the internal/history package, rather than using hand-written HTML which needs to be updated manually after each major Go release. For golang/go#29205. For golang/go#29206. Change-Id: I6a6fa734fb3d873da59d67d796616fc53d684610 Reviewed-on: https://go-review.googlesource.com/c/website/+/229482 Reviewed-by: Alexander Rakoczy <[email protected]>

This page has moved to the x/website repo in CL 229482 (commit golang/website@70f4ee8). Remove the old copy in this repo since it's no longer used. For #29206. Change-Id: Ief093ed8c5dfec43e06d473e4282275f61da74a5 Reviewed-on: https://go-review.googlesource.com/c/go/+/229485 Reviewed-by: Alexander Rakoczy <[email protected]>

This change builds on what was done in CL 229081, and uses the Go release history data from internal/history package to generate the list of major Go versions on the Go project page. This way, this page doesn't need to be manually edited when major Go releases are made. For golang/go#38488. For golang/go#29205. For golang/go#29206. Change-Id: Ie0b12707d828207173a54f0a1bc6a4ef69dcedef Reviewed-on: https://go-review.googlesource.com/c/website/+/229483 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Alexander Rakoczy <[email protected]>

This page has moved to the x/website repo in CL 229482 (commit golang/website@70f4ee8). Remove the old copy in this repo since it's no longer used. For golang#29206. Change-Id: Ief093ed8c5dfec43e06d473e4282275f61da74a5 Reviewed-on: https://go-review.googlesource.com/c/go/+/229485 Reviewed-by: Alexander Rakoczy <[email protected]>

gopherbot · 2020-07-16T19:09:40Z

Change https://golang.org/cl/243118 mentions this issue: internal/dl: add CORS support to JSON endpoint

This change allows users to request golang.org/dl/?mode=json via Cross-Origin Resource Sharing (CORS). It also removes the golangorg build tag as it did not seem necessary and adds tests for the “include” GET parameter. Updates golang/go#29206 Fixes golang/go#40253 Change-Id: I5306a264c4ac2a6e6f49cfb53db01eef6b7f4473 Reviewed-on: https://go-review.googlesource.com/c/website/+/243118 Run-TryBot: Andrew Bonventre <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Alexander Rakoczy <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>

dmitshur · 2020-08-28T18:24:20Z

I think this can be closed. The golang.org website and most of its content has been factored out into the x/website repository, and it's been in production for a long while. There's more that we can optionally deal with in the future, but there's nothing critical left.

I've factored out some simplification opportunities from #29206 (comment) into a separate issue #41102.

If there's anything more that needs to be done, let's open individual issues.

Thanks to everyone who contributed to this successful move!

andybons added Documentation NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Dec 13, 2018

andybons added this to the Go1.13 milestone Dec 13, 2018

andybons assigned cnoellekb Dec 13, 2018

dmitshur mentioned this issue Jan 4, 2019

x/tools/cmd/godoc: add x/website redirect [1.11 backport] #29564

Closed

This was referenced Jan 22, 2019

x/build/cmd/tip: add _tipstatus error monitoring to farmer.golang.org#health #15266

Closed

x/website/internal/dl: store the list of validUsers in a more dynamic way #29898

Open

x/build: reconsider the large number of third-party dependencies #29935

Closed

This was referenced Dec 13, 2019

x/website/_content: consider reusing repo list from x/build/repos instead of maintaining own hard-coded copy #36136

Open

x/build: unify list, configuration of repos, simplifying add new ones #36047

Closed

dmitshur mentioned this issue Aug 28, 2020

x/website/cmd/golangorg: various open simplification opportunities after move from x/tools/cmd/godoc #41102

Closed

dmitshur closed this as completed Aug 28, 2020

dmitshur removed the Documentation label Aug 28, 2020

dmitshur modified the milestones: Backlog, Unreleased Aug 28, 2020

golang locked and limited conversation to collaborators Aug 28, 2021

gopherbot added the FrozenDueToAge label Aug 28, 2021

rsc unassigned dmitshur Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

doc, x/tools/cmd/godoc, x/website: move golang.org content into its own repository #29206

doc, x/tools/cmd/godoc, x/website: move golang.org content into its own repository #29206

andybons commented Dec 13, 2018

bradfitz commented Dec 13, 2018 •

edited by dmitshur

Loading

andybons commented Dec 13, 2018 •

edited

Loading

dmitshur commented Dec 13, 2018 •

edited

Loading

bradfitz commented Dec 13, 2018

cnoellekb commented Dec 20, 2018

bradfitz commented Dec 21, 2018

dmitshur commented Dec 21, 2018 •

edited

Loading

bradfitz commented Jan 2, 2019

cnoellekb commented Jan 3, 2019

gopherbot commented Jan 3, 2019

gopherbot commented Jan 4, 2019

gopherbot commented Jan 4, 2019

gopherbot commented Jan 4, 2019

cnoellekb commented Jan 9, 2019

katiehockman commented Jan 9, 2019

bcmills commented Jan 9, 2019

dmitshur commented Jan 9, 2019

gopherbot commented Jan 25, 2019

gopherbot commented Jan 28, 2019

gopherbot commented Dec 13, 2019

gopherbot commented Dec 13, 2019

gopherbot commented Jan 2, 2020

gopherbot commented Jan 8, 2020

gopherbot commented Apr 20, 2020

gopherbot commented Apr 22, 2020

gopherbot commented Apr 22, 2020

gopherbot commented Apr 22, 2020

gopherbot commented Jul 16, 2020

dmitshur commented Aug 28, 2020 •

edited

Loading

doc, x/tools/cmd/godoc, x/website: move golang.org content into its own repository #29206

doc, x/tools/cmd/godoc, x/website: move golang.org content into its own repository #29206

Comments

andybons commented Dec 13, 2018

bradfitz commented Dec 13, 2018 • edited by dmitshur Loading

andybons commented Dec 13, 2018 • edited Loading

dmitshur commented Dec 13, 2018 • edited Loading

bradfitz commented Dec 13, 2018

cnoellekb commented Dec 20, 2018

bradfitz commented Dec 21, 2018

dmitshur commented Dec 21, 2018 • edited Loading

bradfitz commented Jan 2, 2019

cnoellekb commented Jan 3, 2019

gopherbot commented Jan 3, 2019

gopherbot commented Jan 4, 2019

gopherbot commented Jan 4, 2019

gopherbot commented Jan 4, 2019

cnoellekb commented Jan 9, 2019

katiehockman commented Jan 9, 2019

bcmills commented Jan 9, 2019

dmitshur commented Jan 9, 2019

gopherbot commented Jan 25, 2019

gopherbot commented Jan 28, 2019

gopherbot commented Dec 13, 2019

gopherbot commented Dec 13, 2019

gopherbot commented Jan 2, 2020

gopherbot commented Jan 8, 2020

gopherbot commented Apr 20, 2020

gopherbot commented Apr 22, 2020

gopherbot commented Apr 22, 2020

gopherbot commented Apr 22, 2020

gopherbot commented Jul 16, 2020

dmitshur commented Aug 28, 2020 • edited Loading

bradfitz commented Dec 13, 2018 •

edited by dmitshur

Loading

andybons commented Dec 13, 2018 •

edited

Loading

dmitshur commented Dec 13, 2018 •

edited

Loading

dmitshur commented Dec 21, 2018 •

edited

Loading

dmitshur commented Aug 28, 2020 •

edited

Loading