Best project structure when using go 1.11 mod

[ypeckstadt]

The first thing you will run into when using go.mod, putting code outside GOPATH, and using the recommended folder structure is that your cmd/ does not have access to your internal or pkg folder. Therefore you are automatically required to resort to using the require/replace fix to actually get access to the code or actually split your code into actual modules and upload separately.

While the require/replace fix works and builds, your IDE will probably still have problems picking up the imports for good code navigation unless you manually update your go.mod file in the pkg or internal folder. (you cant use go build to update your go.mod/sum file as there is nothing to build in the folder) Of course, this is more of an IDE problem.

module some-api

replace some-pkg v0.0.0 => ../../pkg

require (
	some-pkg v0.0.0
)

Would people still recommend this folder layout when using go modules, have multiple cmd applications, and especially when putting code outside the gopath? I am curious as I see more and more people starting to use modules and putting their code in a GOPATH-less environment and this is for sure bound to create problems and confusion when using modules as it requires "some" custom setup. Of course, nothing major once you figure it out.

Come to think of it, this is more of a general issue with go mod. You will run into this issue from the moment you put your main.go file in a subfolder and not the root.

[andrewpillar]

I've found that Go modules tend to not work all to well when it comes to the whole mono-repo approach that was encouraged by the use of $GOPATH. Go modules are still in the works however as of Go 1.11, and will be finalised in Go 1.12 as stated here.

For now I've just done as you have, and resorted to using replace in the go.mod file so I can locally pull in packages as I'm developing them. This isn't ideal though, as I've found building the application to be somewhat slower than just having it be pulled from a repo, however the alternative of having to tag, push, then build just to pick up on the changes I've made not that ideal either.

[ypeckstadt]

@andrewpillar Thank you for the feedback. Yeah, I guess in the meantime I will keep doing the same and see which changes might come in the next version or revert back to just using GOPATH.

[andrewpillar]

Yeah, I think the approach I'll be taking for now is alternating between using replace and tagging necessary versions as I go along.

[tiramiseb]

I simply use the whole repository address in go.mod (which is exactly what is described here):

module github.com/user-or-org/some-api

require (
	[other requirements]
)

Do you have a problem with this approach?

[andrewpillar]

@tiramiseb one problem I've encountered is that unless I put the replace [package] => ../../ in the go.mod file I won't be able to build with the latest changes I've made unless I tag it, then push it up to the remote.

For projects where I only have one go.mod file in the root of the project this isn't an issue, but for projects where I would have multiple go.mod files within sub-directories of the cmd/ directory this has been an issue.

project/
├── go.mod
└── cmd/
    ├── command-one/
    |   └── go.mod
    └── command-two/
        └── go.mod

Consider the above project structure, if I made any changes to any packages I have in the top-level project/ directory, I would want the two sub modules, command-one, and command-two to pick up on those changes when I run go build in either of them. With Go modules I can do this by tagging a new release, and pushing up to the repo so when I run go build in either command-one, or command-two the most recent changes will be included. Or I could use replace project => ../../ in both go.mod files during development phases so I don't have to create needless tags.

I hope I've explained this well enough, but the short of it is this: with Go modules I have ran into issues when taking a mono-repo approach in development. If I am missing something, or misunderstanding certain concepts then please do tell me.

[tiramiseb]

Indeed. I only have go.mod at the root of the project. I see your point...

[alexeldeib]

@tiramiseb I’m in the same boat, only a go.mod at the root, but I hit the same issue as @ypeckstadt if I moved my main.go into cmd/something.

Did you (or anyone else) have a project structure that allowed for a single go.mod at repo root, but with all other code contained in subdirs? Or have I missed something here in structuring my code?

Ideally, I’d like to end up with something similar to the standard structure many projects use with Gopkg.toml at the root, but basically all code inside cmd/ or pkg/ (substituting go.mod for Gopkg.toml of course).

[tiramiseb]

This is exactly what I am doing : .go files only in pkg/ or cmd/, with a single go.mod at the root of the repository.

[alexeldeib]

EDIT: Came back to this and noticed it had a lot of reactions! Here's a similar, simple project building binaries in and out of the root folder: https://github.com/alexeldeib/godemo

I must be missing something -- I read that doc several times and haven't been able to figure out my issue. If for example I use this structure (example):

.
├── cmd
│   └── app
│       ├── main.go
│       └── main_test.go
├── go.mod
├── go.sum
├── pkg
    ├── handlers 
    │   └── handlers.go
    ├── types
    │   └── types.go
    └── util
        └── util.go

I can hit the imports successfully, but if I do e.g., go build at the root of the module I see:
can't load package...unknown import path github.com/alexeldeib/repo...cannot find module providing package <same path as before>. Apologies if i'm missing something basic here, I admit I'm no Go aficionado 😊

I'm not sure this necessarily should work, but if not I'm unsure how I would go about this otherwise. Could you possibly share a an example project structure you use, with the application entrypoint in cmd/?

[tiramiseb]

Have you put the complete package address in the go.mod header (and not only its name), as explained above?

[alexeldeib]

$DIR/go.mod:

module github.com/alexeldeib/app

require (
	github.com/go-playground/locales v0.12.1 // indirect
	github.com/go-playground/universal-translator v0.16.0 // indirect
	.
        .
        . // etc
)

$DIR/cmd/app/main.go:

package app # WRONG -- should be package main, see below or github.com/alexeldeib/godemo

import (
	"net/http"
	"os"
)

func main() {
    // Do stuff
}

and then go build run from $DIR yields the previous error. I feel like there's some misalignment between the package naming and module naming that is causing confusion? Should I be using the full repository + pkg name inside my .go files too?

Appreciate your advice!

EDIT: If you're looking at this example, it should be package main 🙂

[tiramiseb]

The only difference I see is that I run go build from the $DIR/cmd/app/ directory...

[alexeldeib]

D'oh. That was it. Thank you!! 😂

[dominictobias]

I like the idea of using Go, mostly for performance reasons (not the nicest or most succinct syntax tbh) but I find it strange that 4 years later there is only a rudimentary, barely working package/module system that doesn't scale at all well once you go beyond a single directory. Guess I check back in another year 😔

[frederikhors]

rudimentary, barely working package/module system...

@dominictobias, can you explain it better please?

[dominictobias]

rudimentary, barely working package/module system...

@dominictobias, can you explain it better please?

As far as I can tell if I want to structure an application how I would in another language (lots of folders), I have to do this require/replace thing which causes the IDE/intellisense not to work properly and Go has no concept of local private packages, in fact I would just describe them as modules that make up an app, they're not intended to be separately published. I wish I could just structure an app how I pleased and do relative imports. Even if I want to put some cloud functions in folders for example gcloud can no longer find anything. Maybe it's my lack of experience but I find this stuff trivial with NPM for example.

The google functions examples show things at one level deep but if I have 60 cloud functions that's 120 files with tests in one folder at minimum, plus shared utility code which could be dozens of files. I find whenever I give Go-a-go again I quickly hit this wall of googling how to structure more than a hello world.

[frederikhors]

I think you need to understand how go mod works.

[ypeckstadt]

Since I created this issue I really haven't run into any problems anymore while working with a project that has many folders and that follows the go-lang standard folder layout recommendations.
As mentioned before just put the go mod in the root and everything works just fine. Having experienced this now, to be honest, I don't even understand my initial problem anymore :) Probably because I tried to have multiple modules while I didn't need to I guess.

[cbrake]

Go does not support relative imports.

https://stackoverflow.com/questions/38517593/relative-imports-in-go

But, like others, I've had no problems structuring large applications with lots of directories. You need to give the module a name, and then use it with all imports. Example:

https://github.com/simpleiot/simpleiot/blob/master/cmd/siot/main.go

If you set up your editor to use use goimports, then the imports automatically get added/removed as needed, so it requires almost no effort to manage imports. Works pretty well for me.

[dominictobias]

Thanks all, I'll give it another go!

[vishal-yadav]

You can have a look into kubernetes project structure which is following a similar topology. Though it also needs require/replace in go.mod

My idea of go-dev can be:

• Using IDE for the purpose of browsing (including navigating to function definition), editing, golint, gofmt with automatic expansion etc. This uses go installation (on VM) with GOPATH set.
• Using layered Dockerfiles with source mounted or copied in container. This uses go setup with GOPATH unset and using GO111MODULE=on. Layered Dockerfile can be divided to mod, build and run.

This has advantage that go mod/build layer can be reused to save overall build time. Also resulting app/run image can be based on alpine to have significantly less image size.

[wangling94]

if project use replace to import package in the same project, other who want to import the package will not found the package, because the path is ../package-name.

[tatarsky-v]

Are there any examples without 'github.com/' and $GOPATH prefixes?

[candlerb]

@alexeldeib: shouldn't it be "package main", rather than "package app", in $DIR/cmd/app/main.go ?

@tatarsky-v: the following works for me with go 1.13, in a random directory outside of GOPATH.

I first did go mod init example.com/foo, then created source files so it looks like this:

==> go.mod <==
module example.com/foo

go 1.13

==> cmd/hello/main.go <==
package main

import (
	"fmt"
	"example.com/foo/internal"
)

func main() {
	ans := ultimate.Answer()
	fmt.Printf("The answer is %d\n", ans)
}

==> internal/answer.go <==
package ultimate

func Answer() int {
    return 42
}

(I chose different package name than the directory name just to demonstrate)

If I cd cmd/hello and go build, I get hello built in the current directory. Good so far.

Also, at the top level I can do go build example.com/foo/cmd/hello. That works, although this time I get the binary written in the top level too. (Can override with go build -o filename ...)

The problem is, if I put public library code in a pkg/ directory, then I have to do:

go build example.com/foo/pkg

... and I presume this means the users would have to import it the same way. If I want users to import example.com/foo then it seems the package code needs to sit in the top-level directory.

Looking in Kubernetes source (which has a go.mod and a top-level pkg directory), it looks like the resulting /pkg/ does indeed appear as part of the path in all the import statements.

I did consider putting another go.mod inside the pkg directory, but I wonder if that will complicate matters sharing code with the cmd executables.

[alexeldeib]

@candlerb Executable commands must always use package main. (quoting https://golang.org/doc/code.html, under "Package names")

Here's a demo project producing three binaries: https://github.com/alexeldeib/godemo

[candlerb]

Correct - I was referring to this comment of yours, second half, where you put your main() function in package app. Typo?

[alexeldeib]

Yup, leaving editing my typo and answering completely for later visitors. 😄

Note also that /pkg/ is convention -- some projects (particularly those which are exclusively used as libraries) forgo the use of top level /pkg and expose their contents directly.

e.g. https://github.com/google/go-cloud

Some go even flatter:
https://github.com/hashicorp/memberlist
https://github.com/packethost/packngo

[davix]

Hi guys, on what condition we can close this issue?

[candlerb]

I think that to close this, a note should be added under the /pkg heading explaining the consequences if you choose to use this directory together with go.mod.

If I understand correctly, they are:

1. The users of your package will need to specify the /pkg suffix when they import your package - e.g. import "github.com/username/libname/pkg"

2. If you release later versions, the version number will be buried in the path (e.g. import "github.com/username/libname/v2/pkg")

3. If your library contains subpackages, then /pkg is further buried (e.g. import "github.com/username/libname/v2/pkg/widget")

In short, /pkg becomes part of your public API, rather than a hidden implementation detail. As far as I can see, the way to avoid this is to put your top-level .go files and subdirectories containing other public packages directly into the top level directory. You can still use /internal for private code.

Someone please correct me if I'm wrong.

[timdadd]

Go does not support relative imports.

https://stackoverflow.com/questions/38517593/relative-imports-in-go

But, like others, I've had no problems structuring large applications with lots of directories. You need to give the module a name, and then use it with all imports. Example:

https://github.com/simpleiot/simpleiot/blob/master/cmd/siot/main.go

If you set up your editor to use use goimports, then the imports automatically get added/removed as needed, so it requires almost no effort to manage imports. Works pretty well for me.

I had a quick look at this example and I see a set of apps under cmd. However these apps are pulling stuff from git hub (e.g. "github.com/simpleiot/simpleiot/api"). Suppose you want to change api and the app at the same time how do you do that without ongoing promotion of api to github? What if you don't want the code on github, just locally?
Imagine a scenario of multiple GO APPS with a Dockerfile in each app directory controlling app specific build stuff. For the image build you don't want a nice clean build for each app, not copying loads of stuff, just to pull from github anyway. Also don't want common pkg stuff having to be copied everywhere or promoted to github with a new tag for every change. How is that project structured? The only solution I can think of is that each app is a standalone GO project with its own mod.go and then the pkg gets copied prior to build and various bash/skaffold/make scripts hold it all together. But it doesn't seem too easy to fool GO into working like that.
Something tells me that it's got to be like that because imagine you have 20 apps and you make a change to something in pkg, do you have to fix all 20 apps to use your change before promoting to build - NO. So really each app needs to be able to work with a different version of pkg, in fact really different versions of sub-packages. So really the current GO model works and the sub-packages should be all on a source repository easy to pull down. But when you're (re)developing a package in unison with an app how do you avoid the endless promotion to github? The only solution I see is to make a copy within the app directory and then copy it back after it all works and promote to github - it sounds very messy. Am I missing something?

[tiramiseb]

Am I missing something?

Yes :)

In go.mod you can put, for instance:

replace git.some.host/somegroup/modname => ../modname

I put such lines at the end of the go.mod file.
... and once the lib is ready for publication, I put it on the git server and I comment this line with //.

This is the main subject of this issue...