Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tool for generating licenses for 3rd party deps #1689

Closed
Closed
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 137 additions & 0 deletions hack/licenser/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

package main

import (
"bufio"
"flag"
"fmt"
"io/ioutil"
"log"
"os"
"path/filepath"
"strings"
"text/template"
)

var (
depFile = flag.String("f", "go.mod", "File with the list of dependencies")
dir = flag.String("d", "", "Project directory")
)

type Dependency struct {
Name string
Version string
License string
}

type Dependencies struct {
List []*Dependency
}

func main() {
flag.Parse()
log.SetFlags(log.LstdFlags | log.Lshortfile)

deps, err := loadFile(filepath.Join(*dir, *depFile))
Copy link
Contributor

@charith-elastic charith-elastic Sep 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if parsing the output of go list -deps -test -json [1] might be better than parsing go.mod itself because the former is less likely to change with newer Go versions.

[1] probably needs to be tweaked a bit to get the desired output

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to get output from go list but wasn't been able to get something useful from it. Instead I decided to use modules.txt file which is generated when go mod vendor is executed. It will produce easy to parse text file like:

# github.com/davecgh/go-spew v1.1.1
github.com/davecgh/go-spew/spew
# github.com/docker/spdystream v0.0.0-20181023171402-6480d4af844c
github.com/docker/spdystream
github.com/docker/spdystream/spdy
....

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had some luck parsing the output of go list and running licence-detector. Happy to collaborate on this if you are up for it.

if err != nil {
log.Fatalf("Can't open file with dependencies: %s", err.Error())
}

issues := checkForLicense(deps)
if issues > 0 {
log.Fatal("Can't create NOTICE.txt, there are issues with dependencies!")
}

createNoticeFile(deps)
}

func loadFile(path string) (*Dependencies, error) {
f, err := os.Open(path)
if err != nil {
return nil, err
}
defer f.Close()

var lines []string
scanner := bufio.NewScanner(f)
for scanner.Scan() {
lines = append(lines, scanner.Text())
}

deps := &Dependencies{}
for i, v := range lines {
if i > 4 && i < (len(lines)-1) {
dep := strings.Split(v, " ")
path := dep[0]
path = strings.Replace(path, " ", "", -1)
deps.List = append(deps.List, &Dependency{Name: path, Version: dep[1]})
}
}

return deps, nil
}

func checkForLicense(deps *Dependencies) int {
var issues []string
licenses := []string{"LICENSE", "LICENSE.txt", "LICENCE"} // Used to keep all possible names of license files
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if using a library like https://github.com/src-d/go-license-detector might be better here so that we don't have to maintain our own list of different spellings and typos.

for _, dep := range deps.List {
counter := len(licenses)
for _, v := range licenses {
bytes, err := ioutil.ReadFile(filepath.Join(*dir, "vendor", dep.Name, v))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we can count on the vendor directory since the default for go modules is to not use it. Looks like the default src cache is $GOPATH/pkg/mod: https://golang.org/cmd/go/#hdr-GOPATH_and_Modules

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we can't count on it out of the box. But I'm thinking about using go mod vendor to collect all deps into vendor folder. In this case we can rely on content of vendor dir.

if err != nil {
counter--
continue
}
dep.License = string(bytes)
break
}
if counter == 0 {
issues = append(issues, fmt.Sprintf("Can't find file with license for %s version %s", dep.Name, dep.Version))
}
}

if len(issues) > 0 {
fmt.Println("Number of issues:", len(issues))
for _, v := range issues {
fmt.Println(v)
}
}

return len(issues)
}

func createNoticeFile(deps *Dependencies) {
var tmpl = `Elastic Cloud on Kubernetes
Copyright 2014-2019 Elasticsearch BV

This product includes software developed by The Apache Software
Foundation (http://www.apache.org/).

==========================================================================
Third party libraries used by the Elastic Cloud on Kubernetes project:
==========================================================================

{{range $i,$v := .}}
Dependency: {{ $v.Name }}
Version: {{ $v.Version }}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Say we have 10 dependencies that are Apache 2 licensed. Aren't we repeating the full text of the Apache licence 10 times in the notice file in that case? Is that the expected behaviour?

{{ $v.License }}
--------------------------------------------------------------------------
{{end}}
`
f, err := os.Create(filepath.Join(*dir, "NOTICE.txt"))
if err != nil {
log.Fatalf("Can't create NOTICE.txt: %s", err.Error())
}

t := template.Must(template.New("notice").Parse(tmpl))
err = t.Execute(f, deps.List)
if err != nil {
log.Fatalf("Failed on creating list of licenses for dependencies: %s", err.Error())
}

fmt.Println("NOTICE.txt was generated!")
}