Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tool for generating licenses for 3rd party deps #1689

Closed
Closed
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions hack/licenser/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Licenser

This tool is used to generate file `NOTICE.txt` which contains certain notices of software components included in ECK.

## Usage

Tool is designed to work with [Go modules](https://github.com/golang/go/wiki/Modules) for managing dependencies, i.e. it requires Go 1.11+ to work. It expects to have all dependencies downloaded in `vendor` directory and file `vendor/modules.txt` exists. It can be achieved by running `go mod vendor`

To run it use:
```go
go mod vendor
cd hack/licenser
go run main.go -d path/to/repo
```
135 changes: 135 additions & 0 deletions hack/licenser/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

package main

import (
"bufio"
"flag"
"fmt"
"io/ioutil"
"log"
"os"
"path/filepath"
"strings"
"text/template"
)

var (
depFile = flag.String("f", "modules.txt", "File with the list of dependencies")
dir = flag.String("d", "", "Project directory")
)

type Dependency struct {
Name string
Version string
License string
}

type Dependencies struct {
List []*Dependency
}

func main() {
flag.Parse()
log.SetFlags(log.LstdFlags | log.Lshortfile)

deps, err := loadFile(filepath.Join(*dir, "vendor", *depFile))
if err != nil {
log.Fatalf("Can't open file with dependencies: %s", err.Error())
}

issues := checkForLicense(deps)
if issues > 0 {
log.Fatal("Can't create NOTICE.txt, there are issues with dependencies!")
}

createNoticeFile(deps)
}

func loadFile(path string) (*Dependencies, error) {
f, err := os.Open(path)
if err != nil {
return nil, err
}
defer f.Close()

var lines []string
scanner := bufio.NewScanner(f)
for scanner.Scan() {
lines = append(lines, scanner.Text())
}

deps := &Dependencies{}
for _, v := range lines {
if strings.HasPrefix(v, "#") {
dep := strings.Split(v, " ")
deps.List = append(deps.List, &Dependency{Name: dep[1], Version: dep[2]})
}
}

return deps, nil
}

func checkForLicense(deps *Dependencies) int {
var issues []string
licenses := []string{"LICENSE", "LICENSE.txt", "LICENCE"} // Used to keep all possible names of license files
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if using a library like https://github.com/src-d/go-license-detector might be better here so that we don't have to maintain our own list of different spellings and typos.

for _, dep := range deps.List {
counter := len(licenses)
for _, v := range licenses {
bytes, err := ioutil.ReadFile(filepath.Join(*dir, "vendor", dep.Name, v))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we can count on the vendor directory since the default for go modules is to not use it. Looks like the default src cache is $GOPATH/pkg/mod: https://golang.org/cmd/go/#hdr-GOPATH_and_Modules

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we can't count on it out of the box. But I'm thinking about using go mod vendor to collect all deps into vendor folder. In this case we can rely on content of vendor dir.

if err != nil {
counter--
continue
}
dep.License = string(bytes)
break
}
if counter == 0 {
issues = append(issues, fmt.Sprintf("Can't find file with license for %s version %s", dep.Name, dep.Version))
}
}

if len(issues) > 0 {
fmt.Println("Number of issues:", len(issues))
for _, v := range issues {
fmt.Println(v)
}
}

return len(issues)
}

func createNoticeFile(deps *Dependencies) {
var tmpl = `Elastic Cloud on Kubernetes
Copyright 2014-2019 Elasticsearch BV

This product includes software developed by The Apache Software
Foundation (http://www.apache.org/).

==========================================================================
Third party libraries used by the Elastic Cloud on Kubernetes project:
==========================================================================

{{range $i,$v := .}}
Dependency: {{ $v.Name }}
Version: {{ $v.Version }}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Say we have 10 dependencies that are Apache 2 licensed. Aren't we repeating the full text of the Apache licence 10 times in the notice file in that case? Is that the expected behaviour?

{{ $v.License }}
--------------------------------------------------------------------------
{{end}}
`
f, err := os.Create(filepath.Join(*dir, "NOTICE.txt"))
if err != nil {
log.Fatalf("Can't create NOTICE.txt: %s", err.Error())
}

t := template.Must(template.New("notice").Parse(tmpl))
err = t.Execute(f, deps.List)
if err != nil {
log.Fatalf("Failed on creating list of licenses for dependencies: %s", err.Error())
}

fmt.Println("NOTICE.txt was generated!")
}