Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Golang library for validation: folder structure validation #14

Merged
merged 46 commits into from
Aug 17, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
3717bb8
Stubbing out golang library API
ycombinator Jul 29, 2020
404b207
Adding spec to golang lib
ycombinator Jul 29, 2020
6964040
Creating stub code in internal/
ycombinator Jul 29, 2020
b64265d
Fleshing out
ycombinator Jul 30, 2020
b6dd885
Adding validation error tests
ycombinator Jul 30, 2020
82dd323
Updating API
ycombinator Jul 30, 2020
0fae987
Updating README
ycombinator Jul 30, 2020
4e131e8
Minor linting
ycombinator Jul 30, 2020
aeabbd5
Add recipe
ycombinator Jul 30, 2020
cbdd935
Add test target to Makefile
ycombinator Jul 30, 2020
d7dcc6e
Adding package tests
ycombinator Jul 30, 2020
22e411e
Consolidating tests
ycombinator Jul 31, 2020
bf0d2a6
Adding spec tests
ycombinator Jul 31, 2020
0772b13
Fleshing out validation logic a bit
ycombinator Jul 31, 2020
54d7bc7
Running go mod tidy
ycombinator Jul 31, 2020
31a3a3f
Adding TODOs
ycombinator Jul 31, 2020
00676ee
Updating bundled specs
ycombinator Jul 31, 2020
67fa8a0
Fleshing out folder validation a bit
ycombinator Aug 5, 2020
5eecf2b
Adding inverse validation
ycombinator Aug 5, 2020
fde8dee
Fleshing out TODO
ycombinator Aug 5, 2020
d35c926
Updating specs
ycombinator Aug 6, 2020
490334b
Removing unnecessary method
ycombinator Aug 6, 2020
81d3078
More validation logic for sub-folders
ycombinator Aug 6, 2020
3b45656
Removing old files
ycombinator Aug 6, 2020
4724bd7
Incorporating statik
ycombinator Aug 6, 2020
35f0097
Allowing for folders to have additionalContents
ycombinator Aug 6, 2020
f3af0a3
Updating specs
ycombinator Aug 6, 2020
0abda16
Spec updates
ycombinator Aug 6, 2020
9969cb5
Fixing parsing
ycombinator Aug 6, 2020
913c573
Better error message
ycombinator Aug 6, 2020
7741495
Sort imports
ycombinator Aug 10, 2020
11f07af
Fixing const visibility
ycombinator Aug 10, 2020
569b89b
Add guard
ycombinator Aug 10, 2020
2e23741
Adding godoc comments
ycombinator Aug 13, 2020
ea814fc
Fixing tests
ycombinator Aug 13, 2020
ac6bd53
Making sample_event.json not required
ycombinator Aug 13, 2020
b3a4829
Making changelog.yml not required
ycombinator Aug 13, 2020
13fe68c
Adding docs folder to test package
ycombinator Aug 13, 2020
b4cfaaf
Removing need for specific agent stream definition files
ycombinator Aug 13, 2020
cb8920b
Return nil when no validation errors
ycombinator Aug 14, 2020
8e7eeeb
Updating unit test
ycombinator Aug 14, 2020
2435fc7
Account for dataset test folder
ycombinator Aug 14, 2020
ad031c0
Updating spec in libs
ycombinator Aug 14, 2020
a3c17b6
Allowing additional ingest pipeline definition files
ycombinator Aug 14, 2020
727b716
Slightly stricter check for additional ingest pipeline YAML files
ycombinator Aug 14, 2020
d6461c6
Updating ingest pipeline file naming specs
ycombinator Aug 14, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ This repository contains:

Please use this repository to discuss any changes to the specification, either my making issues or PRs to the specification.

# Specification Format
# Specification Format

An Elastic Package specification describes:
1. the folder structure of packages and expected files within these folders; and
Expand Down
2 changes: 1 addition & 1 deletion code/go/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,4 @@ check:

# Runs tests
test:
@go test -v ./...
@go test -v ./...
3 changes: 3 additions & 0 deletions code/go/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ module github.com/elastic/package-spec/code/go
go 1.14

require (
github.com/Masterminds/semver/v3 v3.1.0
github.com/pkg/errors v0.9.1
github.com/rakyll/statik v0.1.7
github.com/stretchr/testify v1.6.1
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
)
4 changes: 4 additions & 0 deletions code/go/go.sum
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
github.com/Masterminds/semver/v3 v3.1.0 h1:Y2lUDsFKVRSYGojLJ1yLxSXdMmMYTYls0rCvoqmMUQk=
github.com/Masterminds/semver/v3 v3.1.0/go.mod h1:VPu/7SZ7ePZ3QOrcuXROw5FAcLl4a0cBrbBpGY/8hQs=
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rakyll/statik v0.1.7 h1:OF3QCZUuyPxuGEP7B4ypUa7sB/iHtqOTDYZXGM8KOdQ=
Expand Down
2 changes: 1 addition & 1 deletion code/go/internal/spec/statik.go

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion code/go/internal/spec_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@ package internal
import (
"testing"

_ "github.com/elastic/package-spec/code/go/internal/spec"
"github.com/rakyll/statik/fs"
"github.com/stretchr/testify/require"

_ "github.com/elastic/package-spec/code/go/internal/spec"
)

func TestBundledSpecs(t *testing.T) {
Expand Down
4 changes: 2 additions & 2 deletions code/go/internal/validator/errors.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ func (ve ValidationErrors) Error() string {
errorWord = "error"
}
fmt.Fprintf(&message, "found %v validation %v:\n", len(ve), errorWord)
for _, err := range ve {
fmt.Fprintf(&message, "\t%v\n", err)
for idx, err := range ve {
fmt.Fprintf(&message, "%4d. %v\n", idx+1, err)
}

return message.String()
Expand Down
217 changes: 217 additions & 0 deletions code/go/internal/validator/folder.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,217 @@
package validator

import (
"fmt"
"io/ioutil"
"net/http"
"os"
"path"
"path/filepath"
"regexp"

"github.com/pkg/errors"
"gopkg.in/yaml.v3"
)

const itemTypeFile = "file"
const itemTypeFolder = "folder"

type folderSpec struct {
fs http.FileSystem
specPath string
commonSpec
}

type folderItemSpec struct {
Description string `yaml:"description"`
ItemType string `yaml:"type"`
ContentMediaType string `yaml:"contentMediaType"`
Name string `yaml:"name"`
Pattern string `yaml:"pattern"`
Required bool `yaml:"required"`
Ref string `yaml:"$ref"`
commonSpec `yaml:",inline"`
}

type commonSpec struct {
AdditionalContents bool `yaml:"additionalContents"`
Contents []folderItemSpec `yaml:"contents"`
}

func newFolderSpec(fs http.FileSystem, specPath string) (*folderSpec, error) {
specFile, err := fs.Open(specPath)
if err != nil {
return nil, errors.Wrap(err, "could not open folder specification file")
}
defer specFile.Close()

data, err := ioutil.ReadAll(specFile)
if err != nil {
return nil, errors.Wrap(err, "could not read folder specification file")
}

var wrapper struct {
Spec commonSpec `yaml:"spec"`
}
if err := yaml.Unmarshal(data, &wrapper); err != nil {
return nil, errors.Wrap(err, "could not parse folder specification file")
}

spec := folderSpec{
fs: fs,
specPath: specPath,
commonSpec: wrapper.Spec,
}

return &spec, nil
}

func (s *folderSpec) validate(folderPath string) ValidationErrors {
var errs ValidationErrors
files, err := ioutil.ReadDir(folderPath)
if err != nil {
errs = append(errs, errors.Wrapf(err, "could not read folder [%s]", folderPath))
return errs
}

for _, file := range files {
fileName := file.Name()
itemSpec, err := s.findItemSpec(fileName)
if err != nil {
errs = append(errs, err)
continue
}

if itemSpec == nil && s.AdditionalContents {
// No spec found for current folder item but we do allow additional contents in folder.
continue
}

if itemSpec == nil && !s.AdditionalContents {
// No spec found for current folder item and we do not allow additional contents in folder.
errs = append(errs, fmt.Errorf("item [%s] is not allowed in folder [%s]", fileName, folderPath))
continue
}

if file.IsDir() {
if !itemSpec.isSameType(file) {
errs = append(errs, fmt.Errorf("[%s] is a folder but is expected to be a file", fileName))
continue
}

if itemSpec.Ref == "" && itemSpec.Contents == nil {
// No recursive validation needed
continue
}

var subFolderSpec *folderSpec
if itemSpec.Ref != "" {
subFolderSpecPath := path.Join(filepath.Dir(s.specPath), itemSpec.Ref)
subFolderSpec, err = newFolderSpec(s.fs, subFolderSpecPath)
if err != nil {
errs = append(errs, err)
continue
}
} else if itemSpec.Contents != nil {
subFolderSpec = &folderSpec{
fs: s.fs,
specPath: s.specPath,
commonSpec: commonSpec{
AdditionalContents: itemSpec.AdditionalContents,
Contents: itemSpec.Contents,
},
}
}

subFolderPath := path.Join(folderPath, fileName)
subErrs := subFolderSpec.validate(subFolderPath)
if len(subErrs) > 0 {
errs = append(errs, subErrs...)
}

} else {
if !itemSpec.isSameType(file) {
errs = append(errs, fmt.Errorf("[%s] is a file but is expected to be a folder", fileName))
continue
}
// TODO: more validation for file item
}
}

// validate that required items in spec are all accounted for
for _, itemSpec := range s.Contents {
if !itemSpec.Required {
continue
}

fileFound, err := itemSpec.matchingFileExists(files)
if err != nil {
errs = append(errs, err)
continue
}

if !fileFound {
var err error
if itemSpec.Name != "" {
err = fmt.Errorf("expecting to find [%s] %s in folder [%s]", itemSpec.Name, itemSpec.ItemType, folderPath)
} else if itemSpec.Pattern != "" {
err = fmt.Errorf("expecting to find %s matching pattern [%s] in folder [%s]", itemSpec.ItemType, itemSpec.Pattern, folderPath)
}
errs = append(errs, err)
}
}
return errs
}

func (s *folderSpec) findItemSpec(folderItemName string) (*folderItemSpec, error) {
for _, itemSpec := range s.Contents {
if itemSpec.Name != "" && itemSpec.Name == folderItemName {
return &itemSpec, nil
}
if itemSpec.Pattern != "" {
isMatch, err := regexp.MatchString(itemSpec.Pattern, folderItemName)
if err != nil {
return nil, errors.Wrap(err, "invalid folder item spec pattern")
}
if isMatch {
return &itemSpec, nil
}
}
}

// No item spec found
return nil, nil
}

func (s *folderItemSpec) matchingFileExists(files []os.FileInfo) (bool, error) {
if s.Name != "" {
for _, file := range files {
if file.Name() == s.Name {
return s.isSameType(file), nil
}
}
} else if s.Pattern != "" {
for _, file := range files {
isMatch, err := regexp.MatchString(s.Pattern, file.Name())
if err != nil {
return false, errors.Wrap(err, "invalid folder item spec pattern")
}
if isMatch {
return s.isSameType(file), nil
}
}
}

return false, nil
}

func (s *folderItemSpec) isSameType(file os.FileInfo) bool {
switch s.ItemType {
case itemTypeFile:
return !file.IsDir()
case itemTypeFolder:
return file.IsDir()
}

return false
}
61 changes: 61 additions & 0 deletions code/go/internal/validator/package.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
package validator

import (
"fmt"
"io/ioutil"
"os"
"path"

"github.com/Masterminds/semver/v3"
"github.com/pkg/errors"
"gopkg.in/yaml.v3"
)

// Package represents an Elastic Package Registry package
type Package struct {
SpecVersion *semver.Version
RootPath string
}

// NewPackage creates a new Package from a path to the package's root folder
func NewPackage(pkgRootPath string) (*Package, error) {
info, err := os.Stat(pkgRootPath)
if os.IsNotExist(err) {
return nil, errors.Wrapf(err, "no package found at path [%v]", pkgRootPath)
}

if !info.IsDir() {
return nil, fmt.Errorf("no package folder found at path [%v]", pkgRootPath)
}

pkgManifestPath := path.Join(pkgRootPath, "manifest.yml")
info, err = os.Stat(pkgManifestPath)
if os.IsNotExist(err) {
return nil, errors.Wrapf(err, "no package manifest file found at path [%v]", pkgManifestPath)
}

data, err := ioutil.ReadFile(pkgManifestPath)
if err != nil {
return nil, fmt.Errorf("could not read package manifest file [%v]", pkgManifestPath)
}

var manifest struct {
SpecVersion string `yaml:"format_version"`
}
if err := yaml.Unmarshal(data, &manifest); err != nil {
return nil, errors.Wrapf(err, "could not parse package manifest file [%v]", pkgManifestPath)
}

specVersion, err := semver.NewVersion(manifest.SpecVersion)
if err != nil {
return nil, errors.Wrapf(err, "could not read specification version from package manifest file [%v]", pkgManifestPath)
}

// Instantiate Package object and return it
p := Package{
specVersion,
pkgRootPath,
}

return &p, nil
}
43 changes: 43 additions & 0 deletions code/go/internal/validator/package_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
package validator

import (
"path"
"testing"

"github.com/Masterminds/semver/v3"
"github.com/stretchr/testify/require"
)

func TestNewPackage(t *testing.T) {
tests := map[string]struct {
expectedErrContains string
expectedSpecVersion *semver.Version
}{
"good": {
expectedSpecVersion: semver.MustParse("1.0.4"),
},
"non_existent": {
expectedErrContains: "no package found at",
},
"no_manifest": {
expectedErrContains: "no package manifest file found at",
},
"no_spec_version": {
expectedErrContains: "could not read specification version",
},
}

for pkgName, test := range tests {
pkgRootPath := path.Join("test", "packages", pkgName)
pkg, err := NewPackage(pkgRootPath)
if test.expectedErrContains == "" {
require.NoError(t, err)
require.Equal(t, test.expectedSpecVersion, pkg.SpecVersion)
require.Equal(t, pkgRootPath, pkg.RootPath)
} else {
require.Error(t, err)
require.Contains(t, err.Error(), test.expectedErrContains)
require.Nil(t, pkg)
}
}
}
Loading