Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: add integration plugin tests #7299

Merged
merged 5 commits into from
Sep 2, 2024
Merged

test: add integration plugin tests #7299

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
24c7959
test(integration): add plugin tests
DmitriyLewen Aug 1, 2024
a06bb75
mage: add test:plugin
DmitriyLewen Aug 1, 2024
60cf5a4
refactor:
DmitriyLewen Aug 2, 2024
eeda1c8
ci: add plugin tests to `test` workflow
DmitriyLewen Aug 2, 2024
33cf9d5
refactor: run plugin tests with `integration` tests
DmitriyLewen Aug 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/workflows/test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,27 @@ jobs:
shell: bash
run: |
mage test:module
plugin-test:
name: Plugin Integration Test
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/[email protected]

- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: ${{ env.GO_VERSION }}

- name: Install tools
uses: aquaproj/[email protected]
with:
aqua_version: v1.25.0

- name: Run plugin integration tests
shell: bash
run: |
mage test:plugin

vm-test:
name: VM Integration Test
Expand Down
2 changes: 1 addition & 1 deletion integration/integration_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build integration || vm_integration || module_integration || k8s_integration
//go:build integration || vm_integration || module_integration || k8s_integration || plugin_integration
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The VM, module, and K8s tests take longer to run and require a special setup. That's why they are separated. I think the plugin test can be part of the normal integration tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated in 33cf9d5


package integration

Expand Down
111 changes: 111 additions & 0 deletions integration/plugin_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
//go:build plugin_integration

package integration

import (
"io"
"os"
"path/filepath"
"testing"

"github.com/stretchr/testify/require"

"github.com/aquasecurity/trivy/pkg/utils/fsutils"
)

func TestPlugin(t *testing.T) {
tests := []struct {
name string
plugin string
pluginArgs string
golden string
}{
{
name: "count plugin installed from `index`",
plugin: "[email protected]",
golden: "testdata/count-0.2.0-plugin.txt.golden",
},
{
name: "count plugin installed from github archive",
plugin: "https://github.com/aquasecurity/trivy-plugin-count/archive/refs/tags/v0.1.0.zip",
pluginArgs: "--published-before=2020-01-01",
golden: "testdata/count-0.1.0-plugin-with-before-flag.txt.golden",
},
}

// Set up testing DB
cacheDir := initDB(t)
tempStdOut := setTempStdout(t)

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// We can overwrite stdout for `_default_Manager` only once.
// So we need to clear the temporary stdout file before each test case.
clearFile(t, tempStdOut)

t.Setenv("XDG_DATA_HOME", t.TempDir())

// Install plugin
err := execute([]string{
"plugin",
"install",
tt.plugin,
})
require.NoError(t, err)

// Get list of plugins
err = execute([]string{
"plugin",
"list",
})
require.NoError(t, err)

// Run Trivy with plugin as output
args := []string{
"--cache-dir",
cacheDir,
"fs",
"-f",
"json",
"-o",
"plugin=count",
"testdata/fixtures/repo/pip",
}

if tt.pluginArgs != "" {
args = append(args, "--output-plugin-arg", tt.pluginArgs)
}

err = execute(args)

if *update {
fsutils.CopyFile(tempStdOut.Name(), tt.golden)
}

compareRawFiles(t, tt.golden, tempStdOut.Name())
})
}
}

func setTempStdout(t *testing.T) *os.File {
tmpFile := filepath.Join(t.TempDir(), "output.txt")
f, err := os.Create(tmpFile)
require.NoError(t, err)

// Overwrite Stdout to get output of plugin
defaultStdout := os.Stdout
os.Stdout = f
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if passing io.Writer here?

trivy/integration/integration_test.go

Line 284 in 88ba460

app.SetOut(io.Discard)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean

func execute(osArgs []string, out io.Writer) error {
    if out == nil {
        out = io.Discard
    }
    ...
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to use your idea.
But we use os.Stdout in plugin package.

I used NewManager instead of defaultManager in app.go to use Stdout from cobra.
It works, but there is problem with case when plugin is output.
We need to add output for plugin in Options:

trivy/pkg/flag/options.go

Lines 511 to 533 in 0c6687d

func (o *Options) outputPluginWriter(ctx context.Context) (io.Writer, func() error, error) {
pluginName := strings.TrimPrefix(o.Output, "plugin=")
pr, pw := io.Pipe()
wait, err := plugin.Start(ctx, pluginName, plugin.Options{
Args: o.OutputPluginArgs,
Stdin: pr,
})
if err != nil {
return nil, nil, xerrors.Errorf("plugin start: %w", err)
}
cleanup := func() error {
if err = pw.Close(); err != nil {
return xerrors.Errorf("failed to close pipe: %w", err)
}
if err = wait(); err != nil {
return xerrors.Errorf("plugin error: %w", err)
}
return nil
}
return pw, cleanup, nil
}

i am not sure that we need to do that now.
I would leave this workaround until the tests become more numerous and complex, which would require these changes.

But if you want to do it - tell me and I will continue the research

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, we can improve it later.

t.Cleanup(func() {
os.Stdout = defaultStdout
f.Close()
})
return f
}

func clearFile(t *testing.T, file *os.File) {
_, err := file.Seek(0, io.SeekStart)
require.NoError(t, err)

_, err = file.Write([]byte{})
require.NoError(t, err)
}
5 changes: 5 additions & 0 deletions integration/testdata/count-0.1.0-plugin-with-before-flag.txt.golden
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Installed Plugins:
Name: count
Version: 0.1.0

Number of vulnerabilities: 1
5 changes: 5 additions & 0 deletions integration/testdata/count-0.2.0-plugin.txt.golden
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Installed Plugins:
Name: count
Version: 0.2.0

Number of vulnerabilities: 2
5 changes: 5 additions & 0 deletions magefiles/magefile.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,6 +285,11 @@ func (t Test) K8s() error {
return sh.RunWithV(ENV, "go", "test", "-v", "-tags=k8s_integration", "./integration/...")
}

// Plugin runs plugin integration tests
func (t Test) Plugin() error {
return sh.RunWithV(ENV, "go", "test", "-v", "-tags=plugin_integration", "./integration/...")
}

// Module runs Wasm integration tests
func (t Test) Module() error {
mg.Deps(t.FixtureContainerImages, t.GenerateExampleModules)
Expand Down