Skip to content

Commit

Permalink
feat: add memory cache backend (#7048)
Browse files Browse the repository at this point in the history
Signed-off-by: knqyf263 <[email protected]>
  • Loading branch information
knqyf263 authored Jun 28, 2024
1 parent 14d71ba commit 55ccd06
Show file tree
Hide file tree
Showing 16 changed files with 577 additions and 42 deletions.
50 changes: 40 additions & 10 deletions docs/docs/configuration/cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ The cache option is common to all scanners.
## Clear Caches
`trivy clean` subcommand removes caches.

```
```bash
$ trivy clean --scan-cache
```

Expand All @@ -31,31 +31,59 @@ See `trivy clean --help` for details.
## Cache Directory
Specify where the cache is stored with `--cache-dir`.

```
```bash
$ trivy --cache-dir /tmp/trivy/ image python:3.4-alpine3.9
```

## Cache Backend
## Scan Cache Backend
!!! warning "EXPERIMENTAL"
This feature might change without preserving backwards compatibility.

Trivy supports local filesystem and Redis as the cache backend. This option is useful especially for client/server mode.

Two options:
Trivy utilizes a scan cache to store analysis results, such as package lists.
It supports three types of backends for this cache:

- `fs`
- the cache path can be specified by `--cache-dir`
- `redis://`
- Local File System (`fs`)
- The cache path can be specified by `--cache-dir`
- Memory (`memory`)
- Redis (`redis://`)
- `redis://[HOST]:[PORT]`
- TTL can be configured via `--cache-ttl`

### Local File System
The local file system backend is the default choice for container and VM image scans.
When scanning container images, it stores analysis results on a per-layer basis, using layer IDs as keys.
This approach enables faster scans of the same container image or different images that share layers.

!!! note
Internally, this backend uses [BoltDB][boltdb], which has an important limitation: only one process can access the cache at a time.
Subsequent processes attempting to access the cache will be locked.
For more details on this limitation, refer to the [troubleshooting guide][parallel-run].

### Memory
The memory backend stores analysis results in memory, which means the cache is discarded when the process ends.
This makes it useful in scenarios where caching is not required or desired.
It serves as the default for repository, filesystem and SBOM scans and can also be employed for container image scans when caching is unnecessary.

To use the memory backend for a container image scan, you can use the following command:

```bash
$ trivy image debian:11 --cache-backend memory
```

### Redis

The Redis backend is particularly useful when you need to share the cache across multiple Trivy instances.
You can set up Trivy to use a Redis backend with a command like this:

```bash
$ trivy server --cache-backend redis://localhost:6379
```

This approach allows for centralized caching, which can be beneficial in distributed or high-concurrency environments.

If you want to use TLS with Redis, you can enable it by specifying the `--redis-tls` flag.

```shell
```bash
$ trivy server --cache-backend redis://localhost:6379 --redis-tls
```

Expand All @@ -72,6 +100,8 @@ $ trivy server --cache-backend redis://localhost:6379 \
[trivy-db]: ./db.md#vulnerability-database
[trivy-java-db]: ./db.md#java-index-database
[misconf-checks]: ../scanner/misconfiguration/check/builtin.md
[boltdb]: https://github.com/etcd-io/bbolt
[parallel-run]: https://aquasecurity.github.io/trivy/v0.52/docs/references/troubleshooting/#running-in-parallel-takes-same-time-as-series-run

[^1]: Downloaded when scanning for vulnerabilities
[^2]: Downloaded when scanning `jar/war/par/ear` files
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/references/configuration/cli/trivy_config.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ trivy config [flags] DIR
### Options

```
--cache-backend string cache backend (e.g. redis://localhost:6379) (default "fs")
--cache-backend string [EXPERIMENTAL] cache backend (e.g. redis://localhost:6379) (default "memory")
--cache-ttl duration cache TTL when using redis as cache backend
--cf-params strings specify paths to override the CloudFormation parameters files
--check-namespaces strings Rego namespaces
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/references/configuration/cli/trivy_filesystem.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ trivy filesystem [flags] PATH
### Options

```
--cache-backend string cache backend (e.g. redis://localhost:6379) (default "fs")
--cache-backend string [EXPERIMENTAL] cache backend (e.g. redis://localhost:6379) (default "memory")
--cache-ttl duration cache TTL when using redis as cache backend
--cf-params strings specify paths to override the CloudFormation parameters files
--check-namespaces strings Rego namespaces
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/references/configuration/cli/trivy_image.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ trivy image [flags] IMAGE_NAME
### Options

```
--cache-backend string cache backend (e.g. redis://localhost:6379) (default "fs")
--cache-backend string [EXPERIMENTAL] cache backend (e.g. redis://localhost:6379) (default "fs")
--cache-ttl duration cache TTL when using redis as cache backend
--check-namespaces strings Rego namespaces
--checks-bundle-repository string OCI registry URL to retrieve checks bundle from (default "ghcr.io/aquasecurity/trivy-checks:0")
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/references/configuration/cli/trivy_kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ trivy kubernetes [flags] [CONTEXT]

```
--burst int specify the maximum burst for throttle (default 10)
--cache-backend string cache backend (e.g. redis://localhost:6379) (default "fs")
--cache-backend string [EXPERIMENTAL] cache backend (e.g. redis://localhost:6379) (default "fs")
--cache-ttl duration cache TTL when using redis as cache backend
--check-namespaces strings Rego namespaces
--checks-bundle-repository string OCI registry URL to retrieve checks bundle from (default "ghcr.io/aquasecurity/trivy-checks:0")
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/references/configuration/cli/trivy_repository.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ trivy repository [flags] (REPO_PATH | REPO_URL)

```
--branch string pass the branch name to be scanned
--cache-backend string cache backend (e.g. redis://localhost:6379) (default "fs")
--cache-backend string [EXPERIMENTAL] cache backend (e.g. redis://localhost:6379) (default "memory")
--cache-ttl duration cache TTL when using redis as cache backend
--cf-params strings specify paths to override the CloudFormation parameters files
--check-namespaces strings Rego namespaces
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/references/configuration/cli/trivy_rootfs.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ trivy rootfs [flags] ROOTDIR
### Options

```
--cache-backend string cache backend (e.g. redis://localhost:6379) (default "fs")
--cache-backend string [EXPERIMENTAL] cache backend (e.g. redis://localhost:6379) (default "memory")
--cache-ttl duration cache TTL when using redis as cache backend
--cf-params strings specify paths to override the CloudFormation parameters files
--check-namespaces strings Rego namespaces
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/references/configuration/cli/trivy_sbom.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ trivy sbom [flags] SBOM_PATH
### Options

```
--cache-backend string cache backend (e.g. redis://localhost:6379) (default "fs")
--cache-backend string [EXPERIMENTAL] cache backend (e.g. redis://localhost:6379) (default "memory")
--cache-ttl duration cache TTL when using redis as cache backend
--compliance string compliance report to generate
--custom-headers strings custom headers in client mode
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/references/configuration/cli/trivy_server.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ trivy server [flags]
### Options

```
--cache-backend string cache backend (e.g. redis://localhost:6379) (default "fs")
--cache-backend string [EXPERIMENTAL] cache backend (e.g. redis://localhost:6379) (default "fs")
--cache-ttl duration cache TTL when using redis as cache backend
--db-repository string OCI repository to retrieve trivy-db from (default "ghcr.io/aquasecurity/trivy-db:2")
--download-db-only download/update vulnerability database but don't run a scan
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/references/configuration/cli/trivy_vm.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ trivy vm [flags] VM_IMAGE

```
--aws-region string AWS region to scan
--cache-backend string cache backend (e.g. redis://localhost:6379) (default "fs")
--cache-backend string [EXPERIMENTAL] cache backend (e.g. redis://localhost:6379) (default "fs")
--cache-ttl duration cache TTL when using redis as cache backend
--checks-bundle-repository string OCI registry URL to retrieve checks bundle from (default "ghcr.io/aquasecurity/trivy-checks:0")
--compliance string compliance report to generate
Expand Down
8 changes: 8 additions & 0 deletions pkg/cache/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,15 @@ import (
"time"

"golang.org/x/xerrors"

"github.com/aquasecurity/trivy/pkg/log"
)

const (
TypeUnknown Type = "unknown"
TypeFS Type = "fs"
TypeRedis Type = "redis"
TypeMemory Type = "memory"
)

type Type string
Expand All @@ -33,6 +36,8 @@ func NewType(backend string) Type {
return TypeRedis
case backend == "fs", backend == "":
return TypeFS
case backend == "memory":
return TypeMemory
default:
return TypeUnknown
}
Expand All @@ -44,6 +49,7 @@ func New(opts Options) (Cache, func(), error) {

var cache Cache
t := NewType(opts.Backend)
log.Debug("Initializing scan cache...", log.String("type", string(t)))
switch t {
case TypeRedis:
redisCache, err := NewRedisCache(opts.Backend, opts.RedisCACert, opts.RedisCert, opts.RedisKey, opts.RedisTLS, opts.TTL)
Expand All @@ -58,6 +64,8 @@ func New(opts Options) (Cache, func(), error) {
return nil, cleanup, xerrors.Errorf("unable to initialize fs cache: %w", err)
}
cache = fsCache
case TypeMemory:
cache = NewMemoryCache()
default:
return nil, cleanup, xerrors.Errorf("unknown cache type: %s", t)
}
Expand Down
98 changes: 98 additions & 0 deletions pkg/cache/memory.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
package cache

import (
"sync"

"golang.org/x/xerrors"

"github.com/aquasecurity/trivy/pkg/fanal/types"
)

var _ Cache = &MemoryCache{}

type MemoryCache struct {
artifacts sync.Map // Map to store artifact information
blobs sync.Map // Map to store blob information
}

func NewMemoryCache() *MemoryCache {
return &MemoryCache{}
}

// PutArtifact stores the artifact information in the memory cache
func (c *MemoryCache) PutArtifact(artifactID string, artifactInfo types.ArtifactInfo) error {
c.artifacts.Store(artifactID, artifactInfo)
return nil
}

// PutBlob stores the blob information in the memory cache
func (c *MemoryCache) PutBlob(blobID string, blobInfo types.BlobInfo) error {
c.blobs.Store(blobID, blobInfo)
return nil
}

// DeleteBlobs removes the specified blobs from the memory cache
func (c *MemoryCache) DeleteBlobs(blobIDs []string) error {
for _, blobID := range blobIDs {
c.blobs.Delete(blobID)
}
return nil
}

// GetArtifact retrieves the artifact information from the memory cache
func (c *MemoryCache) GetArtifact(artifactID string) (types.ArtifactInfo, error) {
info, ok := c.artifacts.Load(artifactID)
if !ok {
return types.ArtifactInfo{}, xerrors.Errorf("artifact (%s) not found in memory cache", artifactID)
}
artifactInfo, ok := info.(types.ArtifactInfo)
if !ok {
return types.ArtifactInfo{}, xerrors.Errorf("invalid type for artifact (%s) in memory cache", artifactID)
}
return artifactInfo, nil
}

// GetBlob retrieves the blob information from the memory cache
func (c *MemoryCache) GetBlob(blobID string) (types.BlobInfo, error) {
info, ok := c.blobs.Load(blobID)
if !ok {
return types.BlobInfo{}, xerrors.Errorf("blob (%s) not found in memory cache", blobID)
}
blobInfo, ok := info.(types.BlobInfo)
if !ok {
return types.BlobInfo{}, xerrors.Errorf("invalid type for blob (%s) in memory cache", blobID)
}
return blobInfo, nil
}

// MissingBlobs determines the missing artifact and blob information in the memory cache
func (c *MemoryCache) MissingBlobs(artifactID string, blobIDs []string) (bool, []string, error) {
var missingArtifact bool
var missingBlobIDs []string

if _, err := c.GetArtifact(artifactID); err != nil {
missingArtifact = true
}

for _, blobID := range blobIDs {
if _, err := c.GetBlob(blobID); err != nil {
missingBlobIDs = append(missingBlobIDs, blobID)
}
}

return missingArtifact, missingBlobIDs, nil
}

// Close clears the artifact and blob information from the memory cache
func (c *MemoryCache) Close() error {
c.artifacts = sync.Map{}
c.blobs = sync.Map{}
return nil
}

// Clear clears the artifact and blob information from the memory cache
func (c *MemoryCache) Clear() error {
c.artifacts = sync.Map{}
c.blobs = sync.Map{}
return nil
}
Loading

0 comments on commit 55ccd06

Please sign in to comment.