Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev environment changes #1397

Merged
merged 14 commits into from
Nov 17, 2021
Merged

Dev environment changes #1397

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
7334521
webhook: fix a code smell
hdonnay Oct 12, 2021
1068f49
webhook: add debug server
hdonnay Oct 12, 2021
88cf06e
httptransport: gofmt simplification
hdonnay Oct 12, 2021
96bad4f
httptransport: error message correction
hdonnay Oct 12, 2021
0fd1120
local-dev: clair: update default dev config
hdonnay Oct 14, 2021
3ee44c1
local-dev: traefik: use file configuration
hdonnay Oct 14, 2021
c92b02c
local-dev: grafana: update endpoint
hdonnay Oct 14, 2021
a834836
local-dev: pgadmin: update for multiple databases
hdonnay Oct 14, 2021
2a0943b
local-dev: prometheus: update endpoints
hdonnay Oct 14, 2021
d0ec89d
local-dev: quay: update for other config changes
hdonnay Oct 14, 2021
f4f243f
docker-compose: use profiles and anchors
hdonnay Oct 14, 2021
356a4d7
Makefile: update targets and docs
hdonnay Oct 14, 2021
1695059
docker-compose: add paused skopeo container
hdonnay Nov 3, 2021
3c2ad90
docker-compose: add ActiveMQ container
hdonnay Nov 5, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 39 additions & 64 deletions Documentation/howto/testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Testing Clair

We provide dev tooling in order to quickly get a fully configured Clair and Quay environment stood up locally. This environment can be used to test and develop Clair.
We provide dev tooling in order to quickly get a fully configured Clair and Quay environment stood up locally.
This environment can be used to test and develop Clair's Quay integration.

## Requirements

Expand All @@ -12,15 +13,17 @@ It's very likely your workstation already has make on it.

### Docker and Docker Compose

Currently our local dev tooling is supported by docker and docker-compose. We are making strides to provide a podman native local dev environment but we are not quite there yet.
Currently our local dev tooling is supported by docker and docker-compose.
Podman should work fine since v3.0.

Docker version 19.03.11 and docker-compose version 1.25.4 are confirmed working. Our assumption is most recent versions will not have an issue running the local dev tooling.
Docker version 19.03.11 and docker-compose version 1.25.4 are confirmed working.
Our assumption is most recent versions will not have an issue running the local dev tooling.

See [Install Docker](https://docs.docker.com/get-docker/)

### Go Toolchain

Go v1.13 or higher is required.
Go v1.16 or higher is required.

See [Install Golang](https://golang.org/doc/install)

Expand All @@ -29,97 +32,69 @@ See [Install Golang](https://golang.org/doc/install)
```
git clone [email protected]:quay/clair.git
cd clair
make local-dev-up-with-quay
docker-compose up -d
# or: make local-dev
# or: make local-dev-debug
# or: make local-dev-quay
```

After the local development environment successfully starts, the following infrastructure is available to you:

```
localhost:8080 --- Quay (single node, local storage)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely an improvement here, although can we find somewhere for the credentials to avoid browsing / execing around for the defaults?


localhost:6060 --- Traefik which hosts all ClairV4 endpoints.
ClairV4 services are only accessible via this load balancer.

localhost:5432 --- ClairV4's Postgres DB
Login:
username: clair
database: clair

localhost:5433 --- Quay's Postgres DB
Login:
username: quay
database: quay

localhost:8081 --- Postgres GUI (pgadmin4)
Login:
username: [email protected]
password: clair

localhost:8082 --- OpenAPI Swagger Editor.
You can view ClairV4's public API here.
- `localhost:8080`
Dashboards and debugging services -- See the traefik configs in `local-dev/traefik` for where the various services are served.

localhost:8087 --- RabbitMQ management GUI
Login:
username: guest
password: guest
- `localhost:6060`
Clair services

localhost:8161 --- ActiveMQ management GUI
Login:
username: admin
password: admin
- Quay (if started)
Quay will be started in a single node, local storage configuration.
A random port will be forwarded from localhost, see `podman port` for the mapping.

localhost:7000 --- Traefik Web UI.
Good for troubleshooting http issues.

localhost:9090 --- Prometheus

localhost:3000 --- Grafana
Login:
username: admin
password: admin
```
- PostgreSQL
PostgreSQL will have a random port forwarded from localhost to the database server.
See `local-dev/clair/init.sql` for credentials and permissions and `podman port` for the mapping.

## Pushing to the Local Quay

As mentioned above, Quay is present at `localhost:8080`. You may navigate to this address and create a account. Creating an account named `admin` will ensure you are a super user. An email is required, but is not validated.
As mentioned above, Quay is forwarded to a random port on the host.
You can connect to the server on that port and create a account.
Creating an account named `admin` will ensure you are a super user.
An email is required, but is not validated.
You'll also need to create a namespace.

Once inside, you will create an organization named "clairv4-org". Currently Quay has to explicitly enable Clair v4 security scanning, which is done via an organization allowlist. "clairv4-org" is preconfigured in our local dev configuration.
To push to Quay, you'll need to exec into the skopeo container:

The easiest way to push to Quay is using podman:

```
podman pull ubuntu:latest
podman login --tls-verify=false localhost:8080 # use account created in above steps
podman tag ubuntu:latest localhost:8080/clairv4-org/testing:latest
podman push --tls-verify=false localhost:8080/clairv4-org/testing:latest
```sh
podman exec -it quay-skopeo /usr/bin/skopeo copy --dst-creds '<user>:<pass>' --dst-tls-verify=false <src> clair-quay:8080/<namespace>/<repo>:<tag>
```

Using docker to push is possible, however you will need to add "localhost:8080" as an insecure repository. See [Insecure Repository](https://docs.docker.com/registry/insecure/)

## Viewing Results

By default, Quay displays security scanner results on the Tags page of the given repository.

## Making changes to configuration

You may want to play with either Clair or Quay's configuration.
If so, the configuration files can be found inside the repository at
local-dev/quay/config.yaml` and `local-dev/clair/config.yaml`.
Any changes to the configs will require a restart of the relevant service. Take a look at the `Makefile` for the various restart targets.
If so, the configuration files can be found inside the repository at `local-dev/quay/config.yaml` and `local-dev/clair/config.yaml`.
Any changes to the configs will require a restart of the relevant service.
The quay-specific clair config is autogenerated, see the `Makefile`.

## Tearing it down

```
make local-dev-down
docker-compose down
```

will rip the entire environment down.


## Troubleshooting

The most common issue encountered when standing up the dev environment is port conflicts. Make sure that you do not have any other processes listening on any of the ports outlined above.
The most common issue encountered when standing up the dev environment is port conflicts.
Make sure that you do not have any other processes listening on any of the ports outlined above.

The second issue you may face is your Docker resource settings maybe too constrained to support the local dev stack. This is typically seen on Docker4Mac since a VM is used with a specific set of resources configured. See [Docker For Mac Manual](https://docs.docker.com/docker-for-mac/) for instructions on how to change these resources.
The second issue you may face is your Docker resource settings maybe too constrained to support the local dev stack.
This is typically seen on Docker4Mac since a VM is used with a specific set of resources configured.
See [Docker For Mac Manual](https://docs.docker.com/docker-for-mac/) for instructions on how to change these resources.

Lastly, you can view traefik's ui at `localhost:7000`. If traefik is reporting no routers or services its likely SELinux has blocked its access to `/var/run/docker.socket`. Place SELinux in permissive mode and restart the local development environment.
Lastly, you can view traefik's ui at `localhost:8080/dashboard/`.
152 changes: 28 additions & 124 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,143 +14,47 @@
docker ?= docker
docker-compose ?= docker-compose

# formats all imports to place local modules
# below out of tree modules
# Formats all imports to place local packages below out of tree packages.
.PHONY: goimports-local
goimports-local:
go list -f '{{$$d := .Dir}}{{range .GoFiles}}{{printf "%s/%s\n" $$d .}}{{end}}' ./... | xargs sed -i'' '/import (/,/)/{ /^$$/d }'
go list -f '{{.Dir}}' ./... | xargs goimports -local "$$(go list -m)" -w

# https://github.com/Mermade/widdershins used to convert openapi.yaml to markdown
# you'll need to have npx to run this gen.
.PHONY: gen-api-reference
gen-api-reference:
npx widdershins --search false --language_tabs 'python:Python' 'go:Golang' 'javascript:Javascript' --summary ./openapi.yaml -o ./Documentation/reference/api.md
# Use https://github.com/Mermade/widdershins to convert openapi.yaml to
# markdown. You'll need to have npx to run this.
Documentation/reference/api.md: openapi.yaml
npx widdershins --search false --language_tabs 'python:Python' 'go:Golang' 'javascript:Javascript' --summary $< -o $@

# start a local development environment.
# each services runs in it's own container to test service->service communication.
#
# local dev configuration can be found in "./local-dev/clair/config.yaml"
.PHONY: local-dev-up
local-dev-up: vendor
$(docker-compose) up -d traefik
$(docker-compose) up -d jaeger
$(docker-compose) up -d prometheus
$(docker-compose) up -d grafana
$(docker-compose) up -d rabbitmq
$(docker-compose) up -d activemq
$(docker-compose) up -d clair-db
$(docker) exec -it clair-db bash -c 'while ! pg_isready; do echo "waiting for postgres"; sleep 2; done'
$(docker-compose) up -d pgadmin
$(docker-compose) up -d indexer
$(docker-compose) up -d matcher
$(docker-compose) up -d notifier
$(docker-compose) up -d swagger-ui

.PHONY: local-dev-up-with-quay
local-dev-up-with-quay: vendor
## clair ##
$(docker-compose) up -d traefik
$(docker-compose) up -d jaeger
$(docker-compose) up -d prometheus
$(docker-compose) up -d grafana
$(docker-compose) up -d rabbitmq
$(docker-compose) up -d activemq
$(docker-compose) up -d clair-db
$(docker) exec -it clair-db bash -c 'while ! pg_isready; do echo "waiting for clair postgres"; sleep 2; done'
$(docker-compose) up -d pgadmin
$(docker-compose) up -d indexer-quay
$(docker-compose) up -d matcher
$(docker-compose) up -d notifier
$(docker-compose) up -d swagger-ui
## quay ##
$(docker-compose) up -d redis
$(docker-compose) up -d quay-db
$(docker) exec -it quay-db bash -c 'while ! pg_isready; do echo "waiting for quay postgres"; sleep 2; done'
$(docker) exec -it quay-db /bin/bash -c 'echo "CREATE EXTENSION IF NOT EXISTS pg_trgm" | psql -d quay -U quay'
$(docker-compose) up -d quay
local-dev/clair/quay.yaml: local-dev/clair/config.yaml
sed '/target:/s,webhook-target/,clair-quay:8443/secscan/notification,' <$< >$@

.PHONY: local-dev-restart-quay
local-dev-restart-quay:
$(docker-compose) up -d --force-recreate quay

# starts a local dev environment for testing notifier
# the notifier will create a notification on very notifier.poll_interval value in the local dev configuration.
#
# the notifier will deliver the notification to the configured deliverer in the local dev configuration.
# the default deliverer is rabbitmq/amqp
# Start a local development environment.
#
# local dev configuration can be found in "./local-dev/clair/config.yaml"
.PHONY: local-dev-notifier-test
local-dev-notifier-test: vendor
$(docker-compose) up -d traefik
$(docker-compose) up -d jaeger
$(docker-compose) up -d prometheus
$(docker-compose) up -d rabbitmq
$(docker-compose) up -d activemq
$(docker-compose) up -d clair-db
$(docker) exec -it clair-db bash -c 'while ! pg_isready; do echo "waiting for postgres"; sleep 2; done'
$(docker-compose) up -d notifier-test-mode
$(docker-compose) up -d swagger-ui

.PHONY: local-dev-notifier-test-restart
local-dev-notifier-test-restart: vendor
$(docker-compose) up -d --force-recreate notifier-test-mode

# Each service runs in its own container to test service-to-service
# communication. Local dev configuration can be found in
# "./local-dev/clair/config.yaml"
.PHONY: local-dev
local-dev: vendor
$(docker-compose) up -d
@printf 'postgresql on port:\t%s\n' "$$($(docker-compose) port traefik 5432)"

compose_profiles = $(patsubst %,local-dev-%,quay notifier debug)
.PHONY: $(compose_profiles)
local-dev-%: vendor
$(docker-compose) --profile $* up -d
@printf 'postgresql on port:\t%s\n' "$$($(docker-compose) port traefik 5432)"

local-dev-quay: local-dev/clair/quay.yaml vendor
CLAIR_CONFIG=$(<F) $(docker-compose) --profile quay up -d
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you think cleaning up the local-dev/clair/quay.yaml file here is a bit overreaching?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean "cleaning up?" It needs to exist while things are running, and needs to be slightly different from the standard config. It being a proper target and dependency at least makes sure it gets updated as the standard config changes.

@printf 'postgresql on port:\t%s\n' "$$($(docker-compose) port traefik 5432)"
@printf 'quay on port:\t%s\n' "$$($(docker-compose) port traefik 8443)"

.PHONY: vendor
vendor: vendor/modules.txt

vendor/modules.txt: go.mod
go mod vendor

# tear down the entire local development environment
.PHONY: local-dev-down
local-dev-down:
$(docker-compose) down

# restart the local development database, clearing all it's contents
# often a service should be restarted as well to run migrations on the now schemaless database.
.PHONY: local-dev-db-restart
local-dev-db-restart:
$(docker) kill clair-db && $(docker) rm clair-db
$(docker-compose) up -d --force-recreate clair-db

# restart the local development indexer, any local code changes will take effect
.PHONY: local-dev-indexer-restart
local-dev-indexer-restart:
$(docker-compose) up -d --force-recreate indexer

# restart the local development matcher, any local code changes will take effect
.PHONY: local-dev-matcher-restart
local-dev-matcher-restart:
$(docker-compose) up -d --force-recreate matcher

# restart the local development notifier, any local code changes will take effect
.PHONY: local-dev-notifier-restart
local-dev-notifier-restart:
$(docker-compose) up -d --force-recreate notifier

# restart all clair instances
.PHONY: local-dev-clair-restart
local-dev-clair-restart:
$(docker-compose) up -d --force-recreate indexer
$(docker-compose) up -d --force-recreate matcher
$(docker-compose) up -d --force-recreate notifier

# restart the local development rabbitmq
.PHONY: local-dev-rabbitmq-restart
local-dev-rabbitmq-restart:
$(docker-compose) up -d --force-recreate rabbitmq

# restart the local development swagger-ui, any local code changes will take effect
.PHONY: local-dev-swagger-ui-restart
local-dev-swagger-ui-restart:
$(docker-compose) up -d --force-recreate swagger-ui

# restart the local development swagger-ui, any local code changes will take effect
.PHONY: local-dev-traefik-restart
local-dev-traefik-restart:
$(docker-compose) up -d --force-recreate traefik

.PHONY: container-build
container-build:
$(docker) build -t clair-local:latest .
Expand Down
Loading