Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update buildchain/CI to build the Storage Operator #1410

Closed
slaperche-scality opened this issue Jul 11, 2019 · 0 comments
Closed

Update buildchain/CI to build the Storage Operator #1410

slaperche-scality opened this issue Jul 11, 2019 · 0 comments
Assignees
@slaperche-scality
Labels
complexity:medium Something that requires one or few days to fix topic:build Anything related to building steps topic:ci Continuous integration and build orchestration topic:storage Issues related to storage
Milestone
MetalK8s 2.4.0-al...

Comments

@slaperche-scality
Copy link
Contributor

slaperche-scality commented Jul 11, 2019 •
edited
Loading

Component:

buildchain, CI

Why this is needed:

We need to be able to build and deploy a Kubernetes operator written in Go.

What should be done:

  • Update the documentation about the new building dependencies (go, operator-sdk 0.9.0, …)
  • Add new dependencies in the CI build container
  • Build the container image for the operator
  • Lint the Go code of the operator (at least enforce go-fmt, maybe linting as well)
  • Ensure operator-sdk generate k8s and operator-sdk generate openapi don’t result in a diff, which would otherwise mean codegen wasn’t run as part of the commit/changes

Test plan:

Being able to build an ISO that contains the operator container, both locally and on the CI.
@slaperche-scality slaperche-scality added moonshot topic:ci Continuous integration and build orchestration topic:build Anything related to building steps complexity:medium Something that requires one or few days to fix labels Jul 11, 2019
@slaperche-scality slaperche-scality self-assigned this Jul 12, 2019
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
ci/worker: add operator-sdk to the build container
7bc3bb1 
We now need to build the Storage Operator, which we built using the
Operator SDK.
Consequently, we need to add the Operator SDK to the build image.

Since the SDK depends on Go and dep, we also install those.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/image: split the build and save step
9b45499 
That way, we will be able to selectively override one while reusing the
other (e.g. keep saving with Skopeo, but build the image using the
operator-sdk instead of Docker directly).

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/deps: add a property for the command name
cbddcb7 
Until now, the command-name was simply the name of the enum variant in
lowercase, but this will no longer be the case with
OPERATOR_SDK/operator-sdk.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/deps: workaround a pylint bug
7a34d8a 
Pylint seems to think that `name` is a method instead of a string, which
is wrong.
See pylint-dev/pylint#2062

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/image: build the storage-operator image
30b0d50 
Add a new image to be built: the container image for the Storage
operator.
Since it's very similar to a LocalImage, except for the building part
(we use the operator-sdk command instead of Docker directly, because we
need to compile the Go code), we introduce a new class (OperatorImage)
that inherits from LocalImage and overload the `_do_build` method.

Note that we take care of enabling Go module and compiling from the
`storage-operator` directory.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/lint: add a task to lint the Go code
17f8b50 
For now, we only check that the code is properly formated using `gofmt`.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/lint: update outdated docstring
123f0d2 
We added new tasks but the schema wasn't updated.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/lint: add title for lint tasks
8c44ce8 
Until now, lint tasks weren't run alongside "build" task, and thus the
discrepancy in the display wasn't a bother.
But now, we have this kind of output, which is unsightly:

    -- lint:python
    -- lint:shell
    -- lint:yaml
    .  CHECK CMD    gofmt
    -- lint:go

So let's define a custom title for the lint tasks to have something like
that instead:

    -- LINT         python
    -- LINT         shell
    -- LINT         yaml
    .  CHECK CMD    gofmt
    -- LINT         go

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/format: add a task to auto-format the code
3d74ac4 
For now, we only have a task for Go code.
If we ever use `black` for Python, we will add a new subtask here.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build: compute gofmt args only once
5f1d73e 
The arguments list was computed twice: once in `lint` and once in
`format`.
Let's use a constant instead.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build: add an helper for title with sub-task name
bfafb33 
This reduce code duplication between `lint` and `format`.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/utils: fix a typo in the docstring
725e4fd 
Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
doc/build: update the requirements for operators
5b5e4fc 
Also add a minimum required version for Docker (according to the README
of the Operator SDK)

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/lint: add an action to check generated file
8e5e0b5 
We commit the files generated by the `operator-sdk` command, as such we
want to make sure that when modifying the "source" file we don't forget
to regenerate the scaffolding files.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/codegen: add a task to run Go codegen
8ff8a5e 
For now, I plan to create one task per language, for the sake of
consistency with `lint` and `format`.
If we have more operator in the future, it may make more sense to have
one subtask per operator: we can change when/if it happens.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
operator/storage: re-generate the files with operator-sdk
9b78b95 
We were using an operator-sdk straight from the master branch.
Now that a proper release was made (0.9.0) we use it and thus should
update the already generated files with it.

Closes: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
@slaperche-scality slaperche-scality mentioned this issue Jul 15, 2019
Merged
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
ci/worker: add operator-sdk to the build container
867a3e6 
We now need to build the Storage Operator, which we built using the
Operator SDK.
Consequently, we need to add the Operator SDK to the build image.

Since the SDK depends on Go and dep, we also install those.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/image: split the build and save step
feee51d 
That way, we will be able to selectively override one while reusing the
other (e.g. keep saving with Skopeo, but build the image using the
operator-sdk instead of Docker directly).

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/deps: add a property for the command name
aace58c 
Until now, the command-name was simply the name of the enum variant in
lowercase, but this will no longer be the case with
OPERATOR_SDK/operator-sdk.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/deps: workaround a pylint bug
cc1adea 
Pylint seems to think that `name` is a method instead of a string, which
is wrong.
See pylint-dev/pylint#2062

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 15, 2019
@slaperche-scality
build/image: build the storage-operator image
665b0f1 
Add a new image to be built: the container image for the Storage
operator.
Since it's very similar to a LocalImage, except for the building part
(we use the operator-sdk command instead of Docker directly, because we
need to compile the Go code), we introduce a new class (OperatorImage)
that inherits from LocalImage and overload the `_do_build` method.

Note that we take care of enabling Go module and compiling from the
`storage-operator` directory.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 18, 2019
@slaperche-scality
build: add an helper for title with sub-task name
385da6a 
This reduce code duplication between `lint` and `format`.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 18, 2019
@slaperche-scality
build/utils: fix a typo in the docstring
4530c0b 
Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 18, 2019
@slaperche-scality
doc/build: update the requirements for operators
abc2cb3 
Also add a minimum required version for Docker (according to the README
of the Operator SDK)

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 18, 2019
@slaperche-scality
build/lint: add an action to check generated file
a7c0be8 
We commit the files generated by the `operator-sdk` command, as such we
want to make sure that when modifying the "source" file we don't forget
to regenerate the scaffolding files.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 18, 2019
@slaperche-scality
build/codegen: add a task to run Go codegen
48b53cf 
For now, I plan to create one task per language, for the sake of
consistency with `lint` and `format`.
If we have more operator in the future, it may make more sense to have
one subtask per operator: we can change when/if it happens.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 18, 2019
@slaperche-scality
build: remove vendor filtering
4c77cbb 
Since we're using Operator SDK 0.9.0, we no longer have a `vendor`
directory, so there is no need to filter it and its content.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 18, 2019
@slaperche-scality
ci/worker: pre-download Go dependencies
d718e75 
We need to download a fairly big amount of data when building operator
using the Operator SDK (~3G of source repositories).

In order to avoid stressing the CI (and wasting bandwidth) by repeatedly
doing this at build time for every single build, let's pre-download all
that stuff and keep it in the container image.

Note that since we need to copy the `go.sum` and `go.mod` files, we need
to set the build context in the storage-operator directory.

The `pod-linter` is also affected as we use the `operator-sdk` command
during our linting phase.

Closes: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
slaperche-scality added a commit that referenced this issue Jul 18, 2019
@slaperche-scality
build/utils: return a closure from title_with_*
c9af392 
That way, the callers don't need to use a lambda or an extra function.
This make the code simpler.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build/image: split the build and save step
5e2ba06 
That way, we will be able to selectively override one while reusing the
other (e.g. keep saving with Skopeo, but build the image using the
operator-sdk instead of Docker directly).

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build/deps: add a property for the command name
6e5eafc 
Until now, the command-name was simply the name of the enum variant in
lowercase, but this will no longer be the case with
OPERATOR_SDK/operator-sdk.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build/deps: workaround a pylint bug
5e4fd60 
Pylint seems to think that `name` is a method instead of a string, which
is wrong.
See pylint-dev/pylint#2062

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build/image: build the storage-operator image
a1f9e3a 
Add a new image to be built: the container image for the Storage
operator.
Since it's very similar to a LocalImage, except for the building part
(we use the operator-sdk command instead of Docker directly, because we
need to compile the Go code), we introduce a new class (OperatorImage)
that inherits from LocalImage and overload the `_do_build` method.

Note that we take care of enabling Go module and compiling from the
`storage-operator` directory.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
ci/worker: add operator-sdk to the build container
0f973a3 
We now need to build the Storage Operator, which we built using the
Operator SDK.
Consequently, we need to add the Operator SDK to the build image.

Since the SDK depends on Go and dep, we also install those.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build/lint: add a task to lint the Go code
1cc3784 
For now, we only check that the code is properly formated using `gofmt`.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
ci/worker: add operator-sdk to the lint container
2a7c61d 
Since the lint step for Go uses `gofmt` and `operator-sdk`, we also need
to install them in the lint worker.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build/lint: update outdated docstring
ac0fdfe 
We added new tasks but the schema wasn't updated.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build/lint: add title for lint tasks
cbe3ce7 
Until now, lint tasks weren't run alongside "build" task, and thus the
discrepancy in the display wasn't a bother.
But now, we have this kind of output, which is unsightly:

    -- lint:python
    -- lint:shell
    -- lint:yaml
    .  CHECK CMD    gofmt
    -- lint:go

So let's define a custom title for the lint tasks to have something like
that instead:

    -- LINT         python
    -- LINT         shell
    -- LINT         yaml
    .  CHECK CMD    gofmt
    -- LINT         go

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build/format: add a task to auto-format the code
08c041f 
For now, we only have a task for Go code.
If we ever use `black` for Python, we will add a new subtask here.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build: compute gofmt args only once
e8c07df 
The arguments list was computed twice: once in `lint` and once in
`format`.
Let's use a constant instead.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build: add an helper for title with sub-task name
0d72be6 
This reduce code duplication between `lint` and `format`.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build/utils: fix a typo in the docstring
7866d9c 
Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
doc/build: update the requirements for operators
5af1a72 
Also add a minimum required version for Docker (according to the README
of the Operator SDK)

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build/lint: add an action to check generated file
755ad16 
We commit the files generated by the `operator-sdk` command, as such we
want to make sure that when modifying the "source" file we don't forget
to regenerate the scaffolding files.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build/codegen: add a task to run Go codegen
f0c048b 
For now, I plan to create one task per language, for the sake of
consistency with `lint` and `format`.
If we have more operator in the future, it may make more sense to have
one subtask per operator: we can change when/if it happens.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build: remove vendor filtering
a6159cb 
Since we're using Operator SDK 0.9.0, we no longer have a `vendor`
directory, so there is no need to filter it and its content.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
ci/worker: pre-download Go dependencies
de6f8b2 
We need to download a fairly big amount of data when building operator
using the Operator SDK (~3G of source repositories).

In order to avoid stressing the CI (and wasting bandwidth) by repeatedly
doing this at build time for every single build, let's pre-download all
that stuff and keep it in the container image.

Note that since we need to copy the `go.sum` and `go.mod` files, we need
to set the build context in the storage-operator directory.

The `pod-linter` is also affected as we use the `operator-sdk` command
during our linting phase.

Closes: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
MonPote pushed a commit that referenced this issue Jul 25, 2019
@slaperche-scality @MonPote
build/utils: return a closure from title_with_*
d15f98b 
That way, the callers don't need to use a lambda or an extra function.
This make the code simpler.

Refs: #1410
Signed-off-by: Sylvain Laperche <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@slaperche-scality slaperche-scality

Labels
complexity:medium Something that requires one or few days to fix topic:build Anything related to building steps topic:ci Continuous integration and build orchestration topic:storage Issues related to storage
Projects
None yet
Milestone
MetalK8s 2.4.0-alpha1
Development

No branches or pull requests
2 participants
@NicolasT @slaperche-scality