feat(Packaging): Build and release deb and rpm packages #592

jsirianni · 2022-03-02T02:05:35Z

Description of Changes

Added deb and rpm package building to Goreleaser.

Set binary name to stanza and disabled no_unique_dist_dir as this is the name that will be used during package install (/usr/bin/stanza).

Package install will

Add stanza binary to /usr/bin/stanza, this is a slight divergence from how we traditionally install stanza.
Add plugins (already managed by Goreleaser)
Add default config (config was taken from install script)
Run pre install script: Configure stanza user
Run post install script: Handles service creation and final permissions

Added CINC Test Kitchen config

Wraps vagrant for easy testing
As simple as kitchen test debian-10
Ensures Stanza was installed correctly

Documented how to use CINC / Test Kitchen.
Documented package install process.
Documented how to use a root user when installing with a package manager.

Questions

On release, I believe the binary file names will be

stanza_linux_arm64_1.6.0_linux_arm64
stanza_windows_amd64_1.6.0_windows_amd64.exe
stanza_darwin_amd64_1.6.0_darwin_amd64
stanza_linux_amd64_1.6.0_linux_amd64
stanza_darwin_arm64_1.6.0_darwin_arm64

Which is not going to work with the current install script (line 309):

...
  if [ -z "$version" ] ; then
    agent_download_url="$url/latest/download/${BINARY_NAME}_${os}_${os_arch}"
    plugins_download_url="$url/latest/download/${PLUGINS_PACKAGE}"
  else
    agent_download_url="$url/download/v$version/${BINARY_NAME}_${os}_${os_arch}"
    plugins_download_url="$url/download/v$version/${PLUGINS_PACKAGE}"
  fi
...

Should that be addressed with this PR?

Please check that the PR fulfills these requirements

Tests for the changes have been added (for bug fixes / features)
Docs have been added / updated (for bug fixes / features)
CI passes

codecov · 2022-03-02T02:11:15Z

Codecov Report

Merging #592 (6e26cf8) into master (3b23e5c) will decrease coverage by 1.70%.
The diff coverage is 88.89%.

@@            Coverage Diff             @@
##           master     #592      +/-   ##
==========================================
- Coverage   76.69%   74.99%   -1.70%     
==========================================
  Files         132      133       +1     
  Lines        8308    10086    +1778     
==========================================
+ Hits         6371     7563    +1192     
- Misses       1473     2053     +580     
- Partials      464      470       +6

Impacted Files	Coverage Δ
operator/buffer/buffer.go	`75.86% <ø> (-1.92%)`	⬇️
operator/buffer/memory.go	`71.60% <ø> (-3.74%)`	⬇️
...builtin/input/aws/cloudwatch/cloudwatch_persist.go	`75.00% <ø> (-6.82%)`	⬇️
operator/builtin/input/azure/event_hub.go	`0.00% <0.00%> (ø)`
operator/builtin/input/azure/event_hub_config.go	`90.91% <ø> (-1.09%)`	⬇️
operator/builtin/input/file/finder.go	`100.00% <ø> (ø)`
operator/builtin/input/file/reader.go	`53.05% <0.00%> (-5.73%)`	⬇️
operator/builtin/input/http/config.go	`97.56% <ø> (-0.29%)`	⬇️
operator/builtin/input/journald/journald.go	`58.76% <ø> (-1.83%)`	⬇️
operator/builtin/input/udp/udp.go	`73.49% <ø> (-1.51%)`	⬇️
... and 159 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a5b004b...6e26cf8. Read the comment docs.

.gitignore

build/package/config.yaml

cpheps · 2022-03-02T13:20:55Z

docs/development.md

@@ -24,6 +24,32 @@ make build-darwin-amd64

 Tests can be run with `make test`.

+## Integration Tests with CINC Test Kitchen


What is the purpose of the CINC stuff? I've not seen this before do we have a standard for using it elsewhere?

CINC is an open source fork of Chef. Chef's license changed as is not suitable for us. I use CINC's Auditor (chef inspec) to validate post install state.

We do not have a standard, however, I think it is important that we test the end state. You can run kitchen test <distro>.

This config is using vagrant + virtual box, so it will not work for everyone. I think we can use vmware workstation or maybe GCE but support is not as great. Their virtual box implementation is very nice.

➜ stanza git:(packages-deb-rpm) kitchen test debian-10 -----> Starting Test Kitchen (v3.2.2) -----> Cleaning up any prior instances of <default-debian-10> -----> Destroying <default-debian-10>... Finished destroying <default-debian-10> (0m0.00s). -----> Testing <default-debian-10> -----> Creating <default-debian-10>... Bringing machine 'default' up with 'virtualbox' provider... ==> default: Importing base box 'bento/debian-10'... ==> default: Matching MAC address for NAT networking... ... (Reading database ... 29511 files and directories currently installed.) Preparing to unpack .../stanza_1.6.0_linux_amd64.deb ... Unpacking stanza (1.6.0) ... Setting up stanza (1.6.0) ... detected service file directory: /usr/lib/systemd/system configured systemd service Created symlink /etc/systemd/system/multi-user.target.wants/stanza.service → /lib/systemd/system/stanza.service. Downloading files from <default-debian-10> Finished converging <default-debian-10> (0m1.88s). -----> Setting up <default-debian-10>... Finished setting up <default-debian-10> (0m0.00s). -----> Verifying <default-debian-10>... Loaded tests from {:path=>".home.jsirianni.git.stanza.build.package.test"} Profile: tests from {:path=>"/home/jsirianni/git/stanza/build/package/test"} (tests from {:path=>".home.jsirianni.git.stanza.build.package.test"}) Version: (not specified) Target: ssh://[email protected]:2222 File /opt/observiq/stanza ✔ mode is expected to cmp == "0755" ✔ owner is expected to eq "stanza" ✔ group is expected to eq "stanza" ✔ type is expected to cmp == "directory" File /opt/observiq/stanza/plugins ✔ mode is expected to cmp == "0755" ✔ owner is expected to eq "stanza" ✔ group is expected to eq "stanza" ✔ type is expected to cmp == "directory" File /opt/observiq/stanza/stanza.db ✔ mode is expected to cmp == "0600" ✔ owner is expected to eq "stanza" ✔ group is expected to eq "stanza" ✔ type is expected to cmp == "file" File /opt/observiq/stanza/stanza.log ✔ mode is expected to cmp == "0644" ✔ owner is expected to eq "stanza" ✔ group is expected to eq "stanza" ✔ type is expected to cmp == "file" File /opt/observiq/stanza/config.yaml ✔ mode is expected to cmp == "0640" ✔ owner is expected to eq "stanza" ✔ group is expected to eq "stanza" ✔ type is expected to cmp == "file" File /usr/bin/stanza ✔ mode is expected to cmp == "0755" ✔ owner is expected to eq "root" ✔ group is expected to eq "root" ✔ type is expected to cmp == "file" Service stanza ✔ is expected to be installed ✔ is expected to be enabled ✔ is expected to be running Test Summary: 27 successful, 0 failures, 0 skipped Finished verifying <default-debian-10> (0m1.41s). -----> Destroying <default-debian-10>... ==> default: Forcing shutdown of VM... ==> default: Destroying VM and associated drives... Vagrant instance <default-debian-10> destroyed. Finished destroying <default-debian-10> (0m2.59s). Finished testing <default-debian-10> (0m38.25s). -----> Test Kitchen is finished. (0m38.77s)

I agree testing is worth while. I'm ok with this for now but my fears are:

bloat in our repos (maybe we move this tooling to an internal repo that just runs these types of tests)

Complex tooling that doesn't work everywhere.

As an action item can you write up a proposal for using this or submit something to the standards repo if we want to move forward with this? I'm ok with Stanza being the poc repo for this but I would like some discussion before we put this heavy tooling in everywhere.

Sounds good 👍

build/post.sh

.goreleaser.yml

jsirianni added 5 commits March 1, 2022 20:24

build deb and rpm packages. use test kitchen for testing locally

7731868

fix fedora version and remove suse for now

fcd9409

final os support

78e09a3

document how to install a package from url. Document how to run as root

055e825

newline

2accbf5

jsirianni marked this pull request as ready for review March 2, 2022 02:30

jsirianni requested a review from a team as a code owner March 2, 2022 02:30

jsirianni requested a review from StefanKurek March 2, 2022 02:30

cpheps reviewed Mar 2, 2022

View reviewed changes

newline

6e26cf8

cpheps approved these changes Mar 2, 2022

View reviewed changes

BinaryFissionGames reviewed Mar 2, 2022

View reviewed changes

.goreleaser.yml Show resolved Hide resolved

BinaryFissionGames approved these changes Mar 2, 2022

View reviewed changes

jsirianni merged commit 49d1d31 into master Mar 2, 2022

jsirianni deleted the packages-deb-rpm branch March 2, 2022 18:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(Packaging): Build and release deb and rpm packages #592

feat(Packaging): Build and release deb and rpm packages #592

jsirianni commented Mar 2, 2022 •

edited

Loading

codecov bot commented Mar 2, 2022 •

edited

Loading

cpheps Mar 2, 2022

jsirianni Mar 2, 2022

jsirianni Mar 2, 2022

cpheps Mar 2, 2022

jsirianni Mar 2, 2022

		@@ -24,6 +24,32 @@ make build-darwin-amd64

		Tests can be run with `make test`.

		## Integration Tests with CINC Test Kitchen

feat(Packaging): Build and release deb and rpm packages #592

feat(Packaging): Build and release deb and rpm packages #592

Conversation

jsirianni commented Mar 2, 2022 • edited Loading

Description of Changes

Questions

Please check that the PR fulfills these requirements

codecov bot commented Mar 2, 2022 • edited Loading

Codecov Report

cpheps Mar 2, 2022

Choose a reason for hiding this comment

jsirianni Mar 2, 2022

Choose a reason for hiding this comment

jsirianni Mar 2, 2022

Choose a reason for hiding this comment

cpheps Mar 2, 2022

Choose a reason for hiding this comment

jsirianni Mar 2, 2022

Choose a reason for hiding this comment

jsirianni commented Mar 2, 2022 •

edited

Loading

codecov bot commented Mar 2, 2022 •

edited

Loading