Continuous Releases

[robaerd]

• more Docker images, e.g. CentOS/openSuse/Arch
• Add following external test suites: kconfig and gsettings-tests
• Improve Docker tutorials and add a description that the docker images are run as non-root (and maybe also the reason for this).
• Build Server docu
• Integration of a Windows build server
• Jenkins: Make sending emails if build fail more reliable Build Server stuff #160
• Extracting tests which should be run always and tests which should be run only for releases and/or development to speed up build times testscr_check_kdb_internal_suite: very slow at least on a7 #3512
• automatic publishing of Rust bindings

low prio

• Integration of fuzzer (try out AFL #185)
• Building .dmg packages for macOS (via cpack)
• Add Solaris build agent
• Providing an Elektra gem provide Elektra gem #1293
• https://conan.io/ (C/C++ package manager)
• http://flatpak.org/ (application distribution)
• publish flathub

[markus2330]

Thank you for creating the issue!

I reordered a bit but it actually was already nearly perfectly prioritized. I removed "Continuous Release" header, as everything is Continuous Releases.

As very first step it would be good to implement what we currently do manually every release, see also doc/todo/RELEASE here a list:

• adding more logs and other info in https://github.com/ElektraInitiative/ftp
  • list and sizes all files to be installed
  • changes of infos/status in the plugins
  • traces of tools
  • output of test cases
  • output of ABI tests (maybe wait for Thursday to know what this is about, in most of 0.9.* these tests will fail)

These scripts we could already execute on Thursday for 0.9.3.

The second step would be building of Docker images, this is also already relevant for 0.9.3: it is always great to have Docker images with the currently released Elektra installed. One of these docker images would be: #2913

The third step would be to build DEB (Debian+Ubuntu) and RPM (Fedora+Cent OS) packages for 0.9.3. Here most of the work is already done, so it is "only" about adaptions to the current release. This can be done manually the first time and also released to https://github.com/ElektraInitiative/ftp

What I am missing in the list above is Maven packages. I think they are quite important for Elektra at TU Wien, as Java is the main language there. Is this something you do not want to do?

But most important always is: whenever master does not build or some build server has hiccup (like #3422), this has highest priority as this slows down the whole development of Elektra. Obviously, in the beginning you will get help to fix the issues.

[robaerd]

Thanks for reordering and the detailed description of the next steps!

These scripts we could already execute on Thursday for 0.9.3.

I will try to get as much as possible done till Thursday.

What I am missing in the list above is Maven packages. I think they are quite important for Elektra at TU Wien, as Java is the main language there. Is this something you do not want to do?

You're right, Maven packages are really important. I will update the issue and add Maven packages to it.

But most important always is: whenever master does not build or some build server has hiccup (like #3422), this has highest priority as this slows down the whole development of Elektra. Obviously, in the beginning you will get help to fix the issues.

I understand, thanks.

[markus2330]

Thank you very much for also adding Maven, I think this is more than enough now! The list is quite impressive. 💖

I made now all the points flat as the structure was arbitrary (Adding Arm agent was outside but adding Solaris was inside of "Build Server") and integrated sub-issues that contained only a few points (#160 and #1463).

Then I divided between "in progress" and "backlog", the "in progress" related to what we do tomorrow. We always talk a bit about each task before work on it. For now try to read as much docu as possible and to understand the individual bits of our build infrastructure. I recommend you to also start your written thesis as soon as possible.

[markus2330]

As discussed in #3458 I added

• Build Server docu
• Look at webui Example Web UI Instance is Down #2913 and go integration Integrate go-elektra binding to main repo #3458

in the backlog.

[markus2330]

Btw. please always feel free to update the list when plans change. I now changed that cpack as the only thing in progress (and moved RPM packages to this point as hopefully the RPM packages will be also done via cpack)

[markus2330]

I was currently working on extending the release automation (ABI tests, external test suites etc) but I can of course also create RPM packages for this release. With our current CPack setup this probably shouldn't be that much work.

Yes, I thought also that extending RPM would not be much work and now you are acclimatized with CPack. Did you also plan to do dmg with cpack? Anything else that makes sense to do with cpack?

The ABI or external tests are probably a too big task to be done before the upcoming release.

[robaerd]

Did you also plan to do dmg with cpack?

Yes, I also plan to do dmg with CPack.

Anything else that makes sense to do with cpack?

I'm not sure how well Elektra is supported on Windows, but if we ever decide to also build packages for Windows, the approach would be very similar to the dmg packages with CPack.

[markus2330]

I'm not sure how well Elektra is supported on Windows,

It is working, with some limitations: E.g. the resolver is trivial, not supporting atomar changes of several configs. It is a goal to continue or even extend the support, as the KDE people also support Windows and they could not use Elektra everywhere if we do not support Windows.

but if we ever decide to also build packages for Windows, the approach would be very similar to the dmg packages with CPack.

We already decided to have a Windows build agent and it would be strange to not also produce some packages if we already build it.

Actually we already produce Windows "packages" (zip archive): from the mingw build.

[robaerd]

@markus2330 I'm currently having trouble with the licensing of the RPM packages. You mentioned in my last PR (#3583) that we should avoid maintaing multiple licensing files. CPack RPM apparently doesn't support the %licenses macro in which we could simply add our THIRD_PARTY_LICENSES file and reference to it. Therefore we would need to state all of our liceneses, seperated with an "and", in the "License:" field. For this I wrote a script to extract all the licenses of THIRD_PART_LICENSES and bring it to the desired format. But RPM packages require different short names for the licenses, so rpmlint throws multiple warnings because of this e.g. "Apache 2.0" should be "ASL 2.0" etc.

Should we

1. Just ignore this for now, since these are only warnings, the licenses are stated but only in the wrong format
2. Add the used licenses explicitly to the licese field. Should then only be following licenses: BSD, zlib, MIT. (OFL, Apache 2.0 etc are not needed, because not packaged)
3. Write a script to map the indiviual licenses to rpm license short names

The LICENSE and THIRD_PARTY_LICENSES are currently installed in each rpm package to /usr/share/licenses/${component}, so it should be probably possible to still reference them via comment e.g. for the licensing breakdown.

[markus2330]

I think it is okay to ignore it, as we do not give wrong information. If you already have a script anyway and it is only about a few mappings, you could extend the script to map these licenses. I see it as low priority issue, though.

[markus2330]

As discussed I removed "- [ ] Maven packages" as it will done in the Java overhaul by someone else.

Otherwise, even though the progress was immense, there are still quite many tasks open. I split "backlog" and "low prio" to give you a better perspective.

[robaerd]

@markus2330 I'm currently in the process of adding the external test suites to the release pipeline and ran into some issues.

Kconfig requires the Elektra version to be =0.9.1. With the current version of Elektra only 20% of the tests pass.

How can I get the gsettings-tests to run? There wasn't a "test" make target nor instructions in the README.

However the Elektra tests of oyranos are still working. Maybe we should add them to the pipeline?

[markus2330]

Kconfig requires the Elektra version to be =0.9.1. With the current version of Elektra only 20% of the tests pass.

Thank you, as discussed, can you please create an issue for that? It is probably not about the keyname overhaul then, see #3570

How can I get the gsettings-tests to run? There wasn't a "test" make target nor instructions in the README.

@mpranj can you help out here?

However the Elektra tests of oyranos are still working. Maybe we should add them to the pipeline?

Yes, that would be great. @beku will be happy.

[markus2330]

@mpranj ping, any update on how to run gsettings-tests? (Or even kconfig tests.)

[mpranj]

any update on how to run gsettings-tests? (Or even kconfig tests.)

No update yet. It's still on my TODO, just no time to do it yet.

[markus2330]

I reprioritized the tasks. For macOS build agent we first need to evaluate if we can have satisfactory results with a small time budget, otherwise I would also put this to the backlog.

[kodebach]

Not directly related to releases, but fits the general CI discussion here: Is there a way to tell Jenkins to prioritize the stages up to "Main Build" over the "Full Build" stage when assigning worker nodes to jobs?

It can be very annoying, if there are one or two PRs that use all the nodes for their "Full Build" stage and then other PRs don't even get the basic feedback of the release notes and formatting checks. Similarly or maybe even more annoying, if your PR doesn't get workers and only the website build is missing. Also we could in general prioritize jobs that are expected to run quickly (release notes, formatting, etc.) over other jobs.

[robaerd]

There exist a few such plugins (e.g. Jenkins Priority Sorter Plugin ), but it appears that none of them have an interface to integrate them in a Jenkinsfile (see issue).
Maybe the assignment of priorities is still possible through another way. I will take a closer look at this in the next days.

[markus2330]

I added #3966 here and moved a job done to done.

[github-actions]

I mark this stale as it did not have any activity for one year. I'll close it in two weeks if no further activity occurs. If you want it to be alive again, ping by writing a message here or create a new issue with the remainder of this issue.
Thank you for your contributions 💖

[github-actions]

I closed this now because it has been inactive for more than one year. If I closed it by mistake, please do not hesitate to reopen it or create a new issue with the remainder of this issue.
Thank you for your contributions 💖