Quarkus CLI update test

[mocenas]

Summary

This PR does not work, as it requires quarkus-qe/quarkus-test-framework#1188 to be effective here.

This makes a proposal how Quarkus CLI test could look in the future. Idea is to create common tests and other support methods in parent class and making some kind of a "framework". For each new tested version we can add new class, which should be more or else simple.

Edit: This PR requires quarkus-qe/quarkus-test-framework#1236 and other things in framework 1.5.0.Beta13. So do not merge until this is released and updated in TS.

Please select the relevant options.

• Bug fix (non-breaking change which fixes an issue)
• Dependency update
• Refactoring
• Backport
• New scenario (non-breaking change which adds functionality)
• This change requires a documentation update
• This change requires execution against OCP (use run tests phrase in comment)

Checklist:

• Methods and classes used in PR scenarios are meaningful
• Commits are well encapsulated and follow the best practices

[michalvavrik]

run tests

[michalvavrik]

please ignore run tests comment, I just need to test OCP CI and there was no better PR atm.

[mocenas]

I've added actual tests. @gtroitsk Please have a look at them.

[gtroitsk]

I've added actual tests. @gtroitsk Please have a look at them.

I will look at it later today

[gtroitsk]

I afraid that actual TC is not enough. I think we need to test wider set of changes, scope is up to you. Maybe Michal V. or Rostislav have some comments.
For testing properties ideally we should create real app reflecting the customer's usage scenario including test coverage (taken from TP). We can't relate that new names are syntactically correct.
Also can you please add an option/tests for RHBQ apps.

[mocenas]

I afraid that actual TC is not enough. I think we need to test wider set of changes, scope is up to you. Maybe Michal V. or Rostislav have some comments. For testing properties ideally we should create real app reflecting the customer's usage scenario including test coverage (taken from TP). We can't relate that new names are syntactically correct. Also can you please add an option/tests for RHBQ apps.

I agree that we should tests entire apps. I'm working on it (created first app for hibernate update). I plan to add more for methods and packages renames.

Could you be more specific what would you like to see as TC "wider set of changes"? I'm following recipes and migration guides and testing stuff from them.

Yes I will add RBHQ updates.

[gtroitsk]

I afraid that actual TC is not enough. I think we need to test wider set of changes, scope is up to you. Maybe Michal V. or Rostislav have some comments. For testing properties ideally we should create real app reflecting the customer's usage scenario including test coverage (taken from TP). We can't relate that new names are syntactically correct. Also can you please add an option/tests for RHBQ apps.

I agree that we should tests entire apps. I'm working on it (created first app for hibernate update). I plan to add more for methods and packages renames.

Could you be more specific what would you like to see as TC "wider set of changes"? I'm following recipes and migration guides and testing stuff from them.

Yes I will add RBHQ updates.

Under TC "wider set of changes" I meant test as must as possible recipes and changes from migration guides. You follow it, so I am fine.

[mocenas]

Testing quarkus udpate on RHBQ has a problem that I didn't forsee. Updating between two RHBQ versions requires to have both their repos available and (AFAIK) in single local maven repo. This doesn't fit our workflow for quarkus-test-suite and IMHO it is not worth it to change the workflow.

I think it would be better to drop this PR and move update tests into separate GH repo, for which we can setup appropriate workflow. WDYT @gtroitsk @rsvoboda @michalvavrik ?

[michalvavrik]

Testing quarkus udpate on RHBQ has a problem that I didn't forsee. Updating between two RHBQ versions requires to have both their repos available and (AFAIK) in single local maven repo. This doesn't fit our workflow for quarkus-test-suite and IMHO it is not worth it to change the workflow.

If that's a true it's a problem. I'll be honesty, I don't understand why updating from released RHBQ to RHBQ that is being released would required the released RHBQ locally. I'd expect that update command use remote Maven repository. Can you clarify this a bit?

[mocenas]

After updating local quarkus CLI setup to work with RHBQ, current test design work surprisingly well with RHBQ.

[mocenas]

I've added support for testing updates to specific bom version.

[mocenas]

Can you please review this @gtroitsk @michalvavrik ? The tests should be done, I'm keeping it as a draft before framework is released.
PS: I know it seems that a lot of migration changes are not covered, but these are usually new feature/deprecated/removed/breaking change/etc, which means how have to manually update you app anyway, so I don't see way in testing it.

[michalvavrik]

I skimmed through changes, they LGTM. I am afraid I don't have proper time to in depth review in the next week and half. AFAICT it's ok. Maybe you will need additional reviewer. Thanks!

[michalvavrik]

@mocenas 8 commits for 2 new files is bit excessive, please squash your commits.

[mocenas]

@mocenas 8 commits for 2 new files is bit excessive, please squash your commits.

I intended to squash them during merge, but OK done.

[michalvavrik]

LGTM

[mocenas]

Failures are not related, so I'm merging this.