Tests should be able to set up and tear down CF resources, avoiding manual and stateful configuration

[deniseyu]

To use this checker, you have to spend at least 5 minutes manually copying GUIDs into the config file. This is error-prone and time-consuming, and feels like an inefficient use of Human Cycles ™️ .

It would be great if I could just pass it a minimal config file that only contains my CF API, admin username, and admin password. The test suite should dynamically create a new CF org, space, broker with pre-configured plans (and register the broker), service instances, and bindings for each test run. At the end of the test run, it should force-delete the broker and the CF org, recursively getting rid of all associated resources.

This would lower the barrier to using this checker and cuts down on the general noise caused by stateful tests.

[leonwanghui]

Hi @deniseyu , I think it's a great idea if we can reach to this goal, but something has to be mentioned here is that this test suite is not prepared only for cf but other platforms (svcat, openshift or fusionstage), which means we should treat the context defined in the specification separately.

So my suggestion is that we should define a reusable and extensible test framework for users to deploy. For now I don't have much clue, but we could work together if you are interested : )

[leonwanghui]

Hi @deniseyu the test framework has been refactored recently by #50 , could you have a try and give some feedback? If you are ok with the change, this issue would be closed.