Switching to CloudFoundry v3 API

[Kump3r]

Overview

Switch to CF API v3, because of [RFC] CF API v2 End of Life

What this PR does / why we need it

Concourse issue #9006

Special notes for your reviewer

N/A

Does this PR introduce a user-facing change?

No, backwards compatible

[taylorsilva]

These two docs are helpful for reviewing:

• https://github.com/cloudfoundry/community/blob/rfc-cfapiv2-eol/toc/rfc/rfc-draft-cfapiv2-eol.md
• https://v3-apidocs.cloudfoundry.org/version/3.179.0/index.html#upgrade-guide

[taylorsilva]

One a first red through it looks like this migration is easy: https://v3-apidocs.cloudfoundry.org/version/3.179.0/index.html#users-and-roles-in-v3

I'll dive deeper into this once 7.12.0 is out the door. Thanks again for the PR!

[Kump3r]

I think this could be reviewed/tested together with #8970 as they don't seem to have any conflicts, based on my tests at least. This way we can "kill two birds with one stone" 🤣

[Kump3r]

Seems the v3 API info endpoint, does not provide the required information, to access the UAA login url, so it needed some rework. Check info-in-v3. Currently testing with the changes.

[Kump3r]

As it turns out all of the endpoints used are deprecated in v3 and the whole connector logic will need to be reworked. We will try to work on it, a couple of chunks at a time. Also if someone has time to contribute, I will be more than happy to test it as well. It looks like with v3 we will lower the number of requests, but will need to loop in memory trough larger objects.

[IvanChalukov]

Hi @taylorsilva, I reached out to @Kump3r, and after a brief discussion about the deprecated endpoints in v3, we’ve developed a potential solution for the issue. If you’re comfortable with the proposed approach, we’d be happy to contribute it.

---

CF API V2 Endpoints Used in current implementation

The following endpoints are deprecated in v3 (source: CF API V2 Docs):

• /v2/users/%s/spaces
• /v2/users/%s/audited_spaces
• /v2/users/%s/managed_spaces
• /v2/users/%s/organizations

---

Proposed Solution

To replace the deprecated endpoints, we suggest the following approachs:

1. With 2 requests for spaces and organizations:

   • For /v2/users/%s/organizations - use /v3/roles?user_guids=$USER_ID&types=organization_user and then get all organizations with /v3/organizations to map the name with the GUID.
   • For /v2/users/%s/spaces, /v2/users/%s/audited_spaces, and /v2/users/%s/managed_spaces - fetch all of them for a user with /v3/roles?user_guids=$USER_ID&types=space_developer,space_manager,space_auditor and then get all spaces with /v3/spaces to map the name with the GUID.

2. It could be used one endpoint request to collect organizations and spaces for a particular user /v3/roles?user_guids=$USER_ID&types=space_developer,space_manager,space_auditor,organization_user:

   • For /v2/users/%s/organizations - use output from above request and then get all organizations with /v3/organizations to map the name with the GUID.
   • For /v2/users/%s/spaces, /v2/users/%s/audited_spaces, and /v2/users/%s/managed_spaces - use output from above request and then get all spaces with /v3/spaces to map the name with the GUID.

---

Sample Curls

V2

curl -H "Authorization: $(cf oauth-token)" "<cf_api_url>/v2/users/$USER_ID/organizations"

v2 audited_spaces

curl -H "Authorization: $(cf oauth-token)" "<cf_api_url>/v2/users/$USER_ID/audited_spaces"

v2 spaces

curl -H "Authorization: $(cf oauth-token)" "<cf_api_url>/v2/users/$USER_ID/spaces"

v2 managed_spaces

curl -H "Authorization: $(cf oauth-token)" "<cf_api_url>/v2/users/$USER_ID/managed_spaces"

V3

curl -H "Authorization: $(cf oauth-token)" "<cf_api_url>/v3/users"
curl -H "Authorization: $(cf oauth-token)" "<cf_api_url>/v3/users/$USER_ID"

v3 orgs

curl -H "Authorization: $(cf oauth-token)" "<cf_api_url>/v3/roles?user_guids=$USER_ID&types=organization_user"
curl -H "Authorization: $(cf oauth-token)" "<cf_api_url>/v3/organizations"

v3 spaces

curl -H "Authorization: $(cf oauth-token)" "<auth_url>/v3/roles?user_guids=$USER_ID&types=space_developer,space_manager,space_auditor&per_page=5000"
curl -H "Authorization: $(cf oauth-token)" "<cf_api_url>/v3/spaces"

[Kump3r]

I prefer option number 2, to quote and change a little LOTR:

"One Request to rule them all, One Request to find them, One Request to bring them all and in the darkness bind them."
—The Ring's inscription, translated

[IvanChalukov]

We collaborated with @Kump3r on the implementation and updated it accordingly. Additionally, we tested it locally. Could you take a quick look and share your feedback? If everything looks good to you, we can proceed with the unit tests.

[taylorsilva]

Some things to keep in mind:

• I have no CloudFoundry (CF) infra to test this against
• I have not used CF in quite a few years, so my knowledge about it is quite rusty
• My personal interest in CF is very low, but I know a lot of people in the community are CF users, therefore I want to respect the CF community's needs

That out of the way, there are some things I'm trying to understand about this PR and what's going on with the CF API.

I am making an assumption that there will be some people that may be on a CF that only has the v2 API. If this PR was merged in, would it block those people from upgrading their Concourse? Is this PR a breaking change for anyone that uses the CF connector?

I think that's my biggest concern right now. Maybe it's not valid? Has the v3 API been around for a long time? Is it WILD to think that someone would be running a CF today that doesn't have the v3 API enabled already? Any insight you guys have on that would be helpful.

I can see there's a lot of refactoring work going on here too, totally fine with that stuff. Really appreciate any manual testing you guys can do.

[Kump3r]

To try and answer your question. Concourse v2 API End of life RFC is merged in the CF community So anyone using CloudFoundry will have to migrate to v3 anyway from my understanding. The CF community is certainly deprecating v2 API, as mentioned in the [RFC] CF API v2 End Of Life. Meaning that soon, people that are using the CF Connector for authentication, will be unable to login to Concourse. From my perspective it looks like we need to migrate Concourse, as to when v2 stops working, people will be able to continue using it. I am not sure if there is a way to reach out to the people using CF auth and inform them, that as a prerequisite they need to update/migrate to v3. But for sure I think this should be mentioned as a breaking change I guess (not sure how you handled such deprecations of dependancies in the past). Hopefully that cleared things up. If not, I will be more than happy to reach the people supporting CF internally and ask them additional questions, or also ask the CF community in the RFC.
For your other question let me know if I can test certain scenarios. Also is there someone that might be able to test. I guess since @IvanChalukov also played around with this from his end, at least we got the 4 eyes principal covered.

[Kump3r]

Status update from my side. I was able to collaborate with @IvanChalukov and

1. We managed to test a bunch of scenarios based on CF Authorization, with a couple left to be tested:
   • --cf-user=USERNAME - ✅
   • --cf-org=ORG_NAME - ❌ - doesn't work with previous releases as well. Check Concourse-9052
   • --cf-space=ORG_NAME:SPACE_NAME - ✅ everyone in the space has access
   • --cf-space-with-auditor-role=ORG_NAME:SPACE_NAME - ✅ only auditors in the space have access.
   • --cf-space-guid=SPACE_GUID - ✅
2. The CF connector doesn't have unit tests per se, but rather acceptance/integration test, because you setup a CF connector simple http server and expect certain behaviours, so we adjusted the tests/mocks, to match the existing assertions. Currently tests are passing I am not certain if there is a coverage tool we should check align with.

[IvanChalukov]

I tested the scenario using the following option:

• --cf-space-guid=SPACE_GUID - ✅ Everyone in the space has access.

It looks good from my perspective. Perhaps someone from the Concourse team could review it further and proceed with the PR review.

[taylorsilva]

LGTM - Thanks for testing everything out!