[Decommission Hub, meom-ige] SWOT Ocean Pangeo Team

[jnywong]

Summary

This hub should have been decommissioned July 2022. AirTable currently lists this hub as active. Not to be confused with the DRAKKAR workshop) in Feb 2023.

References:

• https://2i2c.freshdesk.com/a/tickets/1726
• [Incident] MEOM hub is unreachable #1423

Info

• Community Representative: @lesommer
• Link to New Hub issue: Hub for Univ. Grenoble satellite team #207 (I think it's this one – the DRAKKAR workshop hub has already been torn down)
• Proposed end date: 30 June – we have not been able to get a response from the hub champion through FreshDesk (since the contract finished in Jul 2022!), so this seems like a good enough buffer time
  • Very likely that we cannot complete Phase I below as a result.

Task List

Phase I

• Confirm with Community Representative that the hub is no longer in use and it's safe to decommission
  We tried but failed - the contract has ended in 2022 though so we are moving ahead.
• Confirm if there is any data to migrate from the hub before decommissioning
  • If yes, confirm where the data should be migrated to
    • Confirm a 2i2c Engineer has access to the destination in order to complete the data migration
  • If no, confirm it is ok to delete all the data stored in the user home directories
    We tried but failed.

Phase II - Hub Removal

(These steps are described in more detail in the docs at https://infrastructure.2i2c.org/hub-deployment-guide/hubs/delete-hub/)

• Manage existing home directory data (migrate data from the hub or delete it)
• Manage existing cloud bucket data (migrate data, or delete it)
• Delete the hub's authentication application on GitHub or CILogon (note CILogon removal requires the hub config in place)
• Remove the appropriate config/clusters/<cluster_name>/<hub_name>.values.yaml files. A complete list of relevant files can be found under the appropriate entry in the associated cluster.yaml file.
• Remove the associated hub entry from the config/clusters/<cluster_name>/cluster.yaml file.
• Remove the hub deployment
  • helm --namespace HUB_NAME delete HUB_NAME
  • kubectl delete namespace HUB_NAME
  • TIP: Run deployer use-cluster-credentials <cluster_name> before running the above commands

Phase III - Cluster Removal

This phase is only necessary for single hub clusters.

• Remove the cluster's datasource from the central Grafana with:
  • deployer grafana central-ds remove <cluster_name>
• Run terraform plan -destroy and terraform apply from the appropriate workspace, to destroy the cluster
• Delete the terraform workspace: terraform workspace delete <NAME>
• Delete the terraform values file under the projects folder associated with the relevant cloud provider (e.g. terraform/gcp/projects/ for GCP)
• Remove the associated config/clusters/<cluster_name> directory and all its contents
• Remove the cluster from CI:
  • deploy-hubs.yaml
  • deploy-grafana-dashboards.yaml
• Remove A record from Namecheap account

[AIDEA775]

If I understand correctly, we will decommission the staging and prod hub and the cluster too, right?

I have this ERROR when I try to delete homes:

ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=403, message=This API method requires billing to be enabled. Please enable billing on project #meom-ige-cnrs by visiting https://console.developers.google.com/billing/enable?project=meom-ige-cnrs then retry. If you enabled billing for this project recently, wait a few minutes for the action to propagate to our systems and retry.
[...]
CalledProcessError: Command '['gcloud', 'container', 'clusters', '--zone=us-central1-b', '--project=meom-ige-cnrs', 'get-credentials', 'meom-ige-cluster']' returned non-zero exit status 
1.

When I enter the GCP console, I see this:

It's okay if I select any option? I can't run deployer use-cluster-credentials $CLUSTER_NAME either.

[consideRatio]

@AIDEA775 reading the support ticket, it seems indeed that staging / prod representing "SWOT Ocean Pangeo Team ([staging|prod])" should be deleted, and that everything should go - especially considering now that the contract ended 2022.

I think the course of action is to attach it to our billing account and then proceed to tear down as you plan, and its OK to do so.

[consideRatio]

Note that deleting home directory data can be done as part of deleting all cloud infra provisioned as everything is to go, so you can ignore that for the step of deleting things for any individual hub.

[consideRatio]

@AIDEA775 I checked the checkboxes that I perceive to be irrelevant if we also will do phase 3 that deletes everything - not just an individual hub. In "Phase 2" there is only one left about Delete the hub's authentication application on GitHub or CILogon (note CILogon removal requires the hub config in place)

[AIDEA775]

Okey, now when I run deployer cilogon-client delete --client-id cilogon:/client_id/... meom-ige staging I got:

CILogon records are different than the client app stored in the configuration file. Consider updating the file.

[consideRatio]

Okey, now when I run deployer cilogon-client delete --client-id cilogon:/client_id/... meom-ige staging I got:

CILogon records are different than the client app stored in the configuration file. Consider updating the file.

I have no knowledge on this :/ i think it could be ok to proceed, we may end up with a remnant thing, but it could be acceptable i think

[sgibson91]

@AIDEA775 You could also decrypt the secrets file for the staging hub, that will contain the Client ID too. Try providing that to the command and see if that gives you a different output?

[AIDEA775]

Oh! Yes! In this commit consideRatio@ee87fc7 the auth was migrated to GitHubOAuthenticator without removing the CILogon client. Rolling back to the old secrets did the trick.