Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Cloud SQL connection creation when deploying cloudfunctions2 #12578

Open
AlbertMoser opened this issue Sep 16, 2022 · 7 comments
Open

Allow Cloud SQL connection creation when deploying cloudfunctions2 #12578

AlbertMoser opened this issue Sep 16, 2022 · 7 comments

Comments

@AlbertMoser
Copy link

AlbertMoser commented Sep 16, 2022

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

When deploying gen2 cloudfunctions which have to access a SQL database, the database connection needs to be manually added after the deployment. This makes the whole process rather tedious. It seems like the underlying functionality already exists in cloudrun according to this example. However, looking at this issue it currently doesn't work for cloudfunctions2.
For our setup, this is crucial since we sometimes deploy all of our services in a different region and adding the SQL connections manually would make it error prone.

New or Affected Resource(s)

  • google_cloudfunctions2_function

Potential Terraform Configuration

resource "google_cloudfunctions2_function" "function" {
  name = "function"
  location = "us-central1"
  description = "a new function"

  build_config {
    runtime = "nodejs16"
    entry_point = "helloPubSub"  # Set the entry point 
    environment_variables = {
        BUILD_CONFIG_TEST = "build_test"
    }
    source {
      storage_source {
        bucket = google_storage_bucket.bucket.name
        object = google_storage_bucket_object.object.name
      }
    }
  }

  service_config {
    max_instance_count  = 3
    min_instance_count = 1
    available_memory    = "256M"
    timeout_seconds     = 60
    environment_variables = {
        SERVICE_CONFIG_TEST = "config_test"
    }
    ingress_settings = "ALLOW_INTERNAL_ONLY"
    all_traffic_on_latest_revision = true
    service_account_email = google_service_account.account.email
  }
  metadata {
    annotations = {
        "run.googleapis.com/cloudsql-instances" = google_sql_database_instance.instance.connection_name
        "run.googleapis.com/client-name"        = "terraform"
    }
  }

  event_trigger {
    trigger_region = "us-central1"
    event_type = "google.cloud.pubsub.topic.v1.messagePublished"
    pubsub_topic = google_pubsub_topic.topic.id
    retry_policy = "RETRY_POLICY_RETRY"
  }
}

References

@DrFaust92
Copy link
Contributor

it doesnt look like cloudfunction api directly exposes this. there might be an extra step needed here according to : https://cloud.google.com/sql/docs/mysql/connect-functions#public-ip-default.
might be simpler with private conn: https://cloud.google.com/sql/docs/mysql/connect-functions#private-ip

@rileykarson
Copy link
Collaborator

@shuyama1 to investigate the required changes

@verygoodsoftwarenotvirus

AFAICT this page is the structure Terraform is sending to GCP to create the resource, and it doesn't have any references to Cloud SQL metadata, etc. I think this might just be a (very unfortunate) shortcoming of GCP's API.

@matsko
Copy link

matsko commented Mar 1, 2023

I was able to get this to work by this workaround:

locals {
  cloud_function_service = google_cloudfunctions2_function.default.service_config[0].service
  cloud_function_service_name = replace(local.cloud_function_service, "/projects\\/[-a-z0-9]+\\/locations\\/[-a-z0-9]+\\/services\\//", "")
}

resource "google_cloud_run_service" "default" {
  name                       = local.cloud_function_service_name
  project                    = var.project
  location                   = var.region

  template {
    metadata {
      annotations = {
        "run.googleapis.com/cloudsql-instances"   = var.cloudsql_connection
      }
    }
  }

  lifecycle {
    ignore_changes = [
      id,
      status,
      timeouts,
      template[0].metadata[0].annotations["run.googleapis.com/client-name"],
      template[0].metadata[0].annotations["gcf-v2-enable-pubsub-retries"],
      template[0].metadata[0].annotations["run.googleapis.com/startup-cpu-boost"],
      template[0].metadata[0].annotations["autoscaling.knative.dev/minScale"],
      template[0].metadata[0].annotations["client.knative.dev/user-image"],
      template[0].metadata[0].annotations["cloudfunctions.googleapis.com/trigger-type"],
      template[0].metadata[0].annotations["run.googleapis.com/cpu-throttling"],
      template[0].metadata[0].annotations["run.googleapis.com/vpc-access-connector"],
      template[0].metadata[0].annotations["run.googleapis.com/vpc-access-egress"],
    ]
  }
}

Then what you need to do is import the cloud_run service that powers the function and bind it to the google_cloud_run_service resource.

google_cloud_run_service.default region/name-of-newly-created-function-cloud-run-service

@vyacheslav31
Copy link

vyacheslav31 commented Apr 13, 2023

lifecycle {
    ignore_changes = [
      id,
      status,
      timeouts,
      template[0].metadata[0].annotations["run.googleapis.com/client-name"],
      template[0].metadata[0].annotations["gcf-v2-enable-pubsub-retries"],
      template[0].metadata[0].annotations["run.googleapis.com/startup-cpu-boost"],
      template[0].metadata[0].annotations["autoscaling.knative.dev/minScale"],
      template[0].metadata[0].annotations["client.knative.dev/user-image"],
      template[0].metadata[0].annotations["cloudfunctions.googleapis.com/trigger-type"],
      template[0].metadata[0].annotations["run.googleapis.com/cpu-throttling"],
      template[0].metadata[0].annotations["run.googleapis.com/vpc-access-connector"],
      template[0].metadata[0].annotations["run.googleapis.com/vpc-access-egress"],
    ]
  }

@matsko Would this work for the first run ? We cannot import the state of a Cloud Run instance that hasn't been created yet. Any workarounds for this?

@manudawber
Copy link

@vyacheslav31 this worked for me:

resource "null_resource" "bootstrap_cloudsql" {
  depends_on = [google_cloudfunctions2_function.placeholder]
  provisioner "local-exec" {
    command = <<-EOT
      PROJECT_ID="${var.project_id}"
      REGION="${google_cloudfunctions2_function.placeholder.location}"
      SERVICE_NAME="${google_cloudfunctions2_function.placeholder.name}"
      CLOUDSQL_CONNECTION_NAME="${data.google_sql_database_instance.db-shared.connection_name}"
      
      check_cloud_run() {
        gcloud run services list --platform managed --region "$1" --project "$2" --filter "metadata.name=$3" --format "value(metadata.name)" || true
      }
      
      check_cloud_sql() {
        gcloud sql instances list --project "$1" --filter "connectionName:$2" --format "value(connectionName)" || true
      }
      
      check_cloud_sql_added() {
        gcloud run services describe "$1" --platform managed --region "$2" --project "$3" --format "value(metadata.annotations['run.googleapis.com/cloudsql-instances'])" | grep -w "$4" || true
      }
      
      update_cloud_run() {
        gcloud run services update "$1" --platform managed --region "$2" --add-cloudsql-instances "$3" --project "$4"
      }
      
      SERVICE_EXISTS=$(check_cloud_run "$REGION" "$PROJECT_ID" "$SERVICE_NAME")
      if [ -z "$SERVICE_EXISTS" ]; then
        echo "Cloud Run service not found."
        exit 0
      fi
      
      CLOUDSQL_EXISTS=$(check_cloud_sql "$PROJECT_ID" "$CLOUDSQL_CONNECTION_NAME")
      if [ -z "$CLOUDSQL_EXISTS" ]; then
        echo "Cloud SQL instance not found."
        exit 0
      fi
      
      ALREADY_ADDED=$(check_cloud_sql_added "$SERVICE_NAME" "$REGION" "$PROJECT_ID" "$CLOUDSQL_CONNECTION_NAME")
      if [ -n "$ALREADY_ADDED" ]; then
        echo "Cloud SQL instance already added to the Cloud Run service."
        exit 0
      fi
      
      update_cloud_run "$SERVICE_NAME" "$REGION" "$CLOUDSQL_CONNECTION_NAME" "$PROJECT_ID"
    EOT
  }
}

Source: https://www.reddit.com/r/Terraform/comments/12hszd5/comment/jg4e867

@vyacheslav31
Copy link

vyacheslav31 commented Apr 21, 2023

@vyacheslav31 this worked for me:

@manudawber That is my solution, I am the same person from reddit lol. Thanks for posting. 👍

modular-magician added a commit to modular-magician/terraform-provider-google that referenced this issue Dec 18, 2024
… details (hashicorp#12578)

[upstream:dd34343c5a5c2e1a23cc652ea1eafaae54b14019]

Signed-off-by: Modular Magician <[email protected]>
modular-magician added a commit that referenced this issue Dec 18, 2024
… details (#12578) (#20734)

[upstream:dd34343c5a5c2e1a23cc652ea1eafaae54b14019]

Signed-off-by: Modular Magician <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants