Module API calls do not seem to be cached

[shoddyguard]

We recently tried to upgrade from 4.5.2 to 4.16.0 of the provider but found that our plan times went from taking 5 minutes to 1hr+ and after much debugging we think it's a problem in the way module calls are handled.

What seems to happen is that anytime a module is called it does not cache the results of its API call which results in the API call being repeated every single time the module is called which leads to the bloated plan times.

In our case the module we are using is to add our special CI/CD groups to repositories programmatically by giving us a nice reusable module that we can call when needed (see below for the code).

When running a terraform plan and sticking fiddler in the middle to intercept HTTPS we can see that every time the module is called it's making GET requests to the /organizations/<org_id>/team/<team_id>/repos endpoint and downloading the full list of repos (of which there are 400+ so it takes 8-10s!) and /orgs/red-gate/teams/<team_name> to fetch the members (of which there are far less), you can also see the calls being made in the debug log.

Reverting back to 4.5.2 and running a plan it seems that the GET requests to /organizations/<org_id>/team/<team_id>/repos are gone but the requests to /orgs/red-gate/teams/<team_name> remain (we likely hadn’t noticed this as the teams only have a couple of members and the responses come back in 1s or less).

I would have expect that these requests should happen once and then be cached as they aren't going to change.

Due to this error the Debug log ended up several GB's in size so I can't included it, but I have it saved in case there's anything you want me to look for.

Terraform Version

1.0.8

Affected Resource(s)

• github_team_repository (specifically when used in modules)

Terraform Configuration Files

Module teamcity-access.tf:

variable "repo_name" {
  type = string
  description = "The name of the github repo to connect to teamcity"
}

variable "github-commit-hook-secret" {
  type = string
  description = "Secret used to secure our Github -> Teamcity commit hooks."
}

variable "repo_permission" {
  type = string
  default = "pull"
  description = "Teamcity user permission. push or pull. Default to pull"
}

data "github_team" "BuildServersreadaccess" {
  slug = "build-servers-read-access"
}

data "github_team" "TeamCitystatusupdates" {
  slug = "teamcity-status-updates"
}

resource "github_team_repository" "BuildServersreadaccess" {
  team_id = data.github_team.BuildServersreadaccess.id
  repository = var.repo_name
  permission = var.repo_permission
}

resource "github_team_repository" "TeamCitystatusupdates" {
  team_id = data.github_team.TeamCitystatusupdates.id
  repository = var.repo_name
  permission = "push"
}

resource "github_repository_webhook" "Teamcity_commit" {
  repository = var.repo_name
  active = true
  events = ["push", "pull_request"]
  configuration {
    url          = "**REDACTED**"
    secret       = var.github-commit-hook-secret
    content_type = "json"
    insecure_ssl = false
  }
}

terraform {
  required_providers {
    github = {
      source = "integrations/github"
      version = "~> 4.16.0"
    }
  }
  required_version = "~> 1.0.7"
}

Example of a repo configuration in our repos.tf:

resource "github_repository" "ngit" {
  name          = "ngit"
  visibility    = "public"
  description   = "Fork of https://github.com/mono/ngit which has been dead since 2014"
  homepage_url  = ""
  has_issues    = false
  has_wiki      = false
  has_projects  = false
  has_downloads = false
  auto_init     = false
}

module "ngit-teamcity-access" {
  source                    = "../terraform-modules/teamcity-access"
  repo_name                 = github_repository.ngit.name
  github-commit-hook-secret = var.github-commit-hook-secret
}

Expected Behavior

The group members and repository lists should have been updated once and cached

Actual Behavior

Every time the module is called the group members and repository list gets updated

Steps to Reproduce

1. terraform plan

References

• I wonder if this may be the cause of some of the woes in Slow performance when managing dozens of repositories #567?

In case it's helpful please see the following fiddler screen shots showing the repeated calls and time to last block:

[shoddyguard]

WAIT!
I see our mistake now, we're using a data source to calculate the team ID which is why it's getting calculated each time. 🤦‍♂️
Changing this to use a slug has resolved this for us.
Sorry for the false alarm!