Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Managed Certificates #607

Open
SophCarp opened this issue Jan 30, 2023 · 57 comments
Open

Managed Certificates #607

SophCarp opened this issue Jan 30, 2023 · 57 comments
Assignees
Labels
enhancement New feature or request roadmap This feature is on the roadmap

Comments

@SophCarp
Copy link

SophCarp commented Jan 30, 2023

ETA: Public Preview by end of March 2023

@SophCarp SophCarp added enhancement New feature or request roadmap This feature is on the roadmap Networking Related to ACA networking labels Jan 30, 2023
@ghost ghost added the Needs: triage 🔍 Pending a first pass to read, tag, and assign label Jan 30, 2023
@SophCarp
Copy link
Author

Related to #509

@cemerick
Copy link

cemerick commented Mar 1, 2023

This addresses the only shortfall ACA has vs. Cloud Run IME, so really looking forward to it!

Q: do you anticipate ACA managed certificates to have any limitations re: subdomain depth? Obviously "no" is ideal 😉

@anthonychu anthonychu removed the Networking Related to ACA networking label Mar 6, 2023
@anthonychu anthonychu self-assigned this Mar 6, 2023
@source-studio
Copy link

@SophCarp - is there any update on when this feature will enter public preview?

@anthonychu
Copy link
Member

anthonychu commented Apr 7, 2023

We'll have portal support and announce public preview very soon. You can use the CLI if you want to try it now. Requires a publicly available container app.


Container Apps supports apex domains and subdomains. Each domain type requires a different DNS record type and validation method.

Domain type Record type Validation method Notes
Apex domain A record HTTP An apex domain is a domain at the root level of your domain. For example, if your DNS zone is contoso.com, then contoso.com is the apex domain.
Subdomain CNAME CNAME A subdomain is a domain that is part of another domain. For example, if your DNS zone is contoso.com, then www.contoso.com is an example of a subdomain that can be configured in the zone.
  1. Log in to Azure with the Azure CLI.

    az login
    
  2. Next, install the Azure Container Apps extension for the CLI.

    az extension add --name containerapp --upgrade
    
  3. Verify that your container app has HTTP ingress enabled.

    az containerapp ingress show -n <CONTAINER_APP_NAME> -g <RESOURCE_GROUP_NAME>
    

    If ingress isn't enabled, enable it with these steps:

    az containerapp ingress enable -n <CONTAINER_APP_NAME> -g <RESOURCE_GROUP_NAME> \
            --type external --target-port <TARGET_PORT> --transport auto
    

    Replace <CONTAINER_APP_NAME> with the name of your container app, <RESOURCE_GROUP_NAME> with the name of the resource group that contains your container app, and <TARGET_PORT> with the port that your container app is listening on.

  4. If you're configuring an apex domain, get the IP address of your Container Apps environment.

    az containerapp env show -n <ENVIRONMENT_NAME> -g <RESOURCE_GROUP_NAME> -o tsv --query "properties.staticIp"
    

    Replace <ENVIRONMENT_NAME> with the name of your environment, and <RESOURCE_GROUP_NAME> with the name of the resource group that contains your environment.

  5. If you're configuring a subdomain, get the automatically generated domain of your container app.

    az containerapp show -n <CONTAINER_APP_NAME> -g <RESOURCE_GROUP_NAME> -o tsv --query "properties.configuration.ingress.fqdn"
    

    Replace <CONTAINER_APP_NAME> with the name of your container app, and <RESOURCE_GROUP_NAME> with the name of the resource group that contains your container app.

  6. Get the domain verification code.

    az containerapp show -n <CONTAINER_APP_NAME> -g <RESOURCE_GROUP_NAME> -o tsv --query "properties.customDomainVerificationId"
    

    Replace <CONTAINER_APP_NAME> with the name of your container app, and <RESOURCE_GROUP_NAME> with the name of the resource group that contains your container app.

  7. Using the DNS provider that is hosting your domain, create DNS records based on the record type you selected using the values shown in the Domain validation section. The records point the domain to your container app and verify that you own it.

    • If you're configuring an apex domain, create the following DNS records:

      Record type Host Value
      A @ The IP address of your Container Apps environment
      TXT asuid The domain verification code
    • If you're configuring a subdomain, create the following DNS records:

      Record type Host Value
      CNAME The subdomain (for example, www) The automatically generated domain of your container app
      TXT asuid. followed by the subdomain (for example, asuid.www) The domain verification code
  8. Add the domain to your container app.

    az containerapp hostname add --hostname <DOMAIN_NAME> -g <RESOURCE_GROUP_NAME> -n <CONTAINER_APP_NAME>
    

    Replace <DOMAIN_NAME> with the domain name you want to add, <RESOURCE_GROUP_NAME> with the name of the resource group that contains your container app, and <CONTAINER_APP_NAME> with the name of your container app.

  9. Configure the managed certificate and bind the domain to your container app.

    az containerapp hostname bind --hostname <DOMAIN_NAME> -g <RESOURCE_GROUP_NAME> -n <CONTAINER_APP_NAME> --environment <ENVIRONMENT_NAME> --validation-method <VALIDATION_METHOD>
    

    Replace <DOMAIN_NAME> with the domain name you want to add, <RESOURCE_GROUP_NAME> with the name of the resource group that contains your container app, <CONTAINER_APP_NAME> with the name of your container app, and <ENVIRONMENT_NAME> with the name of your environment.

    • If you're configuring an A record, replace <VALIDATION_METHOD> with HTTP.
    • If you're configuring a CNAME, replace <VALIDATION_METHOD> with CNAME.

    It may take several minutes to issue the certificate and add the domain to your container app.

  10. Once the operation is complete, navigate to your domain to verify that it's accessible.


@cemerick
Copy link

cemerick commented Apr 7, 2023

9. The command prompts you for a validation method.

Hit a wall here. It asked for the validation mechanism (TXT, CNAME, HTTP), and none worked. Got an error message in each case of Invalid validation method for domain 'XXXXXX'. Supported validation method(s) for the domain are: CNAME,HTTP,TXT..

@anthonychu
Copy link
Member

Sorry it looks like an az containerapp hostname add step was missing. I've added it.

@cemerick
Copy link

cemerick commented Apr 7, 2023

@anthonychu Thanks for that. I was able to get to the finish line, but I was not able to get the interactive validation method prompt to work; adding --validation-method CNAME to the bind step did work, though.

Any chance this is available via ARM/bicep yet?

@torosent torosent changed the title Roadmap: Managed Certificates Managed Certificates Apr 8, 2023
@source-studio
Copy link

source-studio commented Apr 8, 2023

Thanks @anthonychu .

However, I'm not sure where I'm going wrong, but I created a new environment and container app, both simply called containerapp, added the hostname successfully, but when I attempt to bind the certificate, it appears to initiate the process but then fail after a couple of seconds, claiming the environment can't be found. I've tried multiple validation methods:

az containerapp hostname bind --hostname <hostname> -n containerapp -g containerapp -e containerapp -v CNAME

Argument '--validation-method' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Creating managed certificate 'mc-containerapp-<hostname>-2779' for <hostname>.
It may take up to 20 minutes to create and issue a managed certificate.
(ManagedEnvironmentNotFound) Environment containerapp was not found.

UPDATE:

It appears this was because I hadn't set the --location parameter - there was a disparity between my configured defaults and where I'd provisioned the container app.

This does beg the question though, why does the location need to explicitly specified here, if it can be inferred from the container app. Indeed, why does the environment also need to specified, if this can be inferred in the same way?

@source-studio
Copy link

source-studio commented Apr 9, 2023

Further to this, I tried using the REST API to provision a managed certificate (via Pulumi), but I believe one of the property names is incorrect in the specification - specifically, domainControlValidation should be validationMethod.

@mburumaxwell
Copy link

Used the CLI to create a managed certificate and bind to a new hostname as per steps above. It worked but the certificate chain seems incomplete. Only the leaf/final certificate is returned to all TLS validation tools (https://decoder.link/, https://www.sslshopper.com/ssl-checker.html, and https://www.geocerts.com/ssl-checker).

Is this a bug or by design? [I may have lost touch with how certificates and PKI works]

image

@cemerick
Copy link

cemerick commented May 3, 2023

@mburumaxwell I think you're hitting #709

@torosent torosent moved this from In-Progress (Development) to Public Preview (Shipped and Improving) in Azure Container Apps Roadmap May 23, 2023
@jurepurgar
Copy link

Does anybody know how can we configure managed certificate via bicep or ARM? I could not find any documentation.

@delconis
Copy link

The CLI didn't work for us, we had to use the UI to complete the Custom Domain/Managed Certificate. This was due to our Container App and Container App Environment not being in the same resource group and the CLI only accepting/looking in a single resource group.

I feel the CLI shouldn't require the Container App Environment name and/or resource group as the Container App has the references and a Container App only belongs to 1 Container App Environment.

@thomas11
Copy link

thomas11 commented Jun 5, 2023

@anthonychu seconding what @jurepurgar said above: can we have some documentation for the API to use managed certificates? They don't seem to be reflected in the Azure spec. The latest definition of CustomDomain has only a certificateId property.

@loadaverage
Copy link

I'm wondering why certificate validation for Container Apps is so much difficult (literally broken) comparing to Static Web Apps, where validation works with one CNAME record pointed to Default Host name of Static App domain.

subdomain.custom-domain.tld -> static-web-app.azurestaticapps.net

If this records is exist in Azure Cloud DNS, certificate will be issued right away.

In my case, I have custom domain already on Azure Cloud DNS and I expected that Container Apps certificates should be easily created just by pointing custom (sub)domain to the CNAME of Managed Environment.

For me, this makes Container Apps totally unusable.

@cemerick
Copy link

It's really unfortunate that this remains really very broken, and even worse that there's been zero comment AFAICT from MS staff here for three months after the feature was marked "done" for Preview purposes.

Having a feature like this be available via interactive methods (portal, CLI) is cute, but absent options for robust automation (obviously bicep, but ideally others ofc), I don't know how useful a "Preview" phase can be, as people simply won't be able to use it at scale / beyond tinkerings.

For those following along, our workaround has been to put all of our container apps behind Cloudflare, let it manage our edge certificates, and bind a single Cloudflare-issued "origin" cert to every container app. This is obviously not an ideal arrangement, but it at least allows us to fully automate deployments without shelling to CLIs, performing repeated deploys, or worrying about other related containerapp cert bugs like #709.

@loadaverage
Copy link

Hey, @cemerick. I also had an idea about pointing subdomains to Load Balancer with SNI support to keep Container Apps running under TLS. But not going outside of Azure.
Anyway, that's probably first time I can see that so important feature as Managed Certificates is not supported in Cloud API without chaining CLI commands or doing multi-apply with Terraform or so, really disappointing for me.

@cemerick
Copy link

But not going outside of Azure.

Yeah, I'm not happy about it either. Obviously hurts latency, and just another point of failure.

If you (or anyone else) figures an in-Azure workaround that yields good bicep automation and doesn't involve provisioning yet another load balancer or whatever, please do post about it. 😄

@Adamation
Copy link

We are currently chaining CLI AZ commands in our deploy pipeline (Non-Prod), although working, we are consequently wiping and reapplying certs on each Bicep deployment and burning through issued certs at some pace. We really need to be able to bake this into Bicep before we could use this in any greater capacity.

@vinisoto
Copy link
Collaborator

Agreed that we want to make Managed Certs deployment automatable. We are looking into our options and will provide an update within the next couple of days. Thanks.

@jtatum
Copy link

jtatum commented Jul 25, 2023

This is solved in other services that use managed certificates by providing multiple resources, rather than putting everything in the single resource. See for instance https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/app_service_managed_certificate. The same applies in API/bicep/etc since it's the underlying API that's split out.

@floriankoch
Copy link

Agreed that we want to make Managed Certs deployment automatable. We are looking into our options and will provide an update within the next couple of days. Thanks.

Any news here?

@Adamation
Copy link

Agreed that we want to make Managed Certs deployment automatable. We are looking into our options and will provide an update within the next couple of days. Thanks.

Any news here?

+1

This is becoming a big blocker on our pipelines.

@deen13
Copy link

deen13 commented Jul 27, 2023

@Adamation we managed to work around this by using a small bash script in our deployment pipeline. This however has the consequence of a short downtime on each deployment.

# Deploying Managed Certificates via Bicep is not possible yet
# https://github.com/microsoft/azure-container-apps/issues/607

if az containerapp hostname list -g "$resourceGroup" -n frontend-"${shortEnvironmentName}" | jq -e '. | length == 0' >/dev/null; then
  if [[ $shortEnvironmentName == "prod" ]]; then
    domain="my-prod-domain.com"
  else
    domain="my-staging-domain.com"
  fi

  echo "Adding the hostname to the container app"
  az containerapp hostname add --hostname "$domain" -g "$resourceGroup" -n frontend-"${shortEnvironmentName}" >/dev/null
fi

if az containerapp env certificate list -g "$resourceGroup" -n "$appEnvironmentName" | jq -e '. | length == 0' >/dev/null; then
  echo "Creating the certificate in the app environment"

  az containerapp env certificate create --hostname "$domain" -n "$appEnvironmentName" -g "$resourceGroup" -v CNAME >/dev/null

  timeout_seconds=180
  end_time=$((SECONDS + timeout_seconds))
  while [ $SECONDS -lt $end_time ]; do
    if az containerapp env certificate list -g "$resourceGroup" --name "$appEnvironmentName" | jq -e '.[0].properties.provisioningState == "Succeeded"' >/dev/null; then
      echo "😌 Condition met: Certificate provisioning state is Succeeded."
      break
    else
      echo "😴 Waiting for the certificate to reach provisioning provisioningState 'Succeeded'..."
      sleep 10
    fi
  done

  if [ $SECONDS -ge $end_time ]; then
    echo "Timeout reached. Aborting after $timeout_seconds seconds."
  fi
fi

echo "Binding the hostname..."
az containerapp hostname bind --hostname "$domain" -g "$resourceGroup" -n frontend-"${shortEnvironmentName}" --environment "$appEnvironmentName" --validation-method CNAME

@Adamation
Copy link

Thank you Deen, we are already doing something similar in our pipelines. Not ideal.

@vinisoto
Copy link
Collaborator

We have fixed an issue that caused the step of creating the managed certificate to take a long time and then fail (although the certificate was indeed successfully created.

With this fix, the number of deployment steps is reduced from 3 to 2. We are designing ways to be able to accomplish the whole process in a single deployment template (e.g., removing the need of creating a container app before creating a managed certificate for certain type of domain ownership validation). We don't have an ETA to share at this time.

@rhuanbarreto
Copy link

Could this be solved by issuing a managed wildcard certificate in the CAE before deploying the apps?

@loadaverage
Copy link

We have fixed an issue that caused the step of creating the managed certificate to take a long time and then fail (although the certificate was indeed successfully created.

With this fix, the number of deployment steps is reduced from 3 to 2. We are designing ways to be able to accomplish the whole process in a single deployment template (e.g., removing the need of creating a container app before creating a managed certificate for certain type of domain ownership validation). We don't have an ETA to share at this time.

Sounds very promising, thank you for this update.
I have a question regarding domain validation process.

Azure has Static WebApp, which allows a customer to validate domain ownership using CNAME record pointed to Static WebApp default domain.

Why exactly the same logic can't be used with Container Apps?
E.g.:

  • customer has a domain on Azure DNS (which is reasonable if full automation is preferred)
  • CNAME record on custom domain, pointed to Managed Environment's default_domain is created
  • domain validation finished
  • Certificate is issued

With such logic in place, it will be (finally) possible to use normal Terraform deployment without any hacks.

Example of Static WebApps in Terraform which will deploy Static WebApp on custom domain and issue TLS certificate in one-time deployment:

resource "azurerm_resource_group" "myapp" {
  name     = local.name
  location = local.azurerm_resource_group_location
  tags     = local.tags
}

resource "azurerm_static_site" "myapp" {
  name                = local.name
  resource_group_name = azurerm_resource_group.myapp.name
  location            = local.app_location
  sku_tier            = local.app_sku.tier
  sku_size            = local.app_sku.size
  tags                = local.tags
}

resource "azurerm_static_site_custom_domain" "myapp" {
  static_site_id  = azurerm_static_site.myapp.id
  domain_name     = format("%s.%s", azurerm_dns_cname_record.myapp.name, azurerm_dns_zone.name)
  validation_type = "cname-delegation"
}

resource "azurerm_dns_zone" "myapp" {
  name                = local.dns_zone
  resource_group_name = azurerm_resource_group.myapp.name
  tags                = local.tags
}

resource "azurerm_dns_cname_record" "myapp" {
  name                = local.custom_domain
  zone_name           = azurerm_dns_zone.myapp.name
  resource_group_name = azurerm_resource_group.myapp.name
  ttl                 = local.default_ttl
  record              = azurerm_static_site.myapp.default_host_name
}

@moattarwork
Copy link

In the documentation, there is a phrase "Your container app has HTTP ingress enabled and is publicly accessible."

Is this true? And why this is not a requirement for other services in Azure and only for Container apps?

@mcx808
Copy link

mcx808 commented Nov 2, 2023

I was just following the official documentation which seems to specify parameters that aren't available in azure-cli 2.53.1 or in the Azure cloud shell:
MicrosoftDocs/azure-docs#116721

What's the status of this, is it not implemented yet? Seems odd to have it documented already.

@fdelu
Copy link

fdelu commented Nov 13, 2023

In case someone's looking for some terraform code that uses a managed certificate, I managed to make it work using only the azurerm and azapi providers. Here's the code. It creates the resource group, container app environment, container app, the certificate and even the DNS record.

It seems to be working, the only issue I'm having is that the terraform destroy command fails when deleting the managed certificate (CertificateInUse error), because I'm using the azapi_update_resource to bind the certificate to the container app, which isn't undone when running terraform destroy. Unfortunately I found no way to fix this without using a provisioner.

I agree that it'd be best if we had a way to bind certificates in a similar way as App Services and Static Web Apps do it.

@kobeyy
Copy link

kobeyy commented Nov 22, 2023

Since there is no way to easily setup an managedCertificate using Bicep.
Has anyone found a way of preventing the deletion of a manually added custom domain when redeploying the container app with a new image version? This would also help when you hosted zone is not managed in azure.

@jurepurgar
Copy link

We have fixed an issue that caused the step of creating the managed certificate to take a long time and then fail (although the certificate was indeed successfully created.

With this fix, the number of deployment steps is reduced from 3 to 2. We are designing ways to be able to accomplish the whole process in a single deployment template (e.g., removing the need of creating a container app before creating a managed certificate for certain type of domain ownership validation). We don't have an ETA to share at this time.

@vinisoto is there any news regarding ETA on this?

@linxcat
Copy link

linxcat commented Nov 30, 2023

The --validation-method flag mentioned in the still-current documentation here doesn't even exist in the az containerapp hostname bind subcommand at this point. I don't know how this is meant to be possible, but it sure doesn't look like it is.

@13excite
Copy link

Certificate name of custom domains is generated automatically in the format $domain_name-$resource_group_name[:8]-$datetime . And it's with resource_group_name[:8] that problems arise.
If the slice of 8 characters from the resource_group ends in -, then the code doesn't check this and as a result it generates a double dash in the name, and the resource name looks like this test.example-prod-ds--231211091022

Even according to the Microsoft guidelines, the resource name should look in the format my-super-resource-name
https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/resource-naming

Also, many TF providers have checks for the correctness of the name, for example azurerm
https://github.com/hashicorp/terraform-provider-azurerm/blob/main/internal/services/batch/validate/certificate_name.go#L13

@dbeattie71
Copy link

With Pulumi, I've provisioned afd domains and used a managed certificate, it works great. ACA needs that :)

@mcx808
Copy link

mcx808 commented Apr 2, 2024

@linxcat - there is a resolution here which is that the functionality is provided by an extension:

#MicrosoftDocs/azure-docs#116721

I think the baffling documentation is still an issue though.

@hoxton-webmaster
Copy link

Since there is no way to easily setup an managedCertificate using Bicep. Has anyone found a way of preventing the deletion of a manually added custom domain when redeploying the container app with a new image version? This would also help when you hosted zone is not managed in azure.

@kobeyy did you find the solution for this ?

@kobeyy
Copy link

kobeyy commented May 7, 2024

@hoxton-webmaster I managed to find a work around by using the azure cli to first lookup the id of the managed certificate and fill it in using a shell script.
Since it was this difficult and hacky to get it working I decided to not use this Azure service as it is premature in my opinion. We halted our roll out to production because of this.

I've added the complete script below as text file.

...

# Workaround to prevent deletion of the custom domain binding by Bicep on a redeploy
get_custom_domain_id() {
    local subscription_id=$(az account show --query 'id' -o tsv)
    local managedEnvironments="/subscriptions/$subscription_id/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.App/managedEnvironments/"
    local firstContainerAppEnv=$(az resource show --ids "$managedEnvironments" --query 'value[0].id' -o tsv)
    local customDomainCertificateId=$(az resource show --ids "$firstContainerAppEnv/managedCertificates/" --query 'value[0].id' -o tsv)
    echo $customDomainCertificateId
}
get_custom_domain_name() {
    local subscription_id=$(az account show --query 'id' -o tsv)
    local managedEnvironments="/subscriptions/$subscription_id/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.App/managedEnvironments/"
    local firstContainerAppEnv=$(az resource show --ids "$managedEnvironments" --query 'value[0].id' -o tsv)
    local customDomainCertificateName=$(az resource show --ids "$firstContainerAppEnv/managedCertificates/" --query 'value[0].properties.subjectName' -o tsv)
    echo $customDomainCertificateName
}


# Function to deploy template
deploy_template() {
    TEMPLATE_FILE="$SCRIPT_DIR/$1"
    echo "Deploying template: $TEMPLATE_FILE"

    AZURE_COMMAND="az deployment group create --template-file \"$TEMPLATE_FILE\" \
                               --resource-group \"$RESOURCE_GROUP\" \
                               --parameters \"@$PARAMETER_FILE\""


    # Workaround to prevent deletion of the custom domain binding by Bicep on a redeploy
    # Inject the manually created custom domain binding into the deployment
    local customDomainCertId=$(get_custom_domain_id)
    if [ -n "$customDomainCertId" ]; then
        AZURE_COMMAND+=" customDomainCertificateId='$customDomainCertId'"
    fi

    local customDomainCertName=$(get_custom_domain_name)
    if [ -n "$customDomainCertName" ]; then
        AZURE_COMMAND+=" customDomainName='$customDomainCertName'"
    fi


    # Conditionally add parameters
    if [ -n "$CONTAINER_REGISTRY_PASSWORD" ]; then
        AZURE_COMMAND+=" containerRegistryPassword='$CONTAINER_REGISTRY_PASSWORD'"
    fi
    if [ -n "$CONTAINER_REGISTRY_USER" ]; then
        AZURE_COMMAND+=" containerRegistryUsername='$CONTAINER_REGISTRY_USER'"
    fi
    if [ -n "$PHP_CONTAINER_IMAGE" ]; then
        AZURE_COMMAND+=" phpContainerImage='$PHP_CONTAINER_IMAGE'"
    fi
    if [ -n "$NGINX_CONTAINER_IMAGE" ]; then
        AZURE_COMMAND+=" nginxContainerImage='$NGINX_CONTAINER_IMAGE'"
    fi
    if [ -n "$IMAGE_TAG" ]; then
        AZURE_COMMAND+=" imageTag='$IMAGE_TAG'"
    fi

    echo "Executing command: $AZURE_COMMAND"
    eval $AZURE_COMMAND
}

deploy_template "deploy.bicep"

full script

@ExplorerSunil
Copy link

Is it possible to create wild card subdomain certificate using container app? I couldn't get it to be working. I'm using azure portal to create certificates.

@FileJunkie
Copy link

Pinging on this issue because of a related problem: as far as I see, while currently in 08-02 and 10-02 versions of the API the CustomDomainConfiguration object of Managed Environment is defined, it appears non-functional: it's ignored by the server, and it also assumes that a Managed Environment can only have 1 certificate defined, which is untrue.

Should I report it in another ticket or is it on-topic here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request roadmap This feature is on the roadmap
Projects
Status: Archive (GA older than 6 months)
Development

No branches or pull requests