azurerm_data_factory_pipeline - json: cannot unmarshal object into Go struct field WebActivityTypeProperties.headers of type string

[l33tCod-er]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.7.2

AzureRM Provider Version

3.93.0

Affected Resource(s)/Data Source(s)

azurerm_data_factory_pipeline

Terraform Configuration Files

resource "azurerm_data_factory_pipeline" "oeeana_refresh_powerbi" {
  name            = "<some_name>"
  data_factory_id = var.data_factory_id
  folder          = "<some_folder>"
  parameters = {
    TenantId         = var.aad_tenant_id
    PBIWorkspaceName = "<some_workspace>"
    PBIDatasetName   = "<some_dataset>"

  }
  variables = {
    PBIWorkspaceID = ""
    PBIDatasetID   = ""
    RefreshStatus  = ""
    Keyvault_Env   = "some_key_vault"
  }
  activities_json = templatefile("${path.module}/templates/oeeana-refresh-powerbi.json.tmpl",
    {
      tenant_id          = var.aad_tenant_id
      pbi_workspace_name = "<some_workspace>"
      pbi_dataset_name   = "<some_dataset>"
  })
}

From init:
- Installing hashicorp/azurerm v3.93.0...
- Installed hashicorp/azurerm v3.93.0 (signed by HashiCorp)

Debug Output/Panic Output

reading the state of Data Factory Pipeline "<name>": datafactory.PipelinesClient#Get: Failure responding to request: StatusCode=200 -- Original Error: Error occurred unmarshalling JSON - Error = 'json: cannot unmarshal object into Go struct field WebActivityTypeProperties.headers of type string' JSON = '{

Expected Behaviour

No response

Actual Behaviour

No response

Steps to Reproduce

terraform plan

Important Factoids

No response

References

Marked as fixed, but happens to us even with 3.93.0
#24758

We downgraded to 3.89.0 now to temporarily overcome the issue

[stumueller]

That's strange - I can confirm that the fix in #24758 worked for me. I've tested it three times now :-)

[l33tCod-er]

I double checked as well after your comment: Got the same error every time...?
I also validated it's really 3.93.0 we're using.

[ziyeqf]

Hi @l33tCod-er, thanks for reporting.

Could you please try again with environment variable TF_LOG=debug provide your log here? It will help us diagnositc.
To save the log to file you can set TF_LOG_PATH variable, details of these variables could be found here

Kindly remind the log file may contain sensitive information, you can replace them before upload it.

Thanks.

[l33tCod-er]

debugExtract.txt
Sorry it took a while to prepare the requested. I tested with 3.94.0 and still get the same error.
If there is more you need, let me know: I could only provide you with a small extract of the logs.

[ziyeqf]

Hi @l33tCod-er , thanks for your log. With your log we found the reason. Just submitted a PR(#25178) for this, once it merged & released the issue should be fixed.
Thanks!

[l33tCod-er]

Wow, that was lighning fast - thanks a lot for the fix :)!

[l33tCod-er]

@ziyeqf: We tried again now on our end - and the issue is still there.
Tried with v3.97.0:

Still get:

Error: reading the state of Data Factory Pipeline "<name>": datafactory.PipelinesClient#Get: Failure responding to request: StatusCode=200 -- Original Error: Error occurred unmarshalling JSON - Error = 'json: cannot unmarshal object into Go struct field WebActivityTypeProperties.headers of type string' JSON = '{

Let me know if you need more information.

[ziyeqf]

Hi @l33tCod-er ,thanks for you patience.

Have found the cause is because there is webactivity nested in untill activity, which the previous workaround did not cover.
There is an issue related to this, it decides the way how we fix this. You can subscribe that issue to keep a track. I will also leave a comment here if this issue get fixed.

Thanks!

[tombuildsstuff]

As mentioned on the issue on tombuildstuff/kermit, I've opened this issue on hashicorp/pandora to update the parsing of the API Definitions so that hashicorp/go-azure-sdk can output the correct types.

It's worth noting that tombuildsstuff/kermit is a temporary/legacy SDK that's generated directly from the Microsoft AutoRest tooling/API Definitions - as such any fix for this would need to be made upstream (there's a related meta issue about the use of these custom format values here since this differs from every other Service within Azure) - but as such we're not planning to add a workaround for this within tombuildstuff/kermit and so the fix is going to be migrating this to hashicorp/go-azure-sdk.

As such I'm going to mark this one as upstream/pandora for now, but once the upstream issue is resolved (so that we're parsing these undocumented Data Factory specific types) we can look at switching this over to hashicorp/go-azure-sdk, which should solve this.

[l33tCod-er]

The issue on hashicorp/pandora has been resolved meanwhile: Any schedule this bug here will be fixed? Thanks in advance!

/cc @ziyeqf

[ziyeqf]

Hi @l33tCod-er , thanks for head up.

I am working on this but sorry I can not provide an ETA. To fix this we need to migrate sdk to hashicorp/go-azure-sdk, it will be a large PR. Once the job done I will comment here.

Thanks

[abhilash-gsynergy]

Hi everyone , thanks in advance.

I am also facing same issue, I am trying to create an until activity, within this activity i am using a web activity which requires headers. I am running into the same issue. Is there a fix or work around for this issue?

[prise6]

Hi everyone,
Like @abhilash-gsynergy, my team and i are facing the same issue : your fix is working for "web activity" but when it's nested, like "Until Activity", it doesn't work anymore.

I'm sorry to ask : is there a workaround until you migrate to another sdk ?

thanks :)

[ziyeqf]

Hi everyone, Like @abhilash-gsynergy, my team and i are facing the same issue : your fix is working for "web activity" but when it's nested, like "Until Activity", it doesn't work anymore.

I'm sorry to ask : is there a workaround until you migrate to another sdk ?

thanks :)

Hi @prise6, a temporary workaround may be using the azapi provider for now.

[l33tCod-er]

@ziyeqf: is there any update? The bug has been preventing us from updating the provider version for almost a year now. Any timeline given?

[ziyeqf]

Hi @l33tCod-er , thanks for remind.

It needs a sdk swap, which will be kind of big project. Please keep this issue and will update once PR submitted

[weineran]

I tried upgrading to azurerm v4.20.0 but I'm still getting these errors:

unmarshaling response body: unmarshaling index 0 field 'Activities' for 'Pipeline': unmarshaling into AzureFunctionActivity: json: cannot unmarshal object into Go struct field AzureFunctionActivityTypeProperties.typeProperties.body of type string

unmarshaling response body: unmarshaling index 0 field 'Activities' for 'Pipeline': unmarshaling into WebActivity: json: cannot unmarshal object into Go struct field WebActivityTypeProperties.typeProperties.url of type string

Is anyone else still getting these errors?

For now, it seems I need to stick with azurerm v3.89.0 😦

Here is my version info:
terraform version

Terraform v1.8.2
on windows_amd64
+ provider registry.terraform.io/azure/azapi v1.15.0
+ provider registry.terraform.io/hashicorp/azuread v3.1.0
+ provider registry.terraform.io/hashicorp/azurerm v4.20.0

[stephybun]

@weineran could you provide your config info in #28837 - that'll be the starting point for figuring out if there are further data issues in the swagger definition for activities.