Get Watch API returns 400 error for watch having password in chained http input

[inqueue]

Elasticsearch version: 6.3.1

Watches containing one or more chained http inputs with redacted passwords are not returned by the Get Watch API. The following example uses a single http input. The same occurs with multiple http inputs in the chain.

POST _xpack/watcher/watch/502_errors
{
  "trigger": {
    "schedule": {
      "interval": "15m"
    }
  },
  "input": {
    "chain": {
      "inputs": [
        {
          "bad_gateway": {
            "http": {
              "request": {
                "scheme": "https",
                "host": "host.tld",
                "port": 9200,
                "method": "get",
                "path": "log-*/_search",
                "params": {},
                "headers": {},
                "auth": {
                  "basic": {
                    "username": "username",
                    "password": "(┛ಠДಠ)┛彡┻━┻"
                  }
                },
                "body": """{ "size": 0, "query": { "bool": { "must": [ { "match_phrase": { "log" : "*502 bad gateway" } }, { "match_phrase": { "type": "err*" } }, { "range": { "@timestamp": { "gte": "now-60m" } } } ] } }}"""
              }
            }
          }
        }
      ]
    }
  },
  "condition": {
    "script": {
      "source": "return ctx.payload.bad_gateway.hits.total > 0 || ctx.payload.gateway_timeout.hits.total > 0;",
      "lang": "painless"
    }
  },
  "actions": {
    "email_alert_either": {
      "throttle_period_in_millis": 3600000,
      "condition": {
        "script": {
          "source": "ctx.payload._value.3 == 0 || ctx.payload._value.4 == 0",
          "lang": "painless"
        }
      },
      "email": {
        "profile": "standard",
        "from": "[email protected]",
        "to": [
          "[email protected]"
        ],
        "subject": "{{ctx.payload._value.0}}",
        "body": {
          "html": "watch_body"
        }
      }
    }
  }
}

{
  "_id": "502_errors",
  "_version": 1,
  "created": true
}

Fetch Watch

GET _xpack/watcher/watch/502_errors

Expect
Watch is returned with password field value ::es_redacted::.

Actual

{
  "error": {
    "root_cause": [
      {
        "type": "parse_exception",
        "reason": "found redacted password in field [password]"
      }
    ],
    "type": "parse_exception",
    "reason": "could not parse [http] input for watch [502_errors]. failed to parse http request template",
    "caused_by": {
      "type": "parse_exception",
      "reason": "found redacted password in field [password]"
    }
  },
  "status": 400
}

Workaround
Retrieve the watch by ID directly from the .watches system index.

GET .watches/doc/502_errors

Discovery
This issue surfaced while attempting to edit the watch with the Kibana Watcher UI. Clicking the watch in the UI results in the Watcher: Error 400 Bad Request: [parse_exception] found redacted password in field [password] red banner error at the top of the browser window. The 400 response returned by the Get Watch API prevents Kibana from loading the watch for UI editing.

[elasticmachine]

Pinging @elastic/es-core-infra

[spinscale]

This is fixed in #31721, to be released with 6.3.2.

[inqueue]

Thanks @spinscale!

I can confirm the issue is resolved for watches created while running 6.3.2. The issue will still surface for watches registered in versions prior to 6.3.2. To fix existing watches after upgrading to 6.3.2 or later:

1. Fetch the watch by id from the .watches index; e.g., GET .watches/doc/502_errors/_source.
2. Re-register the watch with the Put Watch API.

[remd]

@inqueue What are the steps to "re-register" the watch with the Put Watch API? Do I need to delete the watch before registering it again?

Just grabbing the source from GET .watches/doc/my-watch/_source and passing the returned JSON to PUT _xpack/watcher/watch/my-watch does not fix the issue. I still get the [parse_exception] found redacted password in field [password] error when making changes through the Edit Watcher UI in Kibana.

I am running ELK 6.8.2.