[Filebeat][httpjson] Add support for accessing response variable of previous request in chain call

[vinit-chauhan]

Describe the enhancement:

Request 1: To get uuid
	Url :  /get/uuid
Response:
 {
“user_uuid” : “123-abc”
}

Request 2: To get asset list 
	Url :  /get/assets/{user_uuid}/list
Response:
 {
  “asset_list” : [
       asset_1, 
       asset_2,
       asset_3,
       asset_4
       ]
 }
Request 3: To asset detail
	Url :  /get/assets/{user_uuid}/detail/{asset_id}
Response:
 {
    "id": "95c2725c-72",
    "name": “user-windows”,
}

Here from request 1 we can get the value of user_uuid which will be used for making a second call to find the list of assets. So here we can make the second call with the use of replace in chain call.
For request 3, we need to give user_uuid and asset_id in url params. We can get the asset_id from the response of request 2 using replace. But we are not able to use user_uuid in the request 3.

For fetching the value of user_uuid from the url params of the second request we have tried to use .last_response in both the request i.e request 2 and request 3 but we are getting .last_response as
{"body":null,"header":null,"page":0,"url":"xxxxx"}

So we are not able to make the request 3 because we don’t have user_uuid.

Describe a specific use case for the enhancement or feature:
https://developer.tenable.com/reference/exports-assets-download-chunk

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[ShourieG]

@vinit-elastic This is definitely possible currently , using a similar config as follows :

• type: httpjson
  enabled: true
  id: sample-httpjson-id
  request.url: /get/uuid
  chain:
  • step:
    request.url: /get/assets/$.exportId/list
    request.method: GET
    replace: $.exportId
  • step:
    request.url: /get/assets/user_uuid/detail/$.asset_list[:].id
    request.method: GET
    replace: $.asset_list[:].id

We do not require the last response do achieve this

[vinit-chauhan]

@ShourieG As you have mentioned is working perfectly.
However, It does not seem to match our use case. We need the value of user_uuid in URL params in Request 3 which we fetch from the response of Request 1. Basically, we would want to replace {user_uuid} with 123-abc, which we get from the initial request, in both steps, and {asset_id} with all the asset IDs in their respective chained call.

[ShourieG]

@vinit-elastic ok got it, but here the response that you want would want is the 1st response and not the last_response , since in step 3 last_response would mean the last_response of the chain. So maybe we could introduce something like first_response , that holds the response from the root request for access ?

[vinit-chauhan]

@ShourieG - for first_response if you are suggesting implementing something like last_response, I'm not sure how we would use it to add previous values in the new request URLs. We could have used the value templates sprintf but is it possible to manipulate the string using it and then add the asset_id again?

[andrewkroh]

Relates / Blocks: elastic/integrations#2338