Change HTTP Gateway for ChainAPI test calls

[aTeoke]

We need to do change HTTP Gateway to make it usable for ChainAPI test API call purposes. The user enters values for each endpoint parameters. e.g. a currency conversion endpoint expects: from, to and amount. The enter these, click execute and the Airnode gateway gets called with these request parameters
The response gets returned to the end user. They can then play with the reserved parameters (if they are user defined) and see the encoded value change in real time. No Airnode/backend calls involved at this point.

The use case:

• Let the user provide the values for the user-defined endpoint parameters
• Call the gateway with these to get the JSON API response
• Let the user provide the values for the reserved parameters and apply them to the JSON at the ChainAPI side to let the user figure things out by trial and error, without requiring additional API calls

Changes to be made:

• Currently the gateway errors if it is called with parameters that would have made a successful API call, but not process it successfully (for example it's missing _type, which is absolutely required to be defined in all requests). For the above to be possible, the gateway needs to respond with the API response JSON and error partially while trying to encode the response.
• The second change is making the reserved parameters inaccessible in pre/post processing to make sure that we can simulate how the reserved parameters will be applied consistently.

Change of the gateway discussion here
Designs on ChainAPI side here

[bbenligiray]

I have an endpoint with the following reserved parameters

"reservedParameters": [
            {
              "name": "_type",
              "fixed": "int256"
            },
            {
              "name": "_path",
              "fixed": "market_data.current_price.usd"
            },
            {
              "name": "_times",
              "fixed": "1000000"
            }
          ]

When I call http-data/ with {"parameters": {"coinId": "bitcoin"}}, I get

{
    "rawValue": {
        "id": "bitcoin",
        "symbol": "btc",
        "name": "Bitcoin,
        ...
    },
    "encodedValue": "0x00000000000000000000000000000000000000000000000000000006d1c2ac40",
    "values": [
        "29289000000"
    ]
}

I update the reserved parameters as below

"reservedParameters": [
            {
              "name": "_type"
            }
          ]

When I call http-data/ with {"parameters": {"coinId": "bitcoin"}}, I get

{
    "id": "bitcoin",
    "symbol": "btc",
    "name": "Bitcoin",
    ...
}

I would have expected

{
    "rawValue": {
        "id": "bitcoin",
        "symbol": "btc",
        "name": "Bitcoin,
        ...
    }
}

[bbenligiray]

I created an endpoint with a user-defined _type. I added console.log(input, endpointParameters) to its pre and post processing. Made http-data, http-signed-data, rrp requests to this endpoint where I specify the _type value in parameters. The node didn't print _type in any of these. I retried the same with Airnode v0.11 and confirmed that the older printed _type.

I reopened the issue to discuss the above

[bbenligiray]

@andreogle for visibility

[dcroote]

Well yes this is expected, _type is a reserved parameter and:

We want both pre-processing and post-processing to not have access to the reserved parameters (this seems to only do it for post-processing)

from #1831 (comment)

[bbenligiray]

No problems with that, I'm referring to this

The JSON API response is at the root of the response body or in rawValue, depending on if encoding fails. The problem is the user can't tell if the encoding will fail at runtime. This means that while ChainAPI is using http-data/ to get the raw API response, it will first need to look for a rawValue in the response body, and if it doesn't exist, fall back to using the entire body. This is ugly and becomes a more significant issue when the API response has a rawValue field at the root level.

[andreogle]

I'm a bit confused by this. My understanding was that this change was essentially to make expose this http-data endpoint as a proxy between ChainAPI and the end API. I would expect this to respond exactly as the end API would respond - after any kind of filtering/transformations are applied. Wrapping everything in a single top level rawValue key doesn't make much sense to me.

i.e. I would expect to see this from your example above.

{
    "id": "bitcoin",
    "symbol": "btc",
    "name": "Bitcoin"
}

I'm not following why encodedValue and values are being returned. They are irrelevant for endpoint testing in ChainAPI

[dcroote]

What you're describing is pre-existing behavior. I only made this change in #1794:

airnode/packages/airnode-node/src/api/index.ts

Lines 237 to 244 in 7f3c492

	if (!goExtractAndEncodeResponse.success) {
	const log = logger.pend('ERROR', goExtractAndEncodeResponse.error.message);
	// The HTTP gateway is a special case for ChainAPI where we return data from a successful API call that failed processing
	if (payload.type === 'http-gateway') {
	return [[log], { success: true, errorMessage: goExtractAndEncodeResponse.error.message, data: rawResponse.data }];
	}
	return [[log], { success: false, errorMessage: goExtractAndEncodeResponse.error.message }];
	}

Here is the corresponding E2E test:

airnode/packages/airnode-node/test/e2e/http.feature.ts

Lines 38 to 55 in 7f3c492

	it('returns data from a successful API call that failed processing', async () => {
	const invalidType = 'invalidType';
	// Use a minimal reserved parameters array with only _type (which is required by OIS) and an invalid value
	const modifiedConfig = { ...config };
	modifiedConfig.ois[0].endpoints[0].reservedParameters = [{ name: '_type', fixed: invalidType }];
	const minimalParameters = { from: 'ETH' };
	const [_err, result] = await processHttpRequest(modifiedConfig, endpointId, minimalParameters);
	const expected: HttpGatewayApiCallResponse = {
	data: { result: '723.39202' },
	success: true,
	errorMessage: `Invalid type: ${invalidType}`,
	};
	expect(result).toEqual(expected);
	});

[bbenligiray]

The only use case for encodedValue and values for the non-signed gateway I can think of is allowing someone like @metobom or a third-party service to test integrations (for example data feed templates wouldn't have any missing reserved parameters) without giving access to any signatures.

[bbenligiray]

Closing this, the remainder will be tracked by #1845