Add config.json based pre and post-processing

[aquarat]

This PR adds relatively simple pre and post-processing to Airnode.
This was originally in support of a specific provider, but I think it's usable in Airnode master and saves us from having more custom Airnodes.

Processing code is supplied per endpoint in the OIS in config.json, in a schema based on this documentation

This is discount airnode-plug:

• Processing code is supplied as an array of chunks
• The array is run through reduce, which executes each chunk in sequence.
• A chunk can expect to have access to input (which is the result of the previous chunk's execution or the initial value)
• Each chunk is expected to produce an output variable. We append an ending statement which returns output the calling code.
• The reduce is called inside go and limited to 500ms gross timeout (perhaps this is too small or maybe this should be in constants.ts

Processing code is not supplied on-chain.
The idea is to support other environments (like WASM) at a later stage.

The idea is that this enables a bunch of high value use cases for limited development time. Security mitigations for processing code are not required because we can reasonably expect the API provider to not hack themselves.

Post processing processes the API response prior to extraction and encoding.
Pre processing processes the parameters used for an API call.

An example from a testing config.json endpoint:

           [
            {
              "operationParameter": {
                "in": "query",
                "name": "sparkline"
              },
              "value": "false"
            }
          ],
          "preProcessingSpecifications": [
            {
              "environment": "Node 14",
              "value": "const output = {...input, coinId: 'nope-bitcoin'};"
            },
            {
              "environment": "Node 14",
              "value": "const output = {...input, coinId: input.coinId.substring(5)};"
            }
          ],
          "postProcessingSpecifications": [
            {
              "environment": "Node 14",
              "value": "const output = parseInt(input.market_data.current_price.usd)*10000;"
            },
            {
              "environment": "Node 14",
              "value": "const output = input * require('crypto').randomInt(100);"
            }
          ],
          "reservedParameters": [
            {}]

Further context, these are discussions between Burak and I on this topic:

• No parameters, we can add them later.
• The array implies that these can be chained.
• We specify the environment, meaning that this can do WASM, different runtime versions of Node, something entirely different, etc. We can require environment to be Node 14 (or whatever we use with Airnode) for now
• There is an argument to be made for inputs where inputs[0] is the output of the previous step and inputs[1] is the initial API request parameters for preprocessing. However, I don't really see why we would need inputs[1] (aquarat says: stack)

[Siegrift]

Overall looks good to me. But I have some questions.

Do we want to land this to master now? Shall it be released as part of 0.6 or we should first release 0.6 and then add this as part of 0.7?

[amarthadan]

Overall looks good to me. But I have some questions.

Do we want to land this to master now? Shall it be released as part of 0.6 or we should first release 0.6 and then add this as part of 0.7?

If possible we should wait till 0.6 is released (shouldn't be long now, everything needed is merged it just needs to be tested). Worst case we should merge it to 0.7 branch and wait for the 0.6 release before merging is to master. But if think keeping it open is fine in this situation.

[Siegrift]

@aquarat I wanted to do sooner, but yesteday got unexpectedly busy for me so I found some more time today. Please take a look.

I will be heading off to vacation shortly, but will be able to take a look on Sunday :D

[Ashar2shahid]

Looks good, just a note, when looking up eval I came across Function which seems to be the preferred over eval due to the scope eval is invoked in . Here is the same code using Function someone else can have a look

function execute(_input: any, code: string) {
  // this is the recommended way of executing code with Function
  return Function(`
 "use strict";
  const ethers = require('ethers');
  const {BigNumber} = ethers;
  const input = _input;

  ${code};

  return output;
  `)(_input);
}

[aquarat]

Function

Interesting, I thought that Function and eval were equivalent, I didn't realise they had scope differences.
We're trying to run trusted pre and post-processing code and I think there are scenarios where the provider would want the code to be able to access parent-scoped objects. The plan is to eventually support multiple execution environments (with different security privileges), but this is a long term low priority goal. We want basic processing soon so we can reduce the usage of custom Airnodes.

I think for now this is fine, but thanks for pointing out Function's difference 😄 💪

[aquarat]

@Siegrift Thanks for introducing these changes. It's perfect for now IMO.

[Siegrift]

I agree with @Ashar2shahid suggestion. I do not see any reason why the postprocessing should have access to Airnode internals that we can are to change freely. That code should only care about input and ethers which we are supplying.

One thing, shall we expose BigNumber from ethers directly?

[Siegrift]

@aquarat shall we have this line here? I am not sure why only expose BigNumber drectly and not other stuff.

[aquarat]

We can remove it, no problem :)
(To avoid ambiguity, the developer can be expected to import everything they need)

[aquarat]

...that was fast @ commit :P

[Ashar2shahid]

Yes the more I think about it , we should definitely use Function instead, say the provider sets an err variable in their code, this would now either throw or give an exception at const[err,processedParameters] whereas it shouldn't have or maybe it should......... 🤔. Well either way I think Function is the safer alternative.

[aquarat]

Yes the more I think about it , we should definitely use Function instead, say the provider sets an err variable in their code, this would now either throw or give an exception at const[err,processedParameters] whereas it shouldn't have or maybe it should......... thinking. Well either way I think Function is the safer alternative.

All good, Function can be used instead. I agree that it is a cleaner alternative and it shouldn't impact the immediate use case for this functionality. This functionality is a stop-gap to enable use cases that will take some time for us to support. The whole thing is expected to be advanced in nature and "here be dragons".

[Siegrift]

I think this is good to go, but since I was the co-author I think someone else should give an approve as well.

Maybe @Ashar2shahid or @amarthadan

[Ashar2shahid]

Yeah looks good to me

[aquarat]

I just tried using this for a real task and I needed to import various libraries. Unfortunately neither import nor require work and this meant I had to customise processing to switch it back to eval.

I could have gotten around the limitation by inserting a chunk of the of the libraries I wanted (lodash), but that just seems unfortunate and unnecessarily inefficient. I think unsafeEvaluate should be exactly that, unsafe. The only variables eval could accidentally clobber in this implementation are input and code. The code is sourced from the trusted config.json and one has to write this code with the context of what it is processing; this isn't the realm of the ignorant. This is not your normal night of theatre.

If we really must have Function then it should be a different procession option imo, like Node 14 Function vs Node 14 eval.

[Siegrift]

Ok, I'll revert it back to use eval but you should be careful about importing libraries as they might change or get removed from Airnode. But I understand that you need to do this for your use case.

[aquarat]

Sorry for being difficult and thank you :)

I'll make a few notes for when we document this (initially and for some time to come for our internal use).

[aquarat]

Just an additional note; I'm actively using this code and I've discovered a few interesting quirks along the way that we may want to address prior to merging/don't merge until I've documented these cases.

As a taster, I think we might want this instead of eval: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/AsyncFunction
and https://nodejs.org/api/vm.html#vmruninnewcontextcode-contextobject-options

[aquarat]

It's probably best if I package up my changes and base them on this branch, opening a separate PR.

[aquarat]

Looks like something was introduced that broke one of the tests. I'll check it out tomorrow.

[aquarat]

I am working on this now fyi

[aquarat]

Tests are now passing locally, now just waiting for CI.