gateway: fix CORs headers

[Stebalien]

• fixes Add user-agent to default list of Access-Control-Allow-Headers #5138 -- always add user-agent to access-control-allow-headers.
• fixes content-type to allowedHeadersArr #5888 -- same with content-type.
• fixes Allow/Expose-Headers headers overwrite user-provided ones #5892 -- extend user-provided headers instead of overriding them.

TODO:

• Sharness tests.
• Remove header defaults from the config.

[Stebalien]

cc @lidel, @popmanhe.

[Stebalien]

We already skip these in the commands lib so this probably isn't necessary.

[lidel]

While we are fixing CORS, it may be worth addressing a nuance related to #5138, #5892 and ipfs/in-web-browsers#132 (comment):

• Access-Control-Allow-Headers is relevant only during OPTIONS request, can be skipped in other response types
• Access-Control-Expose-Headers is relevant only during non-OPTIONS request (GET, POST etc), and can be skipped in OPTIONS response

Right now we return them for every request type which is really confusing and suggests brute-force approach to CORS. Would be nice to address this in default config and remove noise of irrelevant headers (test suggestions below)

[lidel]

Access-Control-Allow-Headers is relevant only during OPTIONS request, should not be returned with GET response:

Suggested change

	grep "< Access-Control-Allow-Headers: Range" curl_output &&
	grep -vL "< Access-Control-Allow-Headers" curl_output &&

Rationale: ipfs/in-web-browsers#132 (comment)

[lidel]

Access-Control-Expose-Headers is relevant only during non-OPTIONS request (GET, PUT etc), and should not be returned with OPTIONS response:

Suggested change

	grep "< Access-Control-Expose-Headers: Content-Range" curl_output
	grep -vL "< Access-Control-Expose-Headers" curl_output

Rationale: ipfs/in-web-browsers#132 (comment)

[Stebalien]

While we are fixing CORS, it may be worth addressing a nuance related to #5138, #5892 and ipfs/in-web-browsers#132 (comment):

We should fix this but it may require a bit of refactoring to get it right (beyond the scope of this PR).

[daviddias]

@Stebalien is it the right time to ask that the IPFS config.json gets organized by system rather than a mix of everything. By this I mean, have a

{
  libp2p: {
    // All things libp2p
  },
  Gateway: {
    // All things gateway
  },
  RemoteAPI: {
    All things RemoteAPI
  }
}

This would be a great improvement to create easy to parse tutorials. Right now we see things such as:

  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "PathPrefixes": [],
    "RootRedirect": "",
    "Writable": false
  },

Which raises questions such as "Why does the Gateway need CORS headers, it only receives GET requests?", "What is a Writable Gateway?" and more.

[lidel]

is it the right time to ask that the IPFS config.json gets organized by system rather than a mix of everything.

@daviddias I feel I miss some context here: what exactly feels to you like mixing concerns?

My understanding is that we already have separate namespaces for API and Gateway ports, and we are able to define different HTTPHeaders for each:

$ ipfs config --json Gateway.HTTPHeaders
$ ipfs config --json API.HTTPHeaders 

APICommands is present in Gateway section because it is an optional override of a "safe" API subset exposed on Gateway port (see below).

Which raises questions such as "Why does the Gateway need CORS headers, it only receives GET requests?"

CORS on Gateway is useful for API subset exposed there. Websites are able to read data via cross-origin XHR to https://ipfs.io/api/v0/dns/ipfs.io?r=true and AFAIK that is why our public Gateway at ipfs.io returns Access-Control-Allow-Origin: * currently.

[hsanjuan]

This looks OK for me (knowing it doesn't fix everything, it does improve some things).

[Stebalien]

@daviddias, @lidel is correct but I do agree that the config could use a bit of a reorg with 20-20 hindsight (same as the commands). However, that would be a massively breaking change. Let's reconsider this when we reorganize the API/commands. When we do that, we can insert a config translation layer such that ipfs-old config sees the old config and ipfs-new config sees the new one.