HTTP Headers Cleanup: API and Gateway

[lidel]

Summary

We should have:

• Interop tests that ensure HTTP responses have the same headers and values, no matter which implementation is the backend

• Diagnostic tool/script that can be run against any HTTP API or Gateway port and provide quick health status

Status

• 2018-Q4: Interop is a mixed bag. Some headers are tested, some are not or different values across implementations. A simple test script exists but is incomplete.

TODO

Go over below headers of interest and ensure proper safeguards are in place.

Gateway

• X-Ipfs-Path: IPFS Path of returned resource
  • Unicode will probably break some header parsers, user-provided example: ʇɐɔ‾ƃuᴉʞnd.jpg
• Etag: resolved CID/multihash of returned payload
  • If-None-Match header
• Cache-Control:
  • disable cache for directory listings and errors,
  • enable heavy caching for immutable assets from /ipfs/ namespace
  • Cache-Control for /ipns/ – https://github.com/ipfs/go-ipfs/issues/1818 / No cache-control on Host: based requests kubo#5968 / disable-last-modified-behaviour-ipns-routes kubo#8074
  • other cache control fixes Meta: HTTP Gateway cache control improvements kubo#8717
  • Stale-While-Revalidate in browsers https://www.mnot.net/blog/2014/06/01/chrome_and_stale-while-revalidate
• Suborigin: use root CID in base32 and literal prefix to conform
  to the current suborigin spec (Suborigins  #66)
• Last-Modified
  • discuss.ipfs.io: user experiencing last-modified from 1970
  • Webgateway: sets 'last-modified' header to now()  kubo#7968
• CORS
  • CORS returning all, instead of just one https://github.com/ipfs/go-ipfs/issues/1659#issuecomment-512241089\
  • CORS headers missing on shash normalization redirect CORS headers not sent for some redirects kubo#8501
  • • all issues labeled with topic/CORS in general: https://github.com/ipfs/go-ipfs/labels/topic%2FCORS
• some gateways (eg. Infura) use custom content sniffing and disable it on the client via X-Content-Type-Options: nosniff which causes CSS files to be returned as text/plain (relative links in files not added ipfs-shipyard/ipfs-deploy#86 (comment))
  • this is a strong signal we need to provide better story around content-type, at least for files used on websites (Storing Explicit Content Type ipld/legacy-unixfs-v2#11)
  • tracking content-type ideas in Content Type set by HTTP Gateway #152

API

• Etag & Cache-Control: without this, all content addressed "gets" skip browser cache which results in degraded performance and wasted bandwidth (details: Add etag and other caching headers to all content addressed gets in api kubo#3543)
• X-Chunked-Output: various API endpoints break without it (eg. Unexpected X-Chunked-Output in /api/v0/version response kubo#5711)
• CORS
  • Access-Control-Expose-Header: without this, JS is unable to see X-Chunked-Output in Chrome ([CORS] Old library makes exploitable CORS configuration (need to update dependency) kubo#5745)
  • User-Agent in default list of Access-Control-Allow-Headers to follow whatwg/fetch spec (Add user-agent to default list of Access-Control-Allow-Headers kubo#5138)
• Suggestion to set Access-Control-Allow-Credentials should be removed from docs as it is not used and may cause security issues in some setups.
• CORS preflight broken with custom header CORS preflight requests to RPC API with custom headers kubo#7667

Related

• Examples of issues that occur if we don't have regression tests for gateway
  • Gateway requests responds with 403 - Forbidden ipfs-shipyard/ipfs-share-files#17 (comment): Gateway requests responds with 403 - Forbidden
  • Unexpected X-Chunked-Output in /api/v0/version response kubo#5711: Unexpected X-Chunked-Output in /api/v0/version response
• feat(gateway): X-Ipfs-Path, Etag, Cache-Control, Suborigin js-ipfs#1537: adding X-Ipfs-Path, Etag, Cache-Control, Suborigin to js-ipfs
• Add etag and other caching headers to all content addressed gets in api kubo#3543: Add etag and other caching headers to all content addressed gets in api
• Gateway has unkillable CORS headers kubo#6213: Gateway has unkillable CORS headers
• public-gateway-smoke-test.sh – Small test suite ensuring Public HTTP Gateway behaves correctly, all CORS headers and permissions are in place etc.
• Snapshot of headers in Q4-2018
• -Allow-Headers vs -Expose-Headers

[lanzafame]

/cc @hsanjuan

[lidel]

For the record, in Q4 2018 headers look like this 🙃

Public Gateway at ipfs.io

$ curl -s -I -X GET https://ipfs.io/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR | grep -i Access-Control
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: X-Requested-With, Range, Content-Range, X-Chunked-Output, X-Stream-Output
Access-Control-Expose-Headers: Content-Range, X-Chunked-Output, X-Stream-Output

go-ipfs v0.4.18 defaults

(equal to ipfs config --json API.HTTPHeaders '{}')

$ curl -s -I -X GET http://127.0.0.1:8080/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR | grep -i Access-Control
Access-Control-Allow-Headers: Content-Range, X-Chunked-Output, X-Stream-Output
Access-Control-Allow-Methods: PUT
Access-Control-Allow-Methods: GET
Access-Control-Allow-Methods: POST
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Range, X-Chunked-Output, X-Stream-Output

js-ipfs v0.33.1 defaults

(no API.HTTPHeaders in config)

$ curl -s -I -X GET http://127.0.0.1:9090/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR | grep -i Access-Control
access-control-allow-headers: X-Stream-Output, X-Chunked-Output, X-Content-Length
access-control-expose-headers: X-Stream-Output, X-Chunked-Output, X-Content-Length

[lidel]

-Allow-Headers vs -Expose-Headers

There is an important nuance: Access-Control-Allow-Headers tells browsers which headers can be used in XHR CORS request and Access-Control-Expose-Headers whitelists which response headers will be available to be read by JS:

The Access-Control-Allow-Headers response header is used in response to a preflight [HTTP OPTIONS] request which includes the Access-Control-Request-Headers to indicate which HTTP headers can be used during the actual request.

The Access-Control-Expose-Headers response header indicates which headers can be exposed as part of the response by listing their names.