Add basic support for robots.txt

[nathanielks]

This is to prevent issues where services, such as Twitter, aren't able to generate a card because /robots.txt is missing from Tachyon's domain.

[rmccue]

We might be better off handling this in API Gateway?

[nathanielks]

@rmccue I had thought about that, but I assumed we'd want support for users who don't use API gateway so I included it in the function itself. What's your preferred method?

[rmccue]

I assumed we'd want support for users who don't use API gateway so I included it in the function itself. What's your preferred method?

IMO, the "function" of Tachyon is to serve images, not to be a full server. The robots.txt stuff I think makes more sense in API Gateway, since that's the routing layer; we were just lazy and said "route everything to Tachyon".

We provide the API Gateway configuration, so we can PR it into there.

Maybe @joehoyle has other thoughts on this though?

[joehoyle]

I think it's fine being here for convenience, there's a fair bit more work when trying to use APIGateway, especially as that also will need to go through cloudformation.

As Japh mentioned, the \n looks wrong. Also, we don't need to include a robots.txt, the problem is it's a 500 now, a 404 would also be fine.

[nathanielks]

Looks wrong, but it's not 😄

$ curl http://localhost:8080/robots.txt
User-agent: *
Allow: /%

(% in my terminal signifies end of the response)

[nathanielks]

@Japh pingy ping

[nathanielks]

@Japh @rmccue pingQ

[nathanielks]

Hey folks! This is getting stale! @Japh @rmccue

[nathanielks]

ᕕ( ᐛ )ᕗ Ping! @Japh

[Japh]

Naise!