robots.txt

[tobyhunt]

Add one.

[taktran]

What do you want in it?

[taktran]

I can think of a use for the robots.txt - to exclude all files on pebblecode-staging.herokuapp.com and pebblecode-sandbox.herokuapp.com, so search engines don't duplicate their content

Robots.txt is also used to avoid what is known as “canonicalization” problems or having multiple “canonical” URLs. This problem is sometimes referred to incorrectly as a “duplicate content” problem.

http://seoroi.com/seo-faq/robotstxt-what-it-is-why-its-used-and-how-to-write-it/

Although there are passwords on these sites, sometimes we need to take off the password during development, so they might get spidered.

[shapeshed]

We should have the following robots.txt for production if we don't have one.

User-agent: *
Disallow: 

Google Webmasters moans if you don't have one and it is good practice.

[taktran]

I read that you can just leave it out if you want to allow everything, and submitting the sitemap to google webmasters seemed to be ok, but no harm adding it in. Have added it in http://pebblecode.com/robots.txt