Robots.txt generator: disallowing specific pages instead of whole sections #2876
Comments
pahupe
added a commit
to pahupe/acs-aem-commons
that referenced
this issue
Jul 22, 2022
Two new OSGi config properties have been introduced: - allow.page.property.names and - disallow.page.property.names to generate ALLOW and DISALLOW rules for single pages (path/page.html) These can be combined with existing properties - allow.property.names and - disallow.property.names to ALLOW single pages (path/page.html) while DISALLOWing nested paths (path/page/), or vice versa
davidjgonzalez
pushed a commit
that referenced
this issue
Aug 22, 2022
* Improvement for #2876 Two new OSGi config properties have been introduced: - allow.page.property.names and - disallow.page.property.names to generate ALLOW and DISALLOW rules for single pages (path/page.html) These can be combined with existing properties - allow.property.names and - disallow.property.names to ALLOW single pages (path/page.html) while DISALLOWing nested paths (path/page/), or vice versa
|
Fixed in v5.3.4 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'd like to request a new feature in the ACS commons sitemap generator.
Currently, when adding the disallowed property on a page, the line thats beeing added in robots.txt looks like this:
Disallow: https://www.mysite.com/somepage/Basicly, this will disallow the whole /somepage/ section. However somepage.html does not match this path, and will be indexed. See the robots.txt spec for details: https://developers.google.com/search/docs/advanced/robots/robots_txt#disallow
Shouldnt it be possible to also disallow somepage.html, or maybe make it configurable wether the single page or whole section should be disallowed?
Removing the trailing slash is not an option, since it would match any page starting with /somepage*
The text was updated successfully, but these errors were encountered: