/sitemap.xml returns ERR_CONTENT_DECODING_FAILED when trying to load a sitemap

[maxpavlov]

Hi.

I have an ASP.NET MVC4 project with Ninject Ioc and an overriden Controller Factory.

At some point I've added the following NuGetPackages:

In NinjectWebCommons.cs (composition root), I have:

RegisterServices(kernel);
MvcSiteMapProvider.SiteMaps.Loader = kernel.Get<ISiteMapLoader>();
var validator = kernel.Get<ISiteMapXmlValidator>();
validator.ValidateXml(HostingEnvironment.MapPath("~/Mvc.sitemap"));

private static void RegisterServices(IKernel kernel)
        {
            kernel.Load(new MPDataModule(), new MPCoreModule(), new MvcSiteMapProviderModule());
        }

I've added the Routes to MvcSitemapProvider manually to my RouteConfig:

routes.Add(new Route("sitemap.xml", new RouteValueDictionary(new { controller = "XmlSiteMap", action = "Index", page = 0 }),  new MvcRouteHandler()));

routes.Add(new Route("sitemap-{page}.xml", new RouteValueDictionary(new { controller = "XmlSiteMap", action = "Index", page = 0 }), new MvcRouteHandler()));    

The section with MvcSitemapProvider appSettings is like this:

<add key="MvcSiteMapProvider_IncludeAssembliesForScan" value="misechko.com" />
<add key="MvcSiteMapProvider_UseExternalDIContainer" value="true" />
<add key="MvcSiteMapProvider_ScanAssembliesForSiteMapNodes" value="true" />

When I debug the custom ControllerFactory's GetControllerInstance method, after the custom logic, the ControllerInstance of Type XmlSiteMapController is being returned by the ControllerFactory:

Still, when I am trying to load an /sitemap.xml, I get an error:

Please help =)

[NightOwl888]

There was a recent bug in the NuGet package that was fixed (#289 and #124) which caused missing web.config sections not to be added. In particular, you need to ensure this section is there for the /sitemap.xml endpoint to execute in MVC 4 or higher.

  <system.webServer>
    <modules>
      <remove name="UrlRoutingModule-4.0" />
      <add name="UrlRoutingModule-4.0" type="System.Web.Routing.UrlRoutingModule" />
    </modules>
  </system.webServer>

While there is nothing wrong with registering the routes manually, you should make sure the following line is not in your application start code, because it adds the routes (again).

// Register the Sitemaps routes for search engines 
XmlSiteMapController.RegisterRoutes(RouteTable.Routes);

[maxpavlov]

@NightOwl888, Thank you for a prompt response. I think you provider the best support for GitHub Repository ever.

I've researched a lot on the problem prior to asking this question, and I saw a suggestion with remove/add UrlRoutingModule. I've checked that first thing and it was in my web.config.

As for the duplicate route adding, I had it, so I guess my routes were being added two times. After commenting out the line XmlSiteMapController.RegisterRoutes(RouteTable.Routes);
the problem, unfortunatly, stayed the same.

If you come up with any other suggestions, I'd be glad to try. Meanwhile I am thinking, would it make sense to get the MvcSitemapProvider sources into the solution and debug with them present to be able to see "deeper"?

[maxpavlov]

Here is how fiddler sees the response that comes from the server (Local IIS):

[NightOwl888]

I agree you could probably save some time by stepping through the source to find the issue. See this post and #294 for some tips on how to set that up.

Since you already confirmed you are getting the right controller type, I am a little stumped. There must be something else going on in your configuration that is interfering with the output stream of XmlSiteMapResult.

[maxpavlov]

Ok, I've debugged the generation code:

As you can see, it generates the XDocument, no errors occure when RetrieveOutputStream is being call to create a stream. Still the same error. Looking into IIS setting for the site.

[NightOwl888]

ExecuteSitemapIndexResult is only executed if the SiteMap contains over 35,000 nodes. At that point it switches to generating an index page with multiple sitemap files (as per the spec).

It sounds like your browser isn't accepting the encoding for some reason. This might have something to do with the the culture of the web page (or culture in web.config), and/or your browser configuration.

It looks like the author of the code assumed that if no encoding were supplied, it would default to UTF-8 (which is how it is hard-coded into the XML). However, after doing some research it sounds like the default encoding would be UTF-16 depending on the culture of the machine that is generating it.

Also, the default includes a byte order mark that might not be accepted by your browser for some reason. However, search engines can generally deal with a BOM with no problem.

Try changing the lines starting at 164 to this, and let me know if it fixes the issue. Try it without the (false) argument, too to see whether it is the encoding or BOM that is causing the problem.

            // Write XML
            using (Stream outputStream = RetrieveOutputStream(context))
            {
                using (var writer = XmlWriter.Create(outputStream, new XmlWriterSettings { Encoding = new System.Text.UTF8Encoding(false) }))
                {
                    xmlSiteMap.WriteTo(writer);
                }
                outputStream.Flush();
            }

[maxpavlov]

Tried your suggestion. Still have the same problem. Switching from local IIS8 to IIS Express doens't solve the problem. Trying same thing in IE11, instead of Chrome, results in the same error. Haven't been able to google anything on such scenarios with IIS/ASP.NET MVC.

If you have any suggestion, you are welcome to ask me to test! Thanks =)

[NightOwl888]

What if you comment out the compression?

        protected virtual Stream RetrieveOutputStream(ControllerContext context)
        {
            // Output stream
            Stream outputStream = context.HttpContext.Response.OutputStream;

            //// Check if output can be GZip compressed
            //var headers = context.RequestContext.HttpContext.Request.Headers;
            //if (headers["Accept-encoding"] != null && headers["Accept-encoding"].ToLowerInvariant().Contains("gzip"))
            //{
            //    context.RequestContext.HttpContext.Response.AppendHeader("Content-encoding", "gzip");
            //    outputStream = new GZipStream(context.HttpContext.Response.OutputStream, CompressionMode.Compress);
            //}
            //else if (headers["Accept-encoding"] != null && headers["Accept-encoding"].ToLowerInvariant().Contains("deflate"))
            //{
            //    context.RequestContext.HttpContext.Response.AppendHeader("Content-encoding", "deflate");
            //    outputStream = new DeflateStream(context.HttpContext.Response.OutputStream, CompressionMode.Compress);
            //}
            return outputStream;
        }

It might help to know what culture your web page is running under. Hit a breakpoint and look at the result of System.Threading.Thread.CurrentThread.CurrentCulture.

[maxpavlov]

Ok, I will try these. Meanwhile, please review the datailed request trace from IIS, maybe you will see something strange there:
https://www.dropbox.com/s/elmx5rof40lsunv/W3SVC9.7z

[maxpavlov]

Ok, it works without compression.

[maxpavlov]

Ok, so at the point of pulling the OutputStream for filling, the thread's culture is uk-UA (Ukraine).
But changing it manually to en-US, as shown on the screen shot, doen't help. Same problem with compression.

[maxpavlov]

Even if I set the System.Threading.Thread.CurrentThread.CurrentCulture to en-US in ControllerFactory (before any controller code is being executed), this results in the same error. So I suppose the problem is in System.Threading.Thread.CurrentThread.CurrentCulture.

[NightOwl888]

Ok, try creating a brand new MVC 4 project and install MvcSiteMapProvider.MVC4. Fire it up and then look at the /sitemap.xml endpoint.

If it works, then I would suggest looking for something in your application that is messing with HTTP compression.

If the sample application doesn't work, then there is probably something going on in your IIS, root web.config, or machine.config settings that isn't normal. You could try running your application on another physical or virtual machine to rule that out.

I wouldn't recommend running it without compression, because the payload will be increased by several dozen times. But you could do that if all else fails.

Changing the culture probably won't have any effect (since we know the browser is not accepting the compressed stream already), but it would be best if you did it before any XML processing started to rule that out. Note that the next version of MvcSiteMapProvider will be generating the XML using the invariant culture (see #292).

[maxpavlov]

I've found the problem.

Our project is using a ClientDependency Framework to manage Dependencies. I assume it's handlers interact with compression at some point and that was leading to a problem.

I commented out these lines:

      <remove name="ClientDependencyModule" />
      <add name="ClientDependencyModule" type="ClientDependency.Core.Module.ClientDependencyModule, ClientDependency.Core" />

And the things started to work even with a compression.

[NightOwl888]

Glad to hear you worked it out.

[maxpavlov]

Just more info - ClientDependencyFramework does automatic gzipping of content for optimization. So we were getting that error becuase an already gzipped stream was being gzipped yet again by a handler code. Right now I am looking for a way on how to set up an exclusion for specifica path with a CDF author.