Unable to read big json file from Request.Body

[Tratcher]

From @anfomin on September 30, 2015 15:1

I'm trying to read big JSON file (UTF8) from Request.Body stream:

string content;
using (var sr = new StreamReader(Request.Body, Encoding.Default))
    content = sr.ReadToEnd();

At the beginning it's ok, but content variable contains invalid character sequence after some offset.

Sample repo: https://github.com/anfomin/DNX-MVC-bigfile. There is file sample.json. Just run dnx kestrel and make:

POST http://localhost:5000

with body = sample.json file.

You will find mismatches since line 84.

Source:

Result:

I've tried latest DNX 1.0.0-beta8-15736 from https://www.myget.org/F/aspnetvnext/api/v2 and 1.0.0-rc1-15775 from https://www.myget.org/F/aspnetcidev/api/v2.

Copied from original issue: aspnet/Mvc#3235

[Tratcher]

From @Eilon on September 30, 2015 18:49

@kichalla can you take a look?

[Tratcher]

From @kichalla on September 30, 2015 19:51

@anfomin I suppose you are running on Mono on Linux? was curious if Encoding.Default was doing something here but looks like some content is entirely missing based on your snapshot above...just to isolate the issue, can you try saving the read content to a file and see if the contents are any different? also does this issue repro consistently for you?

[Tratcher]

From @anfomin on September 30, 2015 20:38

@kichalla I tried this sample both on Windows and Mac, result is the same.

Also I tried to save Request.Body to a text file via File.CreateText("..."). Result file contains the same corruptions. You can remove Encoding.Default, the result does not change.

The problem is Request.Body because when I change input stream to File.Open(...) then output is correct.

[Tratcher]

From @kichalla on September 30, 2015 21:2

@anfomin you are right, I was able to repro it on a Windows machine now..investigating...

left -> actual file content
right -> content read from request body and saved to a file

[Tratcher]

From @Eilon on September 30, 2015 21:5

Weird... putting this on RC. Please enlist @halter73 @lodejard @Tratcher @moozzyk if needed because this might even be a Kestrel issue.

[Tratcher]

From @kichalla on September 30, 2015 21:13

Yeah, this seems like a Kestrel issue as I cannot repro it with weblistener

[lodejard]

I'll take a look also

[kichalla]

@loudej Probably you are the best person to take a look at it...I tried to figure it out but I got lost 😄 ..following are some details:

• This issue seems to repro only for specific request sizes and characters...for example, the below repro works fine if the character is ASCII...so my wild guess is that it has something to do with decoder and its flushing...

• You can use the following repro code
  I used the LargeResponseApp in Kestrel repo for this

  app.Run(async (context) =>
          {
              var reader = new StreamReader(context.Request.Body);
              var content = await reader.ReadToEndAsync();
  
              using (var sw = new StreamWriter(File.Create(@"C:\Users\kichalla\Documents\files\actual.txt")))
              {
                  await sw.WriteAsync(content);
              }
          });

  Client app:

  public class Program
  {
      public void Main(string[] args)
      {
          var filePath = @"C:\users\kichalla\documents\files\expected.txt";
          using (var streamWriter = File.CreateText(filePath))
          {
              streamWriter.Write((new string('й', 5 * 1024 * 1024)));
          }
          //--------
          var httpClient = new HttpClient();
          var response = httpClient.PostAsync("http://localhost:5000", new StreamContent(File.OpenRead(filePath))).Result;
          Console.WriteLine(response.ToString());
          Console.WriteLine(response.Content.ReadAsStringAsync().Result);
      }
  }

[lodejard]

Perfect! Thanks @Tratcher, @kichalla those should repro very well in a unit test also

[MJomaa]

I have the same issue with 1.0.0-beta8-15736. With beta8-15613 it worked fine.

Reproduce:

https://github.com/MJomaa/MalformingDataBug

This Repo just uploads a bunch of encoded HTML.

[muratg]

@MJomaa thanks for the info when it started to fail. This should give @lodejard some pointers in investigation.

[Tratcher]

@lodejard Repro functional test checked in: https://github.com/aspnet/KestrelHttpServer/blob/dev/test/Microsoft.AspNet.Server.Kestrel.FunctionalTests/RequestTests.cs#L19

[edurdias]

hi @halter73, thanks for working on this issue.

Would you be able to confirm which is the package version that contains the fix?

The last version I was able to use was 1.0.0-beta8-15674. If I try to use 1.0.0-beta8-15697 or 1.0.0-beta8-15702 it doesn't seems to work anymore.

[halter73]

@edurdias This should be fixed in the final beat8 package and all the rc1 prerelease packages.

It will probably take a little while digging through the CI logs to find the specific build number the fix was introduced in. Do you really need that?

[edurdias]

We were using older build version in some of our microservices but I did an experiment and we should fine upgrading them to final beta8 version.

Thanks for the quick feedback.