Unable to open large files (543 MB)

[wstaelens]

• VSCode Version: Code 1.6.1 (9e4e44c, 2016-10-13T16:21:53.542Z)
• OS Version: Windows_NT ia32 10.0.14393
• Extensions: ms-vscode.csharp-1.4.1

Steps to Reproduce:

1. I wanted to compare two .reg files (exported registry files. One file was 400+MB other file 500MB+
2. I'm only be able to open it in Visual Studio 2015 and run the Tools.DiffFiles (which takes ages) but even notepad or notepad++ is unable to open these big registry files (and it is only 556 848 KB).

[wojtekmaj]

Erm, half a gigabyte is not "only", I'm surprised you've found a single program capable of opening it, really. All Shakespeare writings should take around 5 394 968 bytes so you could fit it 100 times in your file! ;)

Sources:
http://www.opensourceshakespeare.org/stats/
http://www.wolframalpha.com/input/?i=average+english+word+length

[wstaelens]

We are 2016, not 1616.
More data, bigger log files, bigger hard drives, bigger registry files, bigger game installations, bigger windows installations, everything grows..

It is not so unusual to have such big files. I'm not talking about 1GB or more data files.
If we could not open these big files on Windows XP or Vista, I would agree. We can send space crafts to mars but unable to open a 500MB file.

[wojtekmaj]

Have you tried using software specifically made for comparing files, especially command line based ones? They have far less things to worry about than doing-it-all tools like VSCode or Npp.

[wstaelens]

Comparing the files was optionally. the ticket I've created is about the first step, opening a file that fails. I didn't create a ticket about the comparing files functionality, only described what I did, or wanted to do, which failed at the first step.)

So more general: VS Code is unable to open big files. and I reported this in the hope either somebody would fix it or give a decent reply on why this is technically not possible instead of talking about Shakespeare or other personal reasons why you don't open big files.

(If you want that people give feedback on a product, just don't start asking why other people want to do something their way or give these b*llshit Shakespeare comments. I just want to open a big file, which is in my eyes not that big. ok?).
I almost starting to regret that I've posted feedback.

[kaiwood]

@wstaelens Please don't be upset, @wojtekmaj was only trying to help you out with your problem. You have to keep in mind that this is an open source project and the Github issues are not "support tickets" in the classical way, so normal users like you and me chime in to comment on stuff too, they are not all Microsoft representatives. I'm sure your feedback will be heard, no reason to regret anything.

We are all just trying to move this project further down the road because its obvious that its quite a challenge for the VSCode team to keep up with all the open issues here. :)

About the technical limitations when opening big files:

This is not an easy problem to solve. Code editors are build in a way where they need to read the whole file at once and keep it all in memory. If you open 2 files with 500MB each, that would be a full GB of RAM for the file data alone. But unfortunately that's not the final memory size, the editor needs to attach lots of meta data (for example to make things like syntax highlighting possible) that use even more memory.

Even in 2016, it will take too long to open the file and the whole application will freeze or will be unresponsive to anything when its open – at least if you don't massively optimise for such a task.

That's why specialised tools like command line pagers and diff tools are way better at this. For example, they can read files line by line and "throw away" what they already seen, keeping only the information they need. This way they need only a fraction of the memory a code editor would need.

[wstaelens]

@kaiwood I understand, but it would be great if VSCode could open big files.
Most people who work with big files have already 8GB or 16GB of ram. They understand that it takes a lot of memory and sometimes they don't care to wait 10minutes before the file is loaded.

Most devs/admins want to keep the amount of tools to install on a pc or server as small as possible. Opening/configuring everything with VS Code is great, but also viewing a log file (yes IIS for example still produces a sh*tload on data ;-) ) is something that happens very often.
Having to install another tool only for this is yet another annoying thing.

So would be great to see this in VS Code.

[kaiwood]

Absolutely, I'd love to see support for big files too.

I checked the other issues about the topic, looks like the team already have it on their radar if you read the comments here: #9832

[alexdima]

Let's track in #9832