Applying several delta updates on a big package (400MB+) takes a lot of time

[ncoussemacq]

I'm using Squirrel to deploy updates on an app for which the full package is more than 400MB big.
One of the benefit of Squirrel is the support of delta updates, which, in my case, boils down to a ~10MB download for the user, which is great. The problem is when several delta updates have to be applied in a row. In my case, on a pretty good system, applying ~8 delta updates takes about 20 minutes (on an SSD drive).
Is there anything we can do about this ? Any plan to improve that part of the system ? It looks like a full package is built each time a delta is applied, whereas it's probably not required. I guess that generating the full package only at the end of the process should be enough and time saving.
What do you think ?

Thanks for that great tool !
Nicolas

[anaisbetts]

@ncoussemacq That's pretty rough - we intentionally optimize for bandwidth size at the cost of CPU time. One thing you could do is start to create "partial delta" updates, that had more full files and less deltas, but I'm not sure what a good heuristic to make that happen is.

[gpailler]

I think the point reported by @ncoussemacq is Squirrel generates a full package for each delta package applied.

If you have version 1.0.0 installed and delta packages for 1.0.1 and 1.0.2 available, then Squirrel will apply 1.0.1 delta update, generates full package for 1.0.1 then apply 1.0.2 delta update and generates full package again for 1.0.2. The intermediate full package 1.0.1 seems useless.

Do you think it's easy to not generate intermediate full packages when Squirrel apply multiple delta updates (and save time of zip compression for each intermediate full package)?

[anaisbetts]

The intermediate full package 1.0.1 seems useless.

Delta patches are strictly for the previous version, so in order to get to 1.0.2 via delta, we need 1.0.1

[gpailler]

My point is only to avoid recompressing intermediate full pacakges when applying multiple delta updates.

• You have version 1.0.0 full package
• You uncompress version 1.0.0 in Squirrel temp folder
• You apply delta update 1.0.1 on the temp folder (your uncompressed 1.0.0 version becomes 1.0.1)
• You apply delta update 1.0.2 on the temp folder (your uncompressed 1.0.1 version becomes 1.0.2)
• You recompress the Squirrel temp folder to generate a version 1.0.2 full package

Currently, after each delta update, a full package is recreated but I think Squirrel update process should work without this intermediate full package.

[anaisbetts]

@gpailler Oh yes, I'm into that

[ncoussemacq]

Do you need any help on that ?

I could be able to test any new behavior quite easily, if you want.

[anaisbetts]

@ncoussemacq I'd take that PR

[natahn]

Has there been any update on this issue? I have just recently started using Squirrel and have run into the same issue with larger packages and multiple delta updates.

[sandeep1995]

@paulcbetts @ncoussemacq Any update in this issue?

[stephenfung]

I was prototyping an implementation of this performance optimization here:
https://github.com/ArchonSystemsInc/squirrel-optimization

It

• Extracts the current full .nupkg to a working directory
• Applies each required delta file to that working directory (without repackaging it as a .nupkg)
• Copies the working directory to the final directory
• Repackages the working directory as a .nupkg (since this is referenced for future updates)

This skips various steps compared to the current Squirrel behaviour for applying multiple deltas, and in our testing was a little faster for a single delta and much faster for multiple deltas.

Our organization decided not to proceed with this, since looking at our particular tradeoffs it made more sense to disable deltas and do full installs instead, but I wanted to share this in case it's a useful starting point for anyone else.

Apologies if I've broken any etiquette, I'm pretty new to contributing on Github! I didn't think this was ready for a pull request, since it's only been lightly tested and would break the "Be extremely conservative" Guiding Principle.

[lukeskinner]

I have found an area of slowness during profiling of deltas on packages similar size to what was originally reported.

Squirrel.Windows/src/Squirrel/DeltaPackage.cs

Line 291 in 51f5e2c

File.Move(tempTargetFile, finalTarget);

File.Move(tempTargetFile, finalTarget);

This goes off to the windows api to be moved but its incredibly slow.

Replacing this move with the FMove method from here https://www.codeproject.com/Tips/777322/A-Faster-File-Copy
reduces the time from 62 seconds to 800ms. The main difference being this copies the file using a larger buff size of 500kb and then deletes the original once the copy is complete.

I haven't got this in a pull request as I'm still testing parts of this out to see what can be improved further.

[sandeep1995]

If you migrate to NSIS, then I have created my own solution using binary delta.
Checkout here https://github.com/electron-delta/electron-delta

[anaisbetts]

@caesay I think you've advertised enough. Please don't comment on every issue.

[caesay]

@anaisbetts I don't comment on every issue. Maybe I respond on 1/3 notifications, and I only respond when I have legitimate answers or where people are encountering issues that are already fixed elsewhere. I'm just trying to be helpful. You just deleted a comment that had a very helpful answer.

This library has a flaw in delta updates. I'll repeat briefly because anyone else finding this issue should know. Delta updates in this library are seriously flawed. For each delta that is applied, the whole release package is unzipped and rezipped. If you are applying multiple deltas, this happens multiple times. This is very slow. And totally unnecessary. There are several other flaws too. There is a fork that fixes this. Apparently, I'm no longer allowed to link to it; perhaps some other reader might take that initiative.

Aside from you, @anaisbetts, I suspect I'm the most knowledgeable person on this project because I've been working on it for years. If you don't want me to link to my fork, which fixes most if not all of the issues here, I'll stop linking it here and just mention that one (somewhere) exists, but I'm not sure why you'd want me to stop providing real advice and insights about how this library works - and regularly providing workarounds too for people who want to continue using the original library.

[lukeskinner]

I don't want to stir anything but I am using this library in WPF so that doesn't help me too much anything electron specific.
The delta system is pretty great for me, I don't need instant deltas but any small wins I'm looking for. I'll stop using this issue and move to my pull request when it's ready. apologies!