-
Notifications
You must be signed in to change notification settings - Fork 61
Conversation
Parallel writes to SQLite database over-complicate the code for little benefit Signed-off-by: Anton Gerasimov <[email protected]>
Signed-off-by: Anton Gerasimov <[email protected]>
Before we merge this, we should double-check that no one is relying on something that we will break. I'll look into that. Conceptually, though, I have been convinced that this is the right way forward. |
@patrickvacek What do you mean? Right now we only have sachertorte project as an external user, and I'm pretty sure it's fine. |
@OYTIS I want to make sure that no one else was expecting this mis-feature. :) Also, could you try adding the feature to implicitly pause when calling shutdown via the API? |
Codecov Report
@@ Coverage Diff @@
## master #1031 +/- ##
==========================================
+ Coverage 82.29% 82.31% +0.02%
==========================================
Files 189 189
Lines 13655 13663 +8
==========================================
+ Hits 11237 11247 +10
+ Misses 2418 2416 -2
Continue to review full report at Codecov.
|
Do you think it is reasonable to actually measure the download times with and without this change? I believe this should be an improvement, and I also know it will make things easier to maintain, but it's worth knowing what effects this has in practice. |
@patrickvacek Yes, makes sense to measure. We know the current speed already, it's about 64 kbit/s. When this PR is merged I'll create another one with updated fetcher test, which measures the speed. |
8c988a6
to
eb2a3a0
Compare
That should increase the download speed and reduce the wear load. Signed-off-by: Anton Gerasimov <[email protected]>
Some tests are failing in the debian package manager case (which is pretty obsolete, but still...) |
eb2a3a0
to
d810a39
Compare
@lbonn Yes, it uses WHandle directly, fixed that. |
@eu-smirnov Thank you, I was wondering how this should be tested. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks fine, and I'm curious to see how it works in action!
} | ||
// It is possible for the PutManifestComplete to come before we get the | ||
// InstallTargetComplete depending on the threading, so check for both. | ||
if (allcomplete_FullMultipleSecondaries && complete_FullMultipleSecondaries == 2 && | ||
manifest_FullMultipleSecondaries) { | ||
LOG_ERROR << "SETVALUE"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this leftover from testing or intentional?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@patrickvacek That's debug output, thanks, I'll remove it.
} | ||
|
||
~SQLTargetWHandle() override { | ||
if (!closed_) { | ||
LOG_WARNING << "Handle for file " << target_.filename() << " has not been committed or aborted, forcing abort"; | ||
db_.commitTransaction(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Didn't notice it before, but why do we call commit here? We want to abort the transaction, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not quite sure actually. My idea was that if we don't abort explicitly then we probably want to save what we've already downloaded.
By the way, maybe we don't need explicit pause
from Aktualizr::Shutdown
in the end, @patrickvacek's commit confused me somewhat. Just need to add the test that the file really gets committed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@OYTIS good call, actually! I overlooked the commit in the destructor, which is actually an even better solution than the implicit call to pause.
I locally rebased my changes to fetcher/fetcher test to measure the download speed. It certainly got much better, up to 4 Mbit/s from previous 64 kbit/s. |
@eu-smirnov Could you please share the rebased code on some branch? I might try to debug that as well. |
Signed-off-by: Anton Gerasimov <[email protected]>
@OYTIS Sure, pushed it to |
@eu-smirnov Thanks! |
d810a39
to
4086bda
Compare
OK, as we've found, transactions around This PR already improves the speed significantly, but we can do better by keeping the blob open. Let's merge this one (after @eu-smirnov tries it with his #1032), and make the rest of improvements after that. |
Blob handler is now kept open between the calls to wfeed Signed-off-by: Anton Gerasimov <[email protected]>
12ecdd5
to
51fd83d
Compare
OK, so it was relatively simple, no schema redesign needed. Let's rebase it on @eu-smirnov's PR and measure the speed now. |
This looks fine to me, but I'd prefer someone with better SQL knowledge to sign off if possible.
It doesn't sound like anyone will be upset if we remove it, especially since the numbers clearly demonstrate that this is a major improvement. (Or more accurately, this retracts what we now know to have been a downgrade, and it appears to moderately improve upon what we had before that.) |
int err = statement.step(); | ||
if (err != SQLITE_DONE) { | ||
LOG_ERROR << "Could not save size in db: " << db_.errmsg(); | ||
throw StorageTargetWHandle::WriteError("could not update sise of " + target_.filename() + " in sql storage"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo: "size"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks.
if (blob_ != nullptr) { | ||
sqlite3_blob_close(blob_); | ||
blob_ = nullptr; | ||
auto statement = db_.prepareStatement<int64_t, int64_t>("update target_images SET real_size = ? where rowid = ?;", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
really small, but we upper case sql operations everywhere else, would be nice to keep a consistent style for easy grepping etc
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, will fix.
Looks good. I don't know if I really have more knowledge of sql but it seems to be consistent with the approached we agreed with. |
Awesome, now it takes 7 seconds to download a 100 MB file, which is around 120 Mbit/s or ~2000 times faster. Let's merge it, guys :) |
@eu-smirnov Is this still happening? My guess is no, but just wanted to be sure... |
@patrickvacek nope, that one is also seems to be fixed |
Signed-off-by: Anton Gerasimov <[email protected]>
No description provided.