Add automatic removal of old installed targets #1318

lbonn · 2019-08-23T13:20:13Z

This adds a mechanism to remove old downloaded targets.

Here, "old" is what was before the current and the previous installed version and the cleaning is done at the end of the installation.

It's only enabled when using aktualizr's executable but can be added through the API as well.

This first version is simple. I was planning to add a more complex logic based on the previous log of installed versions (spinned off here: https://github.com/advancedtelematic/aktualizr/tree/feat/install-log) but now I'm not sure it's worth the trouble.

~~Main problem with this version is that it will probably make problems with rollbacks.~~

Also, we'd probably want to have something similar for ostree. (already here?)

pattivacek

Main problem with this version is that it will probably make problems with rollbacks.

Agreed. I like the idea, but I think we need to keep the current and previous targets and only get rid of anything older than that.

Also, we'd probably want to have something similar for ostree.

Agreed, but does OSTree do that automatically? I've never seen ostree admin status return more than two commits. Is it hiding older commits or silently cleaning them up?

tests/metafake.h

codecov-io · 2019-08-23T14:12:25Z

Codecov Report

Merging #1318 into master will increase coverage by 0.1%.
The diff coverage is 92.89%.

@@            Coverage Diff            @@
##           master    #1318     +/-   ##
=========================================
+ Coverage   79.72%   79.83%   +0.1%     
=========================================
  Files         177      178      +1     
  Lines       10475    10547     +72     
=========================================
+ Hits         8351     8420     +69     
- Misses       2124     2127      +3

Impacted Files	Coverage Δ
src/libaktualizr/storage/sqlstorage.h	`50% <ø> (ø)`	⬆️
src/aktualizr_lite/ostree_mock.cc	`100% <ø> (ø)`	⬆️
src/aktualizr_info/main.cc	`90.64% <100%> (-0.14%)`	⬇️
src/uptane_generator/repo.cc	`95.77% <100%> (+0.49%)`	⬆️
src/libaktualizr/primary/aktualizr.h	`100% <100%> (ø)`	⬆️
src/aktualizr_lite/main.cc	`76.75% <100%> (-0.13%)`	⬇️
src/libaktualizr/package_manager/ostreemanager.cc	`76.37% <100%> (-1.58%)`	⬇️
src/libaktualizr/package_manager/debianmanager.cc	`100% <100%> (ø)`	⬆️
...libaktualizr/package_manager/packagemanagerfake.cc	`90.9% <100%> (-0.32%)`	⬇️
src/libaktualizr/storage/invstorage.h	`97.56% <100%> (+0.12%)`	⬆️
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4cfe7a0...33d1973. Read the comment docs.

lbonn · 2019-08-23T16:03:13Z

Agreed. I like the idea, but I think we need to keep the current and previous targets and only get rid of anything older than that.

Ok, I'll push the installation log on top then and will think about a nice api to reuse that easily.

Agreed, but does OSTree do that automatically? I've never seen ostree admin status return more than two commits. Is it hiding older commits or silently cleaning them up?

Hm ok I didn't remember about that. Makes sense

lbonn · 2019-09-05T10:18:56Z

Ok I think it's close to the end. Was more complicated than expected...

pattivacek

Seems pretty cool!

config/sql/rollback/rollback.20.sql

pattivacek · 2019-09-05T14:15:57Z

src/libaktualizr/primary/aktualizr_test.cc

+
+    // only two targets are left: dummy_firmware has been cleaned up
+    std::vector<Uptane::Target> targets = aktualizr.GetStoredTargets();
+    ASSERT_EQ(targets.size(), 2);


This is a pretty awesome test, but do you think it's worth checking that the two remaining are the two we expect at the very end?

Yes, I'll add that.

doanac · 2019-09-05T15:03:53Z

My only nit is that I had to really read commit eb510df to get an understanding of what was going on here. "Feat/ota 3073" doesn't give readers much context. I think the intent of this is something along the lines of "The installed_versions table grows each time an update is applied. This change prunes out old rows from the table that are no longer needed"

lbonn · 2019-09-05T15:09:55Z

My only nit is that I had to really read commit eb510df to get an understanding of what was going on here. "Feat/ota 3073" doesn't give readers much context. I think the intent of this is something along the lines of "The installed_versions table grows each time an update is applied. This change prunes out old rows from the table that are no longer needed"

Fair points, I opened the PR long before it was finally ready. I've updated the opening description.

There should be some doc bits missing as well, actually.

lbonn · 2019-09-06T10:46:40Z

After further tests, the installation log actually doesn't work properly... Will push some fixes soon.

lbonn · 2019-09-09T08:49:39Z

There was a bug in the previous version. It should now be fixed and also simpler and better tested.

pattivacek

Looks cool, still just trying to make sure I understand it all.

pattivacek · 2019-09-09T09:49:41Z

src/libaktualizr/primary/aktualizr.h

+  /**
+   * Get log of installations. The log is indexed for every ECU and contains
+   * every change of versions ordered by time. It may contain duplicates in
+   * case of rollbacks.


Is this still true? Won't anything rolled back erase the prior generation number for that target?

What you describe is close to the old behavior.
It will now create a new generation and have another entry in the table.

pattivacek · 2019-09-09T09:50:15Z

config/sql/migration/migrate.20.sql

+ALTER TABLE installed_versions RENAME TO installed_versions_old;
+CREATE TABLE installed_versions(id INTEGER PRIMARY KEY, ecu_serial TEXT NOT NULL, sha256 TEXT NOT NULL, name TEXT NOT NULL, hashes TEXT NOT NULL, length INTEGER NOT NULL DEFAULT 0, correlation_id TEXT NOT NULL DEFAULT '', generation INTEGER, is_current INTEGER NOT NULL CHECK (is_current IN (0,1)) DEFAULT 0, is_pending INTEGER NOT NULL CHECK (is_pending IN (0,1)) DEFAULT 0, was_installed INTEGER NOT NULL CHECK (was_installed IN (0,1)) DEFAULT 0);
+CREATE TRIGGER trig_installed_versions_gen AFTER INSERT ON installed_versions FOR EACH ROW BEGIN UPDATE installed_versions SET generation = (SELECT IFNULL(MAX(generation), 0) FROM installed_versions WHERE ecu_serial = NEW.ecu_serial)+1 WHERE id = NEW.id; END;
+INSERT INTO installed_versions(ecu_serial, sha256, name, hashes, length, correlation_id, is_current, is_pending, was_installed) SELECT installed_versions_old.ecu_serial, installed_versions_old.sha256, installed_versions_old.name, installed_versions_old.hashes, installed_versions_old.length, installed_versions_old.correlation_id, installed_versions_old.is_current, installed_versions_old.is_pending, 1 FROM installed_versions_old ORDER BY rowid;


What is was_installed actually used for? I'm not quite following why that is needed.

installed_versions has is_current and is_pending columns to distinguish between what's currently running and staged from the rest.

was_installed is used to distinguish historical versions that were never current from the rest. ie: you set something to pending and end up installing some other version. The actual log would only contain versions that have actually been installed.

Additional note: we assume that all versions have been installed when migrating (hence, the 1). It seems to be better than reporting none.

Could we just determine if it was installed if the generation is > 0 (or >= 0)? Or does the generation value get set even if the target is pending, such that if it isn't actually installed, you'd still want to know that it was pending at one point?

That's a good idea but in this code, generation was also incremented for pending version.

In the end I realized it was still complex so I removed the generation field entirely and relies on id which should have sufficient guarantees for this purpose.
was_installed is still there in the new code.

I didn't even realize that id would probably work fine. Makes sense if you're copying entries for rollbacks/reinstalls. I think that means this is good to go, then.

pattivacek · 2019-09-09T09:51:12Z

src/libaktualizr/storage/sqlstorage_test.cc

@@ -346,6 +363,72 @@ TEST(sqlstorage, DbMigration18to19) {
    FAIL() << "Too many rows";
  }
 }
+
+TEST(sqlstorage, DbMigration19to20) {


What exactly is this test covering? Is it just making sure we don't erase things that were installed with version 19 after we migrate to version 20 since they don't have generation numbers?

It's testing the migration with realistic data indeed. It gives a bit more confidence than just checking the table shapes, like schema_migration_test.sh does.

Signed-off-by: Laurent Bonnans <[email protected]>

This function returns empty json for failure. Throwing here ended up masking the real issue. Signed-off-by: Laurent Bonnans <[email protected]>

Signed-off-by: Laurent Bonnans <[email protected]>

Most of the time it was used to get current and pending versions. For the other cases, provide loadInstallationLog which also returns the ordered log of installation. Signed-off-by: Laurent Bonnans <[email protected]>

Put it in a separate source so it can be more easily tested and won't pollute aktualizr_primary's main too much. Only deletes until the second last version. Signed-off-by: Laurent Bonnans <[email protected]>

Signed-off-by: Laurent Bonnans <[email protected]>

lbonn requested review from pattivacek, eu-siemann, mike-sul, Zee314159, xcheng-here and kbushgit August 23, 2019 13:20

lbonn force-pushed the feat/OTA-3073/aktualizr-autoclean branch from 4b30ec9 to 784f4bb Compare August 23, 2019 13:21

pattivacek reviewed Aug 23, 2019

View reviewed changes

tests/metafake.h Outdated Show resolved Hide resolved

lbonn force-pushed the feat/OTA-3073/aktualizr-autoclean branch from 784f4bb to 3bd213d Compare August 23, 2019 16:04

lbonn force-pushed the feat/OTA-3073/aktualizr-autoclean branch 4 times, most recently from e40b89e to a284818 Compare September 5, 2019 10:16

lbonn force-pushed the feat/OTA-3073/aktualizr-autoclean branch from a284818 to f2c95ef Compare September 5, 2019 12:14

pattivacek reviewed Sep 5, 2019

View reviewed changes

lbonn changed the title ~~Feat/ota 3073/aktualizr autoclean~~ Add automatic removal of old installed targets Sep 5, 2019

lbonn added the do-not-merge-until-ready label Sep 6, 2019

lbonn force-pushed the feat/OTA-3073/aktualizr-autoclean branch 3 times, most recently from 95db449 to 3267861 Compare September 6, 2019 16:28

lbonn removed the do-not-merge-until-ready label Sep 9, 2019

pattivacek reviewed Sep 9, 2019

View reviewed changes

lbonn force-pushed the feat/OTA-3073/aktualizr-autoclean branch from 3267861 to 5efa6fe Compare September 9, 2019 10:04

lbonn added 7 commits September 9, 2019 14:31

Fix a missing include

41de002

Signed-off-by: Laurent Bonnans <[email protected]>

Systematic isTypeOf for objects in events.h

abfb0fe

Signed-off-by: Laurent Bonnans <[email protected]>

Remove some zealous throws in uptane-generator

395bdc0

This function returns empty json for failure. Throwing here ended up masking the real issue. Signed-off-by: Laurent Bonnans <[email protected]>

Keep a log of installed versions

85b4162

Signed-off-by: Laurent Bonnans <[email protected]>

Split loadInstalledVersions into two

0ea8674

Most of the time it was used to get current and pending versions. For the other cases, provide loadInstallationLog which also returns the ordered log of installation. Signed-off-by: Laurent Bonnans <[email protected]>

Add a simple autoclean feature for aktualizr daemon

a2a8a40

Put it in a separate source so it can be more easily tested and won't pollute aktualizr_primary's main too much. Only deletes until the second last version. Signed-off-by: Laurent Bonnans <[email protected]>

Document autoclean and release log features

33d1973

Signed-off-by: Laurent Bonnans <[email protected]>

lbonn force-pushed the feat/OTA-3073/aktualizr-autoclean branch from 5efa6fe to 33d1973 Compare September 9, 2019 12:35

pattivacek approved these changes Sep 9, 2019

View reviewed changes

mike-sul approved these changes Sep 9, 2019

View reviewed changes

lbonn merged commit 5d2c0b6 into master Sep 9, 2019

lbonn deleted the feat/OTA-3073/aktualizr-autoclean branch September 9, 2019 14:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add automatic removal of old installed targets #1318

Add automatic removal of old installed targets #1318

lbonn commented Aug 23, 2019 •

edited

Loading

pattivacek left a comment

codecov-io commented Aug 23, 2019 •

edited

Loading

lbonn commented Aug 23, 2019

lbonn commented Sep 5, 2019

pattivacek left a comment

pattivacek Sep 5, 2019

lbonn Sep 5, 2019

doanac commented Sep 5, 2019

lbonn commented Sep 5, 2019

lbonn commented Sep 6, 2019

lbonn commented Sep 9, 2019

pattivacek left a comment

pattivacek Sep 9, 2019

lbonn Sep 9, 2019

pattivacek Sep 9, 2019

lbonn Sep 9, 2019

lbonn Sep 9, 2019

pattivacek Sep 9, 2019

lbonn Sep 9, 2019 •

edited

Loading

pattivacek Sep 9, 2019

pattivacek Sep 9, 2019

lbonn Sep 9, 2019

Add automatic removal of old installed targets #1318

Add automatic removal of old installed targets #1318

Conversation

lbonn commented Aug 23, 2019 • edited Loading

pattivacek left a comment

Choose a reason for hiding this comment

codecov-io commented Aug 23, 2019 • edited Loading

Codecov Report

lbonn commented Aug 23, 2019

lbonn commented Sep 5, 2019

pattivacek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

doanac commented Sep 5, 2019

lbonn commented Sep 5, 2019

lbonn commented Sep 6, 2019

lbonn commented Sep 9, 2019

pattivacek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbonn Sep 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbonn commented Aug 23, 2019 •

edited

Loading

codecov-io commented Aug 23, 2019 •

edited

Loading

lbonn Sep 9, 2019 •

edited

Loading