Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add oe-selftest for aktualizr ptest run on qemu #508

Merged
merged 6 commits into from
Apr 15, 2019

Conversation

lbonn
Copy link
Contributor

@lbonn lbonn commented Mar 22, 2019

Still has some failures at the moment, also the full output is often cropped for some unknown reason (the summary of which tests are passing and which aren't still appears in full)

@lbonn
Copy link
Contributor Author

lbonn commented Mar 22, 2019

Annoyingly, a test using lshw fails because /usr/sbin is not put into $PATH when launching ptest-runner directly through ssh.

@pattivacek
Copy link
Collaborator

Softlink it manually in the oe-selftest code before calling ptest?

@pattivacek
Copy link
Collaborator

Looks fine, but in my case I see a few more errors:

The following tests FAILED:
[ERROR_MESSAGE] 
Problem opening file: /usr/lib/aktualizr/ptest/build/Testing/Temporary/LastTestsFailed.log

Cannot create log file: LastTestsFailed.log
[HANDLER_OUTPUT] 
	  4 - test_utils (Failed)

	 18 - test_uptane (Failed)
	 26 - test_fetcher (Failed)
[ERROR_MESSAGE] 
Errors while running CTest

The first you already know:

[ RUN      ] Utils.getHardwareInfo
sh: lshw: command not found
[2019-03-25 09:32:09.614242] [0x000000000563df00] [warning] Could not execute lshw (is it installed?).
/usr/src/debug/aktualizr/1.0+gitAUTOINC+2e3ccbbdd4-7/git/src/libaktualizr/utilities/utils_test.cc:52: Failure
Expected: (hwinfo) != (Json::Value()), actual: {} vs {}
/usr/src/debug/aktualizr/1.0+gitAUTOINC+2e3ccbbdd4-7/git/src/libaktualizr/utilities/utils_test.cc:53: Failure
Value of: hwinfo.isArray()
  Actual: true
Expected: false
[  FAILED  ] Utils.getHardwareInfo (602 ms)

The second:

/usr/src/debug/aktualizr/1.0+gitAUTOINC+2e3ccbbdd4-7/git/src/libaktualizr/uptane/uptane_test.cc:755: Failure
Expected equality of these values:
  http->system_info_count
    Which is: 0
  1
[  FAILED  ] Uptane.ProvisionOnServer (1844 ms)

fetcher_test.cc appears to be breaking due to a valgrind complaint:

==420== 184 (88 direct, 96 indirect) bytes in 1 blocks are definitely lost in loss record 1,015 of 1,050
==420==    at 0x4835753: malloc (vg_replace_malloc.c:299)
==420==    by 0x516E590: g_malloc (in /usr/lib/libglib-2.0.so.0.5800.3)
==420==    by 0x51861C2: g_slice_alloc (in /usr/lib/libglib-2.0.so.0.5800.3)
==420==    by 0x5156C5D: g_hash_table_new_full (in /usr/lib/libglib-2.0.so.0.5800.3)
==420==    by 0x5142CB0: g_get_language_names_with_category (in /usr/lib/libglib-2.0.so.0.5800.3)
==420==    by 0x515E49B: ??? (in /usr/lib/libglib-2.0.so.0.5800.3)
==420==    by 0x51602FC: g_key_file_new (in /usr/lib/libglib-2.0.so.0.5800.3)
==420==    by 0x4E81369: ostree_repo_reload_config (in /usr/lib/libostree-1.so.1.0.0)
==420==    by 0x4E8243B: ??? (in /usr/lib/libostree-1.so.1.0.0)
==420==    by 0x4E826E1: ??? (in /usr/lib/libostree-1.so.1.0.0)
==420==    by 0x4E82761: ostree_repo_open_at (in /usr/lib/libostree-1.so.1.0.0)
==420==    by 0x4EAC874: ??? (in /usr/lib/libostree-1.so.1.0.0)
==420==    by 0x4EAE8B6: ostree_sysroot_load_if_changed (in /usr/lib/libostree-1.so.1.0.0)
==420==    by 0x1DA78C: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x1DD2CE: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x1B0CA2: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x13EB58: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x54B496E: ??? (in /usr/lib/libstdc++.so.6.0.25)
==420==    by 0x4AC4F04: start_thread (pthread_create.c:486)
==420==    by 0x568DBAE: clone (clone.S:95)
==420==
==420== 400 bytes in 1 blocks are possibly lost in loss record 1,035 of 1,050
==420==    at 0x48379A2: calloc (vg_replace_malloc.c:752)
==420==    by 0x40113E1: allocate_dtv (dl-tls.c:286)
==420==    by 0x4011D2D: _dl_allocate_tls (dl-tls.c:532)
==420==    by 0x4AC5AE6: allocate_stack (allocatestack.c:621)
==420==    by 0x4AC5AE6: pthread_create@@GLIBC_2.2.5 (pthread_create.c:669)
==420==    by 0x54B4BF4: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/libstdc++.so.6.0.25)
==420==    by 0x13FBC3: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x141C6C: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x21A8E9: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x211779: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x2118C7: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x2119A4: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x211E9B: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x21ADF9: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x211FCB: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x13CBE7: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==420==    by 0x55BDB6A: (below main) (libc-start.c:308)

lbonn added 4 commits April 9, 2019 14:33
lshw is installed in `/usr/sbin` which does not appear when a command is
launched outside a non-login shell.

Signed-off-by: Laurent Bonnans <[email protected]>
@lbonn lbonn force-pushed the test/oe-selftest-ptest branch from 2f69ed3 to f501bb9 Compare April 10, 2019 08:26
@lbonn
Copy link
Contributor Author

lbonn commented Apr 10, 2019

I think your second problem comes from not being able to run lshw as well. I've pushed a fix.

I now have:

[ RUN      ] AktualizrCertProviderTest.DeviceCertParams
created: /tmp/aktualizr-0fb7-9361-9758-c87b/0a95-8c81-dir
/usr/src/debug/aktualizr/1.0+gitAUTOINC+56a0c04aff-7/git/src/cert_provider/cert_provider_test.cc:256: Failure
Expected: (openssl.lastStdOut().find(expected_key_str)) != (std::string::npos), actual: 18446744073709551615 vs 18446744073709551615
[  FAILED  ] AktualizrCertProviderTest.DeviceCertParams (74 ms)

@lbonn
Copy link
Contributor Author

lbonn commented Apr 10, 2019

Oh and the problem with fetcher is more than a memory leak I think:

34: [ RUN      ] Fetcher.PauseBinary
34: created: /tmp/aktualizr-3646-0ac6-3181-0dcc/b7bb-8ff2-dir
34: Bootstraping DB to version 19
34: No file 'large_file' with matched hash in the database
34: progress callback: 1
...
34: terminate called after throwing an instance of 'std::future_error'
34:   what():  std::future_error: Promise already satisfied
34: ==286==
34: ==286== Process terminating with default action of signal 6 (SIGABRT)
34: ==286==    at 0x55CC6FA: raise (raise.c:51)
34: ==286==    by 0x55B8523: abort (abort.c:79)
34: ==286==    by 0x5481972: ??? (in /usr/lib/libstdc++.so.6.0.25)
34: ==286==    by 0x5487895: ??? (in /usr/lib/libstdc++.so.6.0.25)
34: ==286==    by 0x54878D0: std::terminate() (in /usr/lib/libstdc++.so.6.0.25)
34: ==286==    by 0x5487B02: __cxa_throw (in /usr/lib/libstdc++.so.6.0.25)
34: ==286==    by 0x548391C: ??? (in /usr/lib/libstdc++.so.6.0.25)
34: ==286==    by 0x14CDC5: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/package_manager/t_fetcher)
34: ==286==    by 0x54B096E: ??? (in /usr/lib/libstdc++.so.6.0.25)
34: ==286==    by 0x4AC3F04: start_thread (pthread_create.c:486)
34: ==286==    by 0x5689BAE: clone (clone.S:95)

The trick to add it in do_compile from cmake.bbclass does not work with
do_compile_ptest_base

Signed-off-by: Laurent Bonnans <[email protected]>
@lbonn
Copy link
Contributor Author

lbonn commented Apr 10, 2019

Fixes in advancedtelematic/aktualizr#1173 and advancedtelematic/aktualizr#1174.

The valgrind leak however remains, I'd suggest that we'd still merge this soon if possible.

@pattivacek
Copy link
Collaborator

The valgrind leak however remains, I'd suggest that we'd still merge this soon if possible.

I don't want to merge if it still breaks, but I'd also like to get it in soon. Can we skip that test for now or do something quick and dirty until we fix it for real?

@lbonn
Copy link
Contributor Author

lbonn commented Apr 10, 2019

I thought about merging it now since it doesn't run with the other oe-selftests, as it is in its own .py file.
But ok I can take a last quick look.

Quick solution would be to run with NO_VALGRIND=1, that I've added in the aktualizr test suite but I don't like it since it would apply to the complete suite.

@lbonn
Copy link
Contributor Author

lbonn commented Apr 11, 2019

Ok I think the leak was introduced here: ostreedev/ostree@21ebc7d, which adds a new char* into the OstreeRepo object but didn't add it to the cleanup function.

@lbonn
Copy link
Contributor Author

lbonn commented Apr 11, 2019

Oh my bad that's a completely different leak, only in master! It looks like the one mentioned earlier has been fixed there actually.

@pattivacek
Copy link
Collaborator

At this point everything looks fine except for fetcher_test:

==741== 184 (88 direct, 96 indirect) bytes in 1 blocks are definitely lost in loss record 1,017 of 1,054
==741==    at 0x4835753: malloc (vg_replace_malloc.c:299)
==741==    by 0x516E5C0: g_malloc (in /usr/lib/libglib-2.0.so.0.5800.3)
==741==    by 0x51865C2: g_slice_alloc (in /usr/lib/libglib-2.0.so.0.5800.3)
==741==    by 0x5156C8D: g_hash_table_new_full (in /usr/lib/libglib-2.0.so.0.5800.3)
==741==    by 0x5142CE0: g_get_language_names_with_category (in /usr/lib/libglib-2.0.so.0.5800.3)
==741==    by 0x515E4CB: ??? (in /usr/lib/libglib-2.0.so.0.5800.3)
==741==    by 0x516032C: g_key_file_new (in /usr/lib/libglib-2.0.so.0.5800.3)
==741==    by 0x4E81369: ostree_repo_reload_config (in /usr/lib/libostree-1.so.1.0.0)
==741==    by 0x4E8243B: ??? (in /usr/lib/libostree-1.so.1.0.0)
==741==    by 0x4E826E1: ??? (in /usr/lib/libostree-1.so.1.0.0)
==741==    by 0x4E82761: ostree_repo_open_at (in /usr/lib/libostree-1.so.1.0.0)
==741==    by 0x4EAC874: ??? (in /usr/lib/libostree-1.so.1.0.0)
==741==    by 0x4EAE8B6: ostree_sysroot_load_if_changed (in /usr/lib/libostree-1.so.1.0.0)
==741==    by 0x1DA78C: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x1DD2CE: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x1B0CA2: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x13EB58: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x54B196E: ??? (in /usr/lib/libstdc++.so.6.0.25)
==741==    by 0x4AC4F04: start_thread (pthread_create.c:486)
==741==    by 0x568ABAE: clone (clone.S:95)
==741== 
==741== 400 bytes in 1 blocks are possibly lost in loss record 1,037 of 1,054
==741==    at 0x48379A2: calloc (vg_replace_malloc.c:752)
==741==    by 0x40113E1: allocate_dtv (dl-tls.c:286)
==741==    by 0x4011D2D: _dl_allocate_tls (dl-tls.c:532)
==741==    by 0x4AC5AE6: allocate_stack (allocatestack.c:621)
==741==    by 0x4AC5AE6: pthread_create@@GLIBC_2.2.5 (pthread_create.c:669)
==741==    by 0x54B1BF4: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/libstdc++.so.6.0.25)
==741==    by 0x13FBC3: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x141C6C: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x21A8E9: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x211779: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x2118C7: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x2119A4: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x211E9B: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x21ADF9: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x211FCB: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x13CBE7: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x55BAB6A: (below main) (libc-start.c:308)
==741== 
==741== 400 bytes in 1 blocks are possibly lost in loss record 1,038 of 1,054
==741==    at 0x48379A2: calloc (vg_replace_malloc.c:752)
==741==    by 0x40113E1: allocate_dtv (dl-tls.c:286)
==741==    by 0x4011D2D: _dl_allocate_tls (dl-tls.c:532)
==741==    by 0x4AC5AE6: allocate_stack (allocatestack.c:621)
==741==    by 0x4AC5AE6: pthread_create@@GLIBC_2.2.5 (pthread_create.c:669)
==741==    by 0x51B2A2F: ??? (in /usr/lib/libglib-2.0.so.0.5800.3)
==741==    by 0x519148E: ??? (in /usr/lib/libglib-2.0.so.0.5800.3)
==741==    by 0x5191537: g_thread_new (in /usr/lib/libglib-2.0.so.0.5800.3)
==741==    by 0x5169CB3: ??? (in /usr/lib/libglib-2.0.so.0.5800.3)
==741==    by 0x4FA65F6: ??? (in /usr/lib/libgio-2.0.so.0.5800.3)
==741==    by 0x4FA69B4: g_task_get_type (in /usr/lib/libgio-2.0.so.0.5800.3)
==741==    by 0x4FA6B78: g_task_new (in /usr/lib/libgio-2.0.so.0.5800.3)
==741==    by 0x4ECEE73: ??? (in /usr/lib/libostree-1.so.1.0.0)
==741==    by 0x4ECFF5F: ??? (in /usr/lib/libostree-1.so.1.0.0)
==741==    by 0x4ECD0AE: ??? (in /usr/lib/libostree-1.so.1.0.0)
==741==    by 0x4E94A64: ??? (in /usr/lib/libostree-1.so.1.0.0)
==741==    by 0x4E9D0D5: ostree_repo_pull_with_options (in /usr/lib/libostree-1.so.1.0.0)
==741==    by 0x1DDBE4: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x1B0CA2: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x13EB58: ??? (in /usr/lib/aktualizr/ptest/build/src/libaktualizr/uptane/t_fetcher)
==741==    by 0x54B196E: ??? (in /usr/lib/libstdc++.so.6.0.25)
==741==    by 0x4AC4F04: start_thread (pthread_create.c:486)
==741==    by 0x568ABAE: clone (clone.S:95)

Is that what you see as well?

@lbonn
Copy link
Contributor Author

lbonn commented Apr 12, 2019

Yes it's a very frustrating one, as I get it on yocto and my Debian system but not on the docker CI, even when trying the same ostree versions.
Will try a little bit more.

@lbonn
Copy link
Contributor Author

lbonn commented Apr 12, 2019

On ostree master (89c00e7eb72ddc5aaee3469f281fb5560ff7bd5d), with debug symbols:

32: ==3671==
32: ==3671== HEAP SUMMARY:
32: ==3671==     in use at exit: 180,461 bytes in 2,401 blocks
32: ==3671==   total heap usage: 47,558 allocs, 45,157 frees, 14,887,354 bytes allocated
32: ==3671==
32: ==3671== 184 (88 direct, 96 indirect) bytes in 1 blocks are definitely lost in loss record 1,863 of 1,951
32: ==3671==    at 0x483577F: malloc (vg_replace_malloc.c:299)
32: ==3671==    by 0x5204900: g_malloc (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5800.3)
32: ==3671==    by 0x521C882: g_slice_alloc (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5800.3)
32: ==3671==    by 0x51ECD3D: g_hash_table_new_full (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5800.3)
32: ==3671==    by 0x51D8CF0: g_get_language_names_with_category (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5800.3)
32: ==3671==    by 0x51F458B: ??? (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5800.3)
32: ==3671==    by 0x51F622C: g_key_file_new (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5800.3)
32: ==3671==    by 0x4EF6A1E: reload_core_config (ostree-repo.c:2835)
32: ==3671==    by 0x4EF7BD9: ostree_repo_reload_config (ostree-repo.c:3158)
32: ==3671==    by 0x4EF8188: ostree_repo_open (ostree-repo.c:3280)
32: ==3671==    by 0x4EF3424: repo_open_at_take_fd (ostree-repo.c:1243)
32: ==3671==    by 0x4EF34B7: ostree_repo_open_at (ostree-repo.c:1269)
32: ==3671==    by 0x4F30E32: ensure_repo (ostree-sysroot.c:782)
32: ==3671==    by 0x4F312F1: ostree_sysroot_load_if_changed (ostree-sysroot.c:871)
32: ==3671==    by 0x4F30DC4: ostree_sysroot_load (ostree-sysroot.c:771)
32: ==3671==    by 0x186D6C: OstreeManager::LoadSysroot(boost::filesystem::path const&) (ostreemanager.cc:338)
32: ==3671==    by 0x18B8CB: OstreeManager::pull(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, KeyManager const&, Uptane::Target const&, api::FlowControlToken const*, std::function<void (Uptane::Target const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>) (ostreemanager.cc:75)
32: ==3671==    by 0x18CCEE: OstreeManager::fetchTarget(Uptane::Target const&, Uptane::Fetcher&, KeyManager const&, std::function<void (Uptane::Target const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>, api::FlowControlToken const*) (ostreemanager.cc:239)
32: ==3671==    by 0x130473: operator() (fetcher_test.cc:83)
32: ==3671==    by 0x130473: __invoke_impl<void, test_pause(const Uptane::Target&, PackageManager)::<lambda()> > (invoke.h:60)
32: ==3671==    by 0x130473: __invoke<test_pause(const Uptane::Target&, PackageManager)::<lambda()> > (invoke.h:95)
32: ==3671==    by 0x130473: _M_invoke<0> (thread:244)
32: ==3671==    by 0x130473: operator() (thread:253)
32: ==3671==    by 0x130473: std::thread::_State_impl<std::thread::_Invoker<std::tuple<test_pause(Uptane::Target const&, PackageManager)::{lambda()#1}> > >::_M_run() (thread:196)
32: ==3671==    by 0x5572B2E: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25)
32: ==3671==    by 0x4ADEFA2: start_thread (pthread_create.c:486)
32: ==3671==    by 0x574E82E: clone (clone.S:95)

It could come from some version of glib...

@lbonn
Copy link
Contributor Author

lbonn commented Apr 12, 2019

@lbonn
Copy link
Contributor Author

lbonn commented Apr 12, 2019

This should help: advancedtelematic/aktualizr#1179

@pattivacek
Copy link
Collaborator

This should help: advancedtelematic/aktualizr#1179

Indeed. It passes when I build with that branch. If only that branch succeeded on Travis...

@lbonn
Copy link
Contributor Author

lbonn commented Apr 15, 2019

Travis finally passed :)

@pattivacek
Copy link
Collaborator

Travis finally passed :)

Great! If you bump aktualizr's version here, I'll approve and we can merge this. Finally!

Signed-off-by: Laurent Bonnans <[email protected]>
Copy link
Collaborator

@pattivacek pattivacek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passes oe-selftest except for test_secondary_listening, which seems unrelated and is going to change anyway.

@pattivacek pattivacek merged commit 8cc72a8 into master Apr 15, 2019
@pattivacek pattivacek deleted the test/oe-selftest-ptest branch April 15, 2019 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants