Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failures on openqa.opensuse.org #9

Open
8 of 9 tasks
dcermak opened this issue Jul 1, 2021 · 9 comments
Open
8 of 9 tasks

Test failures on openqa.opensuse.org #9

dcermak opened this issue Jul 1, 2021 · 9 comments

Comments

@dcermak
Copy link
Collaborator

dcermak commented Jul 1, 2021 •
edited
Loading

All items on that list (except the busted LUKS image) are vmdk images, which leads me to believe that something is wrong with vmdk on o3, whereas it works on my box. @okurz @foursixnine @asmorodskyi any ideas?
@okurz
Copy link

okurz commented Nov 22, 2021

Following https://openqa.opensuse.org/tests/2044724#settings I can see the setting for HDD_1 being kiwi-test-image-disk-simple.x86_64-1.42.1-Build11.19.vmdk but I can not find that asset over https://openqa.opensuse.org/admin/assets so it was either never there or (likely) was pruned already. https://openqa.opensuse.org/admin/job_templates/85 so far is configured for only the default 5GB and 4.8GB are used so likely any other image does not fit meaning other images will need to be deleted. Also the history https://openqa.opensuse.org/tests/2044724#next_previous shows that jobs did work. So I have now assigned 30GB to the group so I suggest you try again.

@dcermak
Copy link
Collaborator Author

dcermak commented Nov 25, 2021

I have retriggered all the tests yesterday but sadly all the tests involving the vmdk images are still failing. Here is the full list of tests:

It really looks like openQA is not downloading these images properly. When I look at https://openqa.opensuse.org/admin/assets then that page only lists the vmdk images for the Leap and none for Tumbleweed (the vmdk images are also missing under the "Logs & Assets" tab there).

@okurz Do you know if the openQA logs could be in some way helpful here?

@asmorodskyi
Copy link

if you open os-autoinst.log you will see such lines in very beginning :

[2021-11-25T01:11:38.102292+01:00] [info] +++ setup notes +++
[2021-11-25T01:11:38.102627+01:00] [info] Running on openqaworker4:7 (Linux 5.3.18-59.34-default #1 SMP Thu Nov 11 12:18:45 UTC 2021 (a2a53aa) x86_64)
[2021-11-25T01:11:38.108770+01:00] [debug] Found HDD_1, caching kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk
[2021-11-25T01:11:38.112266+01:00] [info] Downloading kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk, request #480464 sent to Cache Service
[2021-11-25T01:12:03.561175+01:00] [info] Download of kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk processed:
[info] [#480464]
Cache size of "/var/lib/openqa/cache" is 48 GiB, with limit 50 GiB
[info] [#480464]
Downloading "kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk" from "http://openqa1-opensuse/tests/2055611/asset/hdd/kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk"
[info] [#480464]
Size of "/var/lib/openqa/cache/openqa1-opensuse/kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk" is 954 MiB, with ETag ""3b9d0000-5d18c5fe40064""
[info] [#480464]
Download of "/var/lib/openqa/cache/openqa1-opensuse/kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk" successful, new cache size is 49 GiB

so asset is downloaded BUT it is vmdk not qcow and here I pass mike to @okurz because I have no idea if openQA support VMWare images ( spoiler : most probably not )

@dcermak
Copy link
Collaborator Author

dcermak commented Nov 25, 2021

I have quickly taken a peek at /var/log/factory/hdd from ariel and the assets are not there:

dancermak@ariel:~> ll /var/lib/openqa/factory/hdd/|grep vmdk
-rw-r--r-- 1 geekotest nogroup   816513024 Nov 24 17:43 kiwi-test-image-disk-simple.x86_64-1.15.3-Build11.10.vmdk
-rw-r--r-- 1 geekotest nogroup   844431360 Nov 24 17:43 kiwi-test-image-lvm.x86_64-1.15.3-Build11.10.vmdk
-rw-r--r-- 1 geekotest nogroup   249823232 Nov 24 17:43 kiwi-test-image-overlayroot.x86_64-1.15.3-Build11.10.vmdk

But some assets are there and the corresponding tests still failed… (e.g. https://openqa.opensuse.org/tests/2055673 uses kiwi-test-image-overlayroot.x86_64-1.15.3-Build11.10.vmdk)

@okurz
Copy link

okurz commented Nov 25, 2021

Right now the cases look quite curious. https://openqa.opensuse.org/tests/2055674#step/boot/3 looks like the asset is correctly downloaded and found but looks like some qemu devices can not be correctly read. https://openqa.opensuse.org/tests/2055607/#step/boot/1 says that the disk image can not be read at all. Also https://openqa.opensuse.org/tests/2055607/#downloads does not list it although it was downloaded.

Regarding the "fatal: Invalid revision range" in https://openqa.opensuse.org/tests/2055607/#investigation I reported https://progress.opensuse.org/issues/102942 but that should not concern us for now.

For the other cases, if you can clearly reproduce a case that asssets are downloaded but then tests fail and the asset is gone then please report a ticket for openQA so that we can further look into this.

@asmorodskyi
Copy link

Right now the cases look quite curious. https://openqa.opensuse.org/tests/2055674#step/boot/3 looks like the asset is correctly downloaded and found but looks like some qemu devices can not be correctly read. https://openqa.opensuse.org/tests/2055607/#step/boot/1 says that the disk image can not be read at all. Also https://openqa.opensuse.org/tests/2055607/#downloads does not list it although it was downloaded.

Regarding the "fatal: Invalid revision range" in https://openqa.opensuse.org/tests/2055607/#investigation I reported https://progress.opensuse.org/issues/102942 but that should not concern us for now.

For the other cases, if you can clearly reproduce a case that asssets are downloaded but then tests fail and the asset is gone then please report a ticket for openQA so that we can further look into this.

@okurz something to take into consideration . @dcermak stating that same thing is working on his local instance . I did some comparison of log files from his local instance and one thing which I noticed that he has os-autoinst version=23 which saying about potential regression

@dcermak
Copy link
Collaborator Author

dcermak commented Nov 29, 2021

Reported as https://progress.opensuse.org/issues/103203

@okurz
Copy link

okurz commented Nov 29, 2021

@asmorodskyi good hint. @dcermak thanks for the ticket. Asked for more information (if possible) in the ticket.

@dcermak
Copy link
Collaborator Author

dcermak commented Jan 14, 2022

The vmdk disk failures have been fixed via os-autoinst/os-autoinst#1907.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@okurz @asmorodskyi @dcermak