Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

October build run status #689

Closed
4 tasks done
petefoth opened this issue Sep 30, 2024 · 21 comments
Closed
4 tasks done

October build run status #689

petefoth opened this issue Sep 30, 2024 · 21 comments

Comments

@petefoth
Copy link
Contributor

petefoth commented Sep 30, 2024

All done, with more than an hour to spare before November build run starts :)


  • started:
    • 21.0 devices missed last month
      • started:
      • unexpected failures: renoir,rhode,river,rtwo,sabrina,sagit,sailfish,salami,sargo,shark,shiba,sif
      • completed:
    • 20.0 branch
      • started:
      • unexpected failures: vayu
      • completed:
    • Remaining 21.0 devices
      • started:
      • unexpected failures:
      • completed:
    • newly promoted devices: NB1,nx606j,nx619j
  • completed: >> [Thu Oct 31 22:48:24 UTC 2024]

Builds for the following devices did not get made in September, so they go to the top of the queue this month:

  1. Didn't get built in time: radxa0,radxa0_tab,raven,redfin,renoir,rhode,river,rtwo,sabrina,sagit,sailfish,salami,sargo,shark,shiba,sif
  2. New and promoted devices: cmi,onclite
  3. Failed builds: billie,haydn,pioneer,voyager,xz2c

1 & 2 will be first in line this month.. 3 will be done first in the list of remaining 21.0 builds. So the full target list for October will be

  • Missed & failed 21.0 builds: radxa0_tab,raven,redfin,renoir,rhode,river,rtwo,sabrina,sagit,sailfish,salami,sargo,shark,shiba,sif
  • 20.0 builds: for all 20.0 devices that are in the build target list at 202410-00:01
  • Remaining 21.0 builds: cmi,onclite,billie,haydn,pioneer,voyager,xz2c,a52q,a52sxq,a71,a72q,a73xq,akari,akatsuki,akita,alioth,aura,aurora,axolotl,barbet,beckham,berlin,berlna,beryllium,bluejay,blueline,bonito,borneo,bramble,beyond0lte,beyond1lte,beyond2lte,beyondx,caprip,cebu,channel,cheeseburger,cheetah,cheryl,chiron,coral,crosshatch,d1,d1x,d2s,d2x,davinci,deadpool,devon,dipper,discovery,dopinder,dre,dubai,dumpling,earth,enchilada,equuleus,evert,f62,fajita,felix,flame,foster,foster_tab,FP3,FP4,FP5,g710n,g710ulm,gemini,grus,gta4l,gta4lwifi,gta4xl,gta4xlwifi,gtowifi,gts4lv,gts4lvwifi,guacamole,guacamoleb,guam,guamp,hawao,heart,hotdog,hotdogb,husky,ingot,instantnoodle,instantnoodlep,joan,judyln,judyp,judypn,kane,kebab,kiev,kirin,lake,lemonade,lemonadep,lemonades,lisa,lmi,luigi,lynx,m5,m5_tab,m52xq,marlin,mars,martini,mata,mermaid,messi,Mi439,Mi8917,Mi8937,miatoll,munch,nairo,nash,natrium,nio,nx,nx_tab,ocean,odroidc4,odroidc4_tab,oriole,oscar,panther,payton,pdx203,pdx206,pdx214,pdx215,pdx234,perseus,PL2,quill,quill_pad,socrates,sunfish,surya,sweet,taimen,tangorpro,TP1803,troika,umi,ursa,vela,wade,walleye,xpeng,Z01R,zippo

Originally posted by @petefoth in #683 (comment)

@petefoth
Copy link
Contributor Author

Mon 30 Sep 24

  • Add quill` to the 'Remaining 21.0' builds as it just failed in September build run

@petefoth
Copy link
Contributor Author

petefoth commented Oct 1, 2024

Tue 01 Oct 24

  • builds up to redfin OK, then started failing with missing namespace error - killed
  • cleaned up and started with cmi,onclite,billie,haydn,pioneer,voyager,xz2c, then the 20.0 bulids

@ghost
Copy link

ghost commented Oct 1, 2024

renoir,rhode,river,rtwo,sabrina,sagit,sailfish,salami,sargo,shark,shiba,sif builds fine for official, why is it failing for lineageos for microg?

@petefoth
Copy link
Contributor Author

petefoth commented Oct 1, 2024

renoir,rhode,river,rtwo,sabrina,sagit,sailfish,salami,sargo,shark,shiba,sif builds fine for official, why is it failing for lineageos for microg?

Probably the same reason as the September 21.0 builds failed.I fixed that problem for most devices, but it seems the fix didn;t work for these devices. I will look into it further when I get some free time

@petefoth
Copy link
Contributor Author

petefoth commented Oct 3, 2024

Thu 03 Oct 24

  • builds died during build of apollon >> [Wed Oct 2 21:08:07 UTC 2024]
  • restart using build_list.sh: apollon,bardock,bardockpro,betalm,crownlte,G,gauguin,lancelot,liber,merlinx,mh2lm,nx659j,sake,star2lte,starlte,tissot,vayu,violet,X00TD,xmsirius,z2_plus,zangya,zangyapro

@petefoth
Copy link
Contributor Author

petefoth commented Oct 4, 2024

Fri 04 Oct 24

  • liber buld failed.
[ 62% 541/864] including kernel/motorola/sm6150/drivers/staging/greybus/tools/Android.mk ...
FAILED:
In file included from kernel/google/gs201/private/google-modules/wlan/wcn6740/wlan/qcacld-3.0/Android.mk:29:
In file included from kernel/motorola/sm6150/drivers/staging/greybus/tools/Android.mk:9:
In file included from build/make/core/executable.mk:53:
In file included from build/make/core/executable_internal.mk:29:
In file included from build/make/core/dynamic_binary.mk:34:
In file included from build/make/core/binary.mk:9:
build/make/core/base_rules.mk:338: error: kernel/motorola/sm6150/drivers/staging/greybus/tools: MODULE`
  • Alll subsequent builds failed merlinx,mh2lm,nx659j,sake,star2lte,starlte,tissot,vayu,violet,X00TD,xmsirius,z2_plus,zangya,zangyapro
  • last successful build was lancelot
  • It looks like the same problem as with the 21.0 builds, but with kernel rather than vendor files.

@petefoth
Copy link
Contributor Author

petefoth commented Oct 5, 2024

Sat 05 Oct 24

  • vayu failed - retry later
FAILED: out/target/product/vayu/obj/vendor/qcom/opensource/audio-kernel/include/linux/avtimer.h
/bin/bash -c "\$(cd out/target/product/vayu/obj/KERNEL_OBJ; ../../../../../../../kernel/msm-4.14/scripts/headers_install.sh ../../../../../../../out/target/product/vayu/obj/vendor/qcom/opensource/audio-kernel/include/linux/ ../../../../../../../kernel/motorola/sm6150/techpack/audio/include/uapi/linux/ avtimer.h)"
/bin/bash: line 1: cd: out/target/product/vayu/obj/KERNEL_OBJ: No such file or directory
/bin/bash: line 1: ../../../../../../../kernel/msm-4.14/scripts/headers_install.sh: No such file or directory

@petefoth
Copy link
Contributor Author

petefoth commented Oct 6, 2024

Sun 06 Oct 24

  • 20.0 builds completed
  • 21.0 Started
  • a71 failed - retry later
Error message
[ 99% 186690/186866] build out/target/product/a71/obj/PACKAGING/check_vintf_all_intermediates/check_vintf_compatible.log (priority: 4)
FAILED:
...

List 'out/target/product/a71/product/etc/vintf/': NAME_NOT_FOUND
The following HALs in device manifest are not declared in FCM <= level 5:
  android.hardware.biometrics.fingerprint.IFingerprint/default (@1)
  android.hardware.bluetooth.audio.IBluetoothAudioProviderFactory/default (@4)
  android.hardware.camera.provider.ICameraProvider/internal/0 (@1)
  android.hardware.cas.IMediaCasService/default (@1)
  android.hardware.drm.IDrmFactory/clearkey (@1)
  android.hardware.health.IHealth/default (@1)
  android.hardware.sensors.ISensors/default (@1)
  android.hardware.usb.IUsb/default (@1)
  [email protected]::IUsb/default
  android.hardware.wifi.IWifi/default (@2)
  android.hardware.wifi.hostapd.IHostapd/default (@2)
  android.hardware.wifi.supplicant.ISupplicant/default (@3)
  vendor.samsung.hardware.keymint.ISehKeyMintExtension/default (@1)
  vendor.samsung.hardware.keymint.ISehKeyMintFactory/default (@1)
ERROR: files are incompatible: The following instances are in the device manifest but not specified in framework compatibility matrix:
    vendor.samsung.hardware.keymint.ISehKeyMintExtension/default (@1)
    vendor.samsung.hardware.keymint.ISehKeyMintFactory/default (@1)
Suggested fix:
1. Update deprecated HALs to the latest version.
2. Check for any typos in device manifest or framework compatibility matrices with FCM version >= 5.
3. For new platform HALs, add them to any framework compatibility matrix with FCM version >= 5 where applicable.
4. For device-specific HALs, add to DEVICE_FRAMEWORK_COMPATIBILITY_MATRIX_FILE or DEVICE_PRODUCT_COMPATIBILITY_MATRIX_FILE.: Success
INCOMPATIBLE

@petefoth
Copy link
Contributor Author

Fri 11 Oct 24

  • coral and all subsequent builds. It's because vendor is not being cleaned up
``` [100% 1167/1167] analyzing Android.bp files and generating ninja file at out/soong/build.lineage_coral.ninja FAILED: out/soong/build.lineage_coral.ninja cd "$(dirname "out/host/linux-x86/bin/soong_build")" && BUILDER="$PWD/$(basename "out/host/linux-x86/bin/soong_build")" && cd / && env -i "$BUILDER" --top "$TOP" --soong_out "out/soong" --out "out" --soong_variables out/soong/soong.lineage_coral.variables -o out/soong/build.lineage_coral.ninja --globListDir lineage_coral --globFile out/soong/globs-lineage_coral.ninja -l out/.module_paths/Android.bp.list --available_env out/soong/soong.environment.available --used_env out/soong/soong.environment.used.lineage_coral.build Android.bp error: vendor/sony/sm8250-common/Andqroid.bp:14767:1: "[email protected]" depends on undefined module "[email protected]". Or did you mean ["[email protected]" "[email protected]" "[email protected]" "[email protected]_hal" "[email protected]_hal" "[email protected]_hal" "[email protected]_hal" "[email protected]_hal" "vendor.qti.hardware.qspa-V1-java" "vendor.qti.hardware.qspa-V1-rust" "vendor.qti.hardware.qspa-V2-java" "vendor.qti.hardware.qspa-V2-rust" "vendor.qti.hardware.radio.am.xml" "[email protected]" "vendor.qti.hardware.systemhelper" "[email protected]" "[email protected]_hal" "[email protected]"]? ```
  • put in some debug info
  • restart builds from coral

@petefoth
Copy link
Contributor Author

petefoth commented Oct 15, 2024

Tue 15 Oct 24

  • Several builds, starting with grus have failed. All the failures so far have been missing namespace errors
    and the missing namespaces have all been in vendor/xiaomi repos.
  • the problem is that changes were made in the last couple of hours in TheMuppest git for sweet, davinci, & sm6150-common repos, and matching groups fields added for those repos in the manifest rpo. But we are using a proprietary.xml that doesn't have those changes, because it is generated once, outside the build loop. Should be enough to just restart the build (from grus). Longer term we could generate the proprietary.xml inside the main loop

Wed 16 Oct 24

  • restarting the build seems to have fixed it

Thu 17 Oct 24

  • hawao failed, same problem: upstream changes to manifest,. Kill & restart from hawao
  • moved the code that generates proprietary.xml inside the main loop, so it is done for each device, minimising (with luck) the chances of this happening again.

@hshl2
Copy link

hshl2 commented Oct 21, 2024

Is this the same issue here?

docker-lineage-cicd-1  | Set cache size limit to 50.0 GB
docker-lineage-cicd-1  | >> [Mon Oct 21 04:57:26 UTC 2024] Branch:  lineage-21.0
docker-lineage-cicd-1  | >> [Mon Oct 21 04:57:26 UTC 2024] Devices: lemonadep
docker-lineage-cicd-1  | >> [Mon Oct 21 04:57:26 UTC 2024] Copying '/srv/local_manifests/*.xml' to '.repo/local_manifests/'
docker-lineage-cicd-1  | >> [Mon Oct 21 04:57:26 UTC 2024] (Re)initializing branch repository
docker-lineage-cicd-1  | sed: can't read vendor/lineage/config/common.mk: No such file or directory
docker-lineage-cicd-1 exited with code 2

@petefoth
Copy link
Contributor Author

docker-lineage-cicd-1 | >> [Mon Oct 21 04:57:26 UTC 2024] (Re)initializing branch repository
docker-lineage-cicd-1 | sed: can't read vendor/lineage/config/common.mk: No such file or directory
docker-lineage-cicd-1 exited with code 2

That's not the error that we were seeing in the build failures. And it's not one I recall seeing in any of our earlier build failures. So it looks like you are having a different problem. I don't want to clutter this issue, so if you are seeing this problem repeatedly and consistently it may be worth creating a new issue where it ca be dealt with.

From that error message,if file vendor/lineage/config/common.mk does not exist in your source tree, that tree is in a very strange state. Try removing ``vendor/lineage/orcd vendor/lineage && git reset --hard` then re-running your build.

If that doesn't work, please open a new issue, with as much detail as possible. Thanks

@ghost
Copy link

ghost commented Oct 26, 2024

Builds skipped quill,quill_pad and radxa0,radxa0_tab,raven,redfin,renoir,rhode,river,rtwo,sabrina,sagit,sailfish,salami,sargo,shark,shiba,sif and straight to socrates did something break?

@Carlos-Rocha
Copy link

Carlos-Rocha commented Oct 27, 2024

Those were the first. Please see first post

@ghost
Copy link

ghost commented Oct 27, 2024

Those were the first. Please see first post

#693

renoir,rhode,river,rtwo,sabrina,sagit,sailfish,sargo,shark,shiba,sif last build 08/28/2024
quill,quill_pad last build 07/21/2024
salami last build 06/14/2024

You can easily check yourself https://download.lineage.microg.org/

@Carlos-Rocha
Copy link

The build has failed for quill and quill_pad, not sure about salami, but radxa0 was the first to be built
[https://download.lineage.microg.org/radxa0/lineage-21.0-20241001-microG-radxa0.zip]

@ghost
Copy link

ghost commented Oct 27, 2024

radxa0 was the first to be built

radxa0,radxa0_tab,raven,redfin did get a build at the beginning of October, my error.

renoir,rhode,river,rtwo,sabrina,sagit,sailfish,salami,sargo,shark,shiba,sif was skipped

salami had it the worst since it hasn't had a new build since June, official has restarted at the end of August, the others at least got a build in August.

@petefoth
Copy link
Contributor Author

Builds skipped quill,quill_pad and ``

They weren't skipped, they failed, for the same reason as last month. A change upstream (in the official LineageOS git repos for these devices) will probably be needed.

renoir,rhode,river,rtwo,sabrina,sagit,sailfish,salami,sargo,shark,shiba,sif was skipped

They weren't skipped, they failed, because cleanup didn't happen correctly after the failed builds. That issue has - I hope - now been fixed

salami had it the worst since it hasn't had a new build since June, official has restarted at the end of August, the others at least got a build in August.

That's unfortunate - sorry.

The current build run will probably finish on Tuesday 29th. This leaves just over 2 days before the next build run starts. That should be enough time for builds of these devices - salami will be at the first in the list :)

Sorry again.

@petefoth
Copy link
Contributor Author

petefoth commented Oct 28, 2024

renoir,rhode,river,rtwo,sabrina,sagit,sailfish,salami,sargo,shark,shiba,sif was skipped

salami had it the worst since it hasn't had a new build since June, official has restarted at the end of August, the others at least got a build in August.

Builds are now available for salami and renoir. A build for rhode should be available in the next couple of hours. See here for details: #693 (comment)

Feedback welcome in #693

@petefoth
Copy link
Contributor Author

Tue 29 Oct 24 17:05:08 UTC

  • 21.0 builds for river,sabrina,sagit,sailfish,sargo,shark,shiba,sif started. Should be finished before next build run starts :)

@petefoth
Copy link
Contributor Author

petefoth commented Oct 31, 2024

Thu 31 Oct 24

  • redo failed build for vayu &
  • build promoted devices nx606j,nx619j

Sorry, something went wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants