Automate build of iOS signed ipa file for App Store submission #2625

danryu · 2022-05-09T17:53:32Z

This PR adds automation to create a signed ipa (installer) file for direct submission to the iOS App Store.

CHANGELOG: adds iOS signed pkg build automation

Context: automates building of signed pkg file for iOS App Store

Does this change need documentation? What needs to be documented and how?

Required:

In Apple Developer Account, create the following resources in in https://developer.apple.com/account/resources/certificates/list

Certificates:
- iOS Distribution
Identifier:
- app ID (bundleID)
Profile:
- iOS Distribution (AppStore) - ProvisioningProfile

Add the certs and provisioning profile to Github Secrets as per https://docs.github.com/en/actions/deployment/deploying-xcode-applications/installing-an-apple-certificate-on-macos-runners-for-xcode-development

Status of this Pull Request

What is missing until this pull request can be merged?

Checklist

I've verified that this Pull Request follows the general code principles
I tested my code and it does what I want
My code follows the style guide
I waited some time after this Pull Request was opened and all GitHub checks completed without errors.
I've filled all the content above

ann0see

Interesting! To me it seems as if this still needs manual work. In an ideal workflow a release push would upload the app to the app store automatically and just wait on someone to press a "publish" button there.

ann0see · 2022-05-09T19:47:51Z

.github/autobuild/ios.sh

+    security find-identity -v 
+
+    # Tell Github Workflow that we need notarization & stapling:
+    echo "::set-output name=macos_signed::true"


should be renamed to apple_signed if we merge this.

or maybe ios_signed

@ann0see it is actually possible to upload the binary to the App Store, using "altool" as outlined here: https://help.apple.com/app-store-connect/#/devb1c185036

However, as the App Store submission process involves a number of manual setup steps anyway (ie creating the actual submission on the App Store Connect website, adding screenshots, descriptions, compliance details and all sorts) and you likely only want to upload specific versions to the App Store (rather than having it run automatically), I thought the generation of the signed, verifiable IPA is a big enough win for this automation.

Quick note: the only manual step remaining once you have the signed IPA is to drag and drop the IPA file into Transporter (on macOS), and click the Upload button.

If I have time today though I will experiment with adding the upload directly using xcrun altool.

ann0see · 2022-05-09T19:48:38Z

.github/autobuild/ios.sh

 }

 pass_artifact_to_job() {
-    local artifact="jamulus_${JAMULUS_BUILD_VERSION}_iOSUnsigned${ARTIFACT_SUFFIX:-}.ipa"
+    local artifact="jamulus_${JAMULUS_BUILD_VERSION}_iOS_unsigned${ARTIFACT_SUFFIX:-}.ipa"


Not sure how much a signed IPA on the web without the app store helps?

I don't think there's any point to that, no - the signed IPA is purely for App Store submission. So it's really for @emlynmac or whoever is managing the Jamulus App Store account to have a ready-built file for submission.

I assume the unsigned IPA is useful for general testing purposes so obviously I've kept that in.

emlynmac

Looks like this PR needs a bit more work to make it happy with the case of not having certificates around.

emlynmac · 2022-05-09T22:04:32Z

.github/autobuild/ios.sh

+    security find-identity -v 
+
+    # Tell Github Workflow that we need notarization & stapling:
+    echo "::set-output name=macos_signed::true"


or maybe ios_signed

emlynmac · 2022-05-09T22:06:43Z

.github/autobuild/ios.sh

    echo "Moving build artifact to deploy/${artifact}"
-    mv ./deploy/Jamulus.ipa "./deploy/${artifact}"
+    mv ./deploy/Jamulus_unsigned.ipa "./deploy/${artifact}"


Probably not worth keeping this around, only signed version would be useful past making sure the build works.

Ok, I left a note above, I wasn't sure if people were maybe using the unsigned IPA for local testing or somesuch.
No problem to remove it if you prefer.

Should not be removed IMO.

emlynmac · 2022-05-09T22:07:47Z

.github/workflows/autobuild.yml

-            building_on_os:         macos-10.15
-            base_command:           QT_VERSION=5.15.2 ./.github/autobuild/ios.sh
+            building_on_os:         macos-11
+            base_command:           QT_VERSION=5.15.2 SIGN_IF_POSSIBLE=1 ./.github/autobuild/ios.sh


Could you use the presence of a certificate in the environment to determine whether to sign, so not needing another parameter?

I would rather follow the lead as being done for macOS, which uses that flag. Or we can remove from both macOS and iOS but to have in just one feels weird.

ann0see · 2022-05-15T20:06:42Z

@danryu The CI failed. We currently don't have a valid cert on this repo.

danryu · 2022-05-16T07:29:12Z

@danryu The CI failed. We currently don't have a valid cert on this repo.

Ok I've pushed a fix now to check for the cert non-zero value before attempting to move the signed IPA for download. This should resolve this CI failure where there is no valid cert in place.

Please note also that if signing is requested and dependencies are satisfied, the script will now attempt to validate and upload to App Store using the existing NOTARIZATION_USER and NOTARIZATION_PASSWORD. We may need to fine-tune this behaviour if eg re-uploading the same version (eg 3.9.0) to the App Store repeatedly (I think it will fail rather than updating the existing similar version). Let me know what you think should happen in this case.

ann0see · 2022-05-16T07:37:39Z

Hmm. I think the upload should only happen if we have a release build (there's a GitHub variable for that, I think). Look for the variable which enables/disabled uploading to GitHub releases.

danryu · 2022-05-16T08:05:17Z

Ok, the Github release stuff switches on needs.create_release.outputs.publish_to_release == 'true' - so maybe I should add a separate extra step in autobuild.yml for iOS App Store upload - after the Notarization and Staple steps perhaps?
Then I can split out the validate and upload stages from the main build run, which seems sensible.

ann0see · 2022-05-16T19:18:26Z

Yes. I think so.

danryu · 2022-05-19T16:10:33Z

I'm working on adding the extra step now.

Also: I've noticed that the iOS App Store upload validation (to appear in TestFlight in App Connect) requires an images assets catalog in a directory entitled Images.xcassets - I have this working in branch CI as a subdir of ios. However the asset catalog is a collection of ~26 files, normally generated by XCode or another utility. I'm thinking of integrating a tool like https://github.com/alfianlosari/AppIconGenCLI, which would then mean we just have to include a single 1024x1024 png file to generate all the others, and saving us from putting all 26+ files in source control.

ann0see · 2022-06-08T17:41:21Z

I think both approaches have advantages and disadvantages (another CI job vs more data in the repo). Any other comments on this @ngocdh @emlynmac

ann0see · 2022-07-24T11:20:40Z

@danryu any updates here?

danryu · 2022-07-25T06:07:29Z

Hi @ann0see yes, I have Store submission working as part of the release procedure (also for macOS for the other PR).
I just need to confirm what the desired behavior is before updating the PR. Do we still want to keep the unsigned IPA if the signed one is generated?

ann0see · 2022-07-25T09:24:00Z

Can the signed ipa be re-signed for debugging? If yes, no need to have the unsigned ipa. Else just provide the unsigned one and upload the signed one to Apple.

danryu · 2022-07-26T09:33:56Z

I'm actually not 100% sure on that point, so I will keep the unsigned IPA for now, to not break the existing pattern.

danryu · 2022-10-11T13:20:56Z

I am re-assessing this PR in the light of errors that I get when running apps in TestFlight, that have been built and submitted by this automation. The app builds fine and is validated and accepted by TestFlight - but when the app actually starts it quits immediately with a SIG11 error.
This does NOT happen when the same build and submission is carried out locally on macOS/XCode.
I will update this PR when I have worked out the reason for the CI failure.

ann0see · 2022-10-11T20:19:01Z

Strange. Do you use Qt5?

danryu · 2022-10-12T05:05:08Z

Strange. Do you use Qt5?

No, Qt 6.3.2 and actually 6.4.0 for iOS (for some touch-handling fixes).

It is indeed very strange - after all, if TestFlight validates the package, one would expect it to run! I suspect it may be related to something like the handling of Bitcode in the build, or something equally obscure.

Incidentally, the result is the same when I use a dedicated Github Action for iOS building - yukiarrr/[email protected] - which I think might be the simplest way of doing iOS builds going forward.

ann0see · 2022-10-12T05:59:06Z

Yes, the iOS build crashes with Qt6 for me too (at least on GH actions). It's a known issue with Qt6

#2711

danryu · 2022-10-12T07:06:42Z

Yes, the iOS build crashes with Qt6 for me too (at least on GH actions). It's a known issue with Qt6

#2711

Hmmm, I think I may have fixed that particular issue by including this in .pro file for iOS:
QMAKE_LFLAGS += -Wl,-e,_qt_main_wrapper

Though I don't have the reference for the fix to hand.
With this in place, I've been building and running the iOS app for several months, with Qt 6.3.2 and now 6.4.0.
However, within the last 4-6 weeks (roughly), TestFlight has been validating and accepting the CI iOS builds, only to have them crash with SIG11 on startup.
I spent considerable time building from historical commits, exactly replicating previous configurations - but couldn't shake the error.
To me it seems like an opaque change in how TestFlight is handling the builds in each case - very frustrating.
The workaround currently for me is to do the local XCode build as mentioned.

Please check and try the Koord app in the iOS App Store for proof that Qt6 works fine with iOS :)

ann0see · 2022-10-12T14:02:12Z

Koord app

Great. That's a modified Jamulus client I suppose? If you plan to maintain your app and keep it compatible and up to date, we could link it as Jamulus compatible 3rd party App here in the KB: https://jamulus.io/kb/2022/03/04/Alternative-Install-Methods.html

danryu · 2022-10-14T09:53:30Z

Great. That's a modified Jamulus client I suppose? If you plan to maintain your app and keep it compatible and up to date, we could link it as Jamulus compatible 3rd party App here in the KB: https://jamulus.io/kb/2022/03/04/Alternative-Install-Methods.html

That would be great! We definitely intend on maintaining it :) and also keeping upstream compatibility!
https://github.com/koord-live/koord-app
We are a few months behind upstream master, and that will be merged in shortly.

There are a few points of interest in the app - but I might open a quick Discussion for that!

ann0see · 2023-05-21T20:11:48Z

Adding this stackoverflow post as reference for the new flags: https://stackoverflow.com/questions/45508043/qt-ios-linker-error-entry-point-main-undefined

ann0see · 2023-07-01T19:59:39Z

Needs a rebase. Not closing yet, but might in future...

ann0see · 2024-07-23T20:34:13Z

@danryu I'd suggest you add a very brief note to compatibility at https://github.com/jamulussoftware/jamuluswebsite/edit/release/_posts/2022-03-04-Alternative-Install-Methods.md without any big links to Koord. It can and should not be an ad.

danryu · 2024-07-28T17:53:44Z

@danryu I'd suggest you add a very brief note to compatibility at https://github.com/jamulussoftware/jamuluswebsite/edit/release/_posts/2022-03-04-Alternative-Install-Methods.md without any big links to Koord. It can and should not be an ad.

Sorry I don't understand this request. Why would I mention Koord, and why is a specific note required?
This PR adds all the logic to be able to submit a working version of Jamulus to the App Store. If there's no interest in this, I'm happy to close the PR.

ann0see · 2024-07-28T18:15:12Z

No. There is interest in this PR. This was only outstanding from above (#2625 (comment)).

ann0see · 2024-07-28T18:16:21Z

.github/autobuild/ios.sh

+    # add notarization/validation/upload password to keychain
+    xcrun altool --store-password-in-keychain-item --keychain build.keychain APPCONNAUTH -u $NOTARIZATION_USER -p $NOTARIZATION_PASSWORD
+    # set lock timeout on keychain to 6 hours
+    security set-keychain-settings -lut 21600


As in the macOS case - maybe this is no longer needed.

ann0see · 2024-07-28T18:16:48Z

.github/autobuild/ios.sh

+    # Signing was requested, now check all prerequisites:
+    [[ -n "${IOSDIST_CERTIFICATE:-}" ]] || return 1
+    [[ -n "${IOSDIST_CERTIFICATE_ID:-}" ]] || return 1
+    [[ -n "${IOSDIST_CERTIFICATE_PWD:-}" ]] || return 1
+    [[ -n "${NOTARIZATION_PASSWORD:-}" ]] || return 1
+    [[ -n "${IOS_PROV_PROFILE_B64:-}" ]] || return 1
+    [[ -n "${KEYCHAIN_PASSWORD:-}" ]] || return 1
+
+    echo "Signing was requested and all dependencies are satisfied"
+
+    # use this as filename for Provisioning Profile
+    IOS_PP_PATH="embedded.mobileprovision"
+
+    ## Put the cert to a file
+    # IOSDIST_CERTIFICATE - iOS Distribution
+    echo "${IOSDIST_CERTIFICATE}" | base64 --decode > iosdist_certificate.p12
+
+    ## Echo Provisioning Profile to file
+    echo -n "${IOS_PROV_PROFILE_B64}" | base64 --decode > $IOS_PP_PATH


As in the macOS case, probably worth putting distribution related stuff in an if.

danryu added 2 commits May 9, 2022 19:45

add autobuild for iOS signed ipa for App Store

2e5acc4

sanitize

5da2236

ann0see added the feature request Feature request label May 9, 2022

ann0see marked this pull request as draft May 9, 2022 19:39

ann0see reviewed May 9, 2022

View reviewed changes

emlynmac reviewed May 9, 2022

View reviewed changes

danryu added 3 commits May 10, 2022 14:29

add draft validate/upload workflow

57e7f6e

add draft validate/upload workflow

2806482

check for 2nd artifact

1ad1d31

check for cert before moving signed ipa

731e46c

danryu mentioned this pull request May 16, 2022

Automate signed pkg build for macOS App Store submission #2624

Merged

5 tasks

hoffie mentioned this pull request Oct 12, 2022

iOS: Investigate Qt6 crash & update afterwards #2711

Closed

danryu changed the title ~~Draft: automate build of iOS signed ipa file for App Store submission~~ Automate build of iOS signed ipa file for App Store submission Oct 17, 2022

ann0see changed the base branch from master to main December 26, 2022 19:08

Merge branch 'main' into ios_sign

1f3fe64

pljones added tooling Changes to the automated build system and removed feature request Feature request labels Aug 12, 2023

pljones assigned danryu May 6, 2024

ann0see reviewed Jul 28, 2024

View reviewed changes

ann0see mentioned this pull request Oct 12, 2024

Add iOS Icon #3410

Merged

4 tasks

Automate build of iOS signed ipa file for App Store submission #2625

Are you sure you want to change the base?

Automate build of iOS signed ipa file for App Store submission #2625

Conversation

danryu commented May 9, 2022 • edited Loading

Required:

Checklist

ann0see left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emlynmac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ann0see commented May 15, 2022

danryu commented May 16, 2022 • edited Loading

ann0see commented May 16, 2022

danryu commented May 16, 2022 • edited Loading

ann0see commented May 16, 2022

danryu commented May 19, 2022

ann0see commented Jun 8, 2022

ann0see commented Jul 24, 2022

danryu commented Jul 25, 2022

ann0see commented Jul 25, 2022 • edited Loading

danryu commented Jul 26, 2022

danryu commented Oct 11, 2022

ann0see commented Oct 11, 2022

danryu commented Oct 12, 2022 • edited Loading

ann0see commented Oct 12, 2022 • edited Loading

danryu commented Oct 12, 2022

ann0see commented Oct 12, 2022

danryu commented Oct 14, 2022 • edited Loading

ann0see commented May 21, 2023

ann0see commented Jul 1, 2023 • edited Loading

ann0see commented Jul 23, 2024

danryu commented Jul 28, 2024

ann0see commented Jul 28, 2024

Choose a reason for hiding this comment

ann0see Jul 28, 2024 • edited Loading

Choose a reason for hiding this comment

danryu commented May 9, 2022 •

edited

Loading

danryu commented May 16, 2022 •

edited

Loading

danryu commented May 16, 2022 •

edited

Loading

ann0see commented Jul 25, 2022 •

edited

Loading

danryu commented Oct 12, 2022 •

edited

Loading

ann0see commented Oct 12, 2022 •

edited

Loading

danryu commented Oct 14, 2022 •

edited

Loading

ann0see commented Jul 1, 2023 •

edited

Loading

ann0see Jul 28, 2024 •

edited

Loading