Provide automatic macOS releases with Apple Silicon support #14839

vit9696 · 2021-09-12T23:02:46Z

This change implements FAT (arm64 + x86_64) binary generation for macOS with custom ./b.sh commands, allowing the resulting PPSSPPSDL.app to run on both Intel and Apple Silicon macOS instances. The resulting binaries:

will be uploaded to GitHub artifacts as usual on per-commit basis.
will cause a GitHub release creation if a new tag is pushed.

I committed the changeset from #13771 to include an M1-compatible MoltenVK implementation without any changes. The SDL2.framework comes from the official releases unmodified.

A release example will look as follows. I believe one should link to this location on ppsspp.org to make macOS builds clearly accessible.

closes #11799
closes #13708
closes #14815

closes #11799 closes #13708 closes #14815

hrydgard · 2021-09-12T23:06:29Z

Sweet!

But, regarding checking in all the SDL headers... Little reluctant to just have them in the repo like that, rather than a submodule or something. Hm.

vit9696 · 2021-09-12T23:21:55Z

Well, this is literally just an extraction of SDL2-2.0.16.dmg. I believe I can make the CI download and extract it, but I am not sure whether it is worth the effort. Doing this will:

bring manual downloading requirements;
add dmg extraction code, which is a bit convoluted as dmg needs mounting;
complicate b.sh interface, which would require specifying SDL.framework path;
complicate macbundle.sh, which would need to bridge with b.sh.

It would also not help updating SDL as there is no "latest" link from what I can tell. Furthermore, the users get a much easier manual compilation the way I did it as they no longer need brew and have no reason to think about SDL installation separately.

The current solution is not great, but the amount of downsides appearing from doing it the other way around makes me feel we are best to preserve what we have at least for the nearest future.

Maybe we can just put this directory to a hrydgard/ppsspp-sdl submodule? This sounds like a fair solution to me.

unknownbrackets

If there's a way for us to at least use a submodule for the .h files, even if a script has to copy them in the right place, that might make it easier to update. Agreed it's not great to add all the SDL headers. This is just so a user compiling it wouldn't need to download SDL headers on their own?

Can we just use something like https://stackoverflow.com/a/4133639?

I'm not sure if my experience is unique, but I've never really found it a particular challenge to install development libraries on macOS. It's a pain on Windows mainly, because everyone has :opinions: on where they ought to go and no one can manage to have the same opinions or opinions that are generally compatible with where Windows normally puts files.

But on macOS, it's all pretty straight forward - or so I thought? Brew does that weird thing with Cellar or whatever that reminds me of CVS's Attic, but it generally uses /usr/local/ just like packages or anyone else.

-[Unknown]

unknownbrackets · 2021-09-13T00:55:46Z

.github/workflows/build.yml

+      if: startsWith(github.ref, 'refs/tags/') && runner.os == 'macOS' && matrix.extra == 'test'
+      with:
+        files: ppsspp/*.zip
+        body: >


I'm not really a fan of creating releases with generic messages for each release. That's exactly the terrible practice Android apps have started using, and I hate it there too.

-[Unknown]

Well, it feels natural to me to edit the release message afterwards with e.g. a changelog manually. This code is just a convenience message to have something before this happens. There is no restriction to the amount of edits, so I feel there is no big issue in leaving it as is, especially since there is no good place to source the release message automatically anyway. Correct me if you think I am wrong.

Well, I don't think anyone will edit them, but okay. Honestly, I'd prefer the release notes were just the version again like now:
https://github.com/hrydgard/ppsspp/releases

Rather than a message that repeats for every single release. Those are not "release notes", those are a description (this repo already has one of those.)

-[Unknown]

vit9696 · 2021-09-13T05:26:49Z

Hmm. Just to make it clear, on macOS libraries are often distributed as frameworks. A framework is essentially a directory with a shared library, however, it may not be just the shared library. It is common sense to additionally have resource files and interface headers right inside the framework, and this is exactly how SDL is distributed.

Following the suggestion I will just replace SDL/macOS with e.g. ext/SDL submodule, that would contain the macOS directory in its root and all the contents currently pushed here relative to it. Sounds good?

vit9696 · 2021-09-13T05:38:24Z

SDL/macbundle.sh


 GIT_VERSION_LINE=$(grep "PPSSPP_GIT_VERSION = " "$(dirname "${0}")/../git-version.cpp")
+echo "Setting version to ${GIT_VERSION_LINE}..."


I am worried about the version string in the builds. Currently there seem to be 2 bugs:

The generated version string only has the commit hash and not the version number (1.2.3). See this print from the log.

How can I make sure there is no commit suffix in the tagged releases? I.e. I have the 1.2.3 version name alone if it is a tagged build.

I believe fixed (1) in https://github.com/vit9696/ppsspp/commit/ed046aebf5608869fd49fb55204715b01bc5620b. I am not sure what route shall I take to fix (2). Is it alright to make git-version.cpp with PPSSPP_GIT_VERSION_NO_UPDATE set to 1 in one of the CI stages for tagged commits?

I guess that's fine, as long as it has the right values. I would think you'd need to fetch tags, i.e. git fetch --tags or more ideally git fetch <remote> 'refs/tags/*:refs/tags/* to be able to have git describe fetch the right thing. That would probably fetch less and be faster.

-[Unknown]

I tried that, but it did not work for me. I believe the reason is that we not only need to fetch the tags, but also connect tags to the history, otherwise one would get ge1cb1c9a6 instead of v1.11.3-1350-ge1cb1c9a6 for intermediate releases. I changed the code to only fetch history on macOS, however, which should improve the situation quite a bit.

.github/workflows/build.yml

vit9696 · 2021-09-13T05:46:10Z

@hrydgard @unknownbrackets do you think we can schedule that for 1.12?

hrydgard · 2021-09-13T19:00:25Z

Since we're currently not making official releases for Mac, the timing is not that important.

However I do want to get some form of this in of course, this is a good start.

But can't we rely on like brew-installed SDL, or something? I guess maybe not working when we want to package for distribution?

vit9696 · 2021-09-13T23:10:46Z

brew only builds for the current architecture, and we need to support both ARM64 and x86_64. For this reason brew is not an option.

We either need to build SDL2 ourselves or use a prebuilt one. The latter is much easier as SDL project offers prebuilt binaries with optimal configuration: they are neatly crafted to target very low macOS versions.

vit9696 · 2021-09-15T18:10:33Z

@hrydgard are we decided on the SDL inclusion nuance? I personally believe the way it is now is ok at least personally, but that needs a project decision really.

hrydgard · 2021-09-15T19:00:18Z

Yeah, I understand now that we might have little choice, in order to keep the build convenient.

There's still the option of keeping it in a submodule, but on the other hand, we already have some fairly big dependencies in the repo, and even if it's in a submodule, everybody will end up syncing it down anyway.

If @unknownbrackets feels that submodule is better, we can do that, but to me, like I said, I don't see any huge pros.

Regarding the changelog thing, not related to this PR but I had to resort to a generic message on Android after a release got blocked when I mentioned the names of some fixed games in the log (plus it's limited to 500 chars which isn't enough).

unknownbrackets

Okay, if SDL needs to be updated is it just a matter of copying something from an SDL dmg? Just want to make sure if SDL requires an update it won't block a release now or something. A submodule is at least relatively straight-obvious to upgrade.

I personally don't have a macOS device I can use for PPSSPP development anymore, so a submodule also seemed a bit more accessible to me.

-[Unknown]

unknownbrackets · 2021-09-16T04:16:22Z

CMakeLists.txt

@@ -959,7 +973,9 @@ else()
 	set(PNG_REQUIRED_VERSION 1.6)
 endif()

+if(USE_SYSTEM_LIBPNG)
 find_package(PNG ${PNG_REQUIRED_VERSION})


Can this be indented?

-[Unknown]

Yes, this one is finding "system" libpng, which on macOS can come from a package manager, such as brew or MacPorts. Doing this will result in linking to a library that does not come with macOS and is thus broken.

I just mean mechanically, the find_package(PNG ${PNG_REQUIRED_VERSION}) should have a tab character before it.

-[Unknown]

unknownbrackets · 2021-09-16T04:30:17Z

Regarding the changelog thing, not related to this PR but I had to resort to a generic message on Android after a release got blocked when I mentioned the names of some fixed games in the log (plus it's limited to 500 chars which isn't enough).

A bad policy results in bad results... sigh. But this is most apps now, and although there's a good reason in this case I suppose... I consider the general state of release notes a real regression from the past.

I guess everyone's just hoping for the day where it's someone else's device you're renting, that you have no rights for whatsoever, and is entirely in the cloud. That way no one has to bother themselves with owning anything or having any distracting ability to fiddle with things or become a programmer as a child like I did.

Oh well. I shouldn't complain about people getting the future they want if it's the future they want...

-[Unknown]

See hrydgard/ppsspp#14839

hrydgard · 2021-09-16T06:53:58Z

At least we do provide proper release notes on the website, and I refer to it from the Play release notes as a workaround, but it's indeed a shame.

I changed my mind again, let's do the submodule. I created this repository with the contents: https://github.com/hrydgard/ppsspp-mac-sdl

Please just replace the macOS directory under SDL with a submodule pointer to this repository.

vit9696 · 2021-09-16T09:23:06Z

Done. A bit confused why does the CI fail at the moment though. Is not checkout recursive for test jobs (https://github.com/hrydgard/ppsspp/pull/14839/checks?check_run_id=3618812569)?

hrydgard · 2021-09-16T12:21:49Z

Huh, I would have thought that should have worked. We already rely on multiple submodules...

unknownbrackets · 2021-09-16T14:18:09Z

So, doing a shallow (--depth=1) clone into a new directory, and then git describe --always gives the sha (as expected.)

But then doing:

git fetch --deepen=15000 --tags

Does fetch a good chunk of data, but won't mess with submodules. That's enough to get a good git describe --always result.

-[Unknown]

vit9696 · 2021-09-16T14:32:37Z

@hrydgard are you sure you committed all the symlinks correctly?

hrydgard · 2021-09-16T16:09:41Z

@vit9696 I guess maybe I did not, since I did it on Windows! Wasn't aware there were symlinks.

I'll re-do it from Linux or Mac later. Maybe I should just extract SDL instead of copying from this branch.

vit9696 · 2021-09-16T16:12:55Z

Yes, frameworks heavily rely on symlinks.

hrydgard · 2021-09-16T16:39:30Z

Done, you're gonna have to update the submodule pointer to f19a1d54b8a5af6cc378ea307e0ec676922eb4cc .

vit9696 · 2021-09-16T21:49:22Z

So, doing a shallow (--depth=1) clone into a new directory, and then git describe --always gives the sha (as expected.)

But then doing:
git fetch --deepen=15000 --tags
Does fetch a good chunk of data, but won't mess with submodules. That's enough to get a good git describe --always result.

-[Unknown]

The CI run seems to disagree: https://github.com/vit9696/ppsspp/runs/3626248500

EricFromCanada · 2021-10-20T22:23:10Z

See Endless Sky for another implementation of automatic download & install of SDL with GitHub Actions.

Halo-Michael and others added 2 commits September 12, 2021 22:56

Update libMoltenVK.dylib to support M1 chips

8f60d99

Add FAT support for macOS with automatic GitHub releases

14c2231

closes #11799 closes #13708 closes #14815

vit9696 mentioned this pull request Sep 12, 2021

PPSSPP should use its own bundled ffmpeg Homebrew/homebrew-core#84737

Closed

2 tasks

unknownbrackets reviewed Sep 13, 2021

View reviewed changes

vit9696 commented Sep 13, 2021

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

vit9696 added 2 commits September 15, 2021 21:02

Fix git describe not fetching version tag

ed046ae

Include macOS name in the zip archive

b583d16

unknownbrackets approved these changes Sep 16, 2021

View reviewed changes

hrydgard added a commit to hrydgard/ppsspp-mac-sdl that referenced this pull request Sep 16, 2021

Initial commit.

111650f

See hrydgard/ppsspp#14839

vit9696 added 3 commits September 16, 2021 10:17

Drop bundled SDL2 for macOS

5fc7344

Switch to submodules and fix release version generation

e1cb1c9

Silence submodule fetch error screaming

90cb3d3

Update SDL and try faster commit history fetch

82249ce

Try to workaround git tag fetch failure

c8204a2

hrydgard merged commit cd05f3a into hrydgard:master Sep 17, 2021

unknownbrackets added this to the v1.12.0 milestone Sep 17, 2021

vit9696 mentioned this pull request Oct 8, 2021

macOS builds for 1.12 release #14997

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provide automatic macOS releases with Apple Silicon support #14839

Provide automatic macOS releases with Apple Silicon support #14839

vit9696 commented Sep 12, 2021

hrydgard commented Sep 12, 2021

vit9696 commented Sep 12, 2021 •

edited

Loading

unknownbrackets left a comment •

edited

Loading

unknownbrackets Sep 13, 2021

vit9696 Sep 13, 2021

unknownbrackets Sep 16, 2021

vit9696 commented Sep 13, 2021

vit9696 Sep 13, 2021 •

edited

Loading

vit9696 Sep 15, 2021

unknownbrackets Sep 16, 2021

vit9696 Sep 16, 2021

vit9696 commented Sep 13, 2021

hrydgard commented Sep 13, 2021

vit9696 commented Sep 13, 2021

vit9696 commented Sep 15, 2021

hrydgard commented Sep 15, 2021

unknownbrackets left a comment

unknownbrackets Sep 16, 2021

vit9696 Sep 16, 2021

unknownbrackets Sep 16, 2021

unknownbrackets commented Sep 16, 2021

hrydgard commented Sep 16, 2021

vit9696 commented Sep 16, 2021

hrydgard commented Sep 16, 2021

unknownbrackets commented Sep 16, 2021

vit9696 commented Sep 16, 2021

hrydgard commented Sep 16, 2021 •

edited

Loading

vit9696 commented Sep 16, 2021

hrydgard commented Sep 16, 2021

vit9696 commented Sep 16, 2021

EricFromCanada commented Oct 20, 2021


		GIT_VERSION_LINE=$(grep "PPSSPP_GIT_VERSION = " "$(dirname "${0}")/../git-version.cpp")
		echo "Setting version to ${GIT_VERSION_LINE}..."

Provide automatic macOS releases with Apple Silicon support #14839

Provide automatic macOS releases with Apple Silicon support #14839

Conversation

vit9696 commented Sep 12, 2021

hrydgard commented Sep 12, 2021

vit9696 commented Sep 12, 2021 • edited Loading

unknownbrackets left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vit9696 commented Sep 13, 2021

vit9696 Sep 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vit9696 commented Sep 13, 2021

hrydgard commented Sep 13, 2021

vit9696 commented Sep 13, 2021

vit9696 commented Sep 15, 2021

hrydgard commented Sep 15, 2021

unknownbrackets left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

unknownbrackets commented Sep 16, 2021

hrydgard commented Sep 16, 2021

vit9696 commented Sep 16, 2021

hrydgard commented Sep 16, 2021

unknownbrackets commented Sep 16, 2021

vit9696 commented Sep 16, 2021

hrydgard commented Sep 16, 2021 • edited Loading

vit9696 commented Sep 16, 2021

hrydgard commented Sep 16, 2021

vit9696 commented Sep 16, 2021

EricFromCanada commented Oct 20, 2021

vit9696 commented Sep 12, 2021 •

edited

Loading

unknownbrackets left a comment •

edited

Loading

vit9696 Sep 13, 2021 •

edited

Loading

hrydgard commented Sep 16, 2021 •

edited

Loading