Versioning and APIs?

[AstraLuma]

I just fielded a support question in PPB about an older system (defaulting to Python 3.5 and SDL 2.0.4, which granted are fairly old at this point), but I thought I would open the question in general:

Can we provide a smarter way to handle older SDLs? I know a lot of this can be alleviated by a-hurst/pysdl2-dll#2, but maybe there's a more graceful way to handle old versions? Failing at import isn't great.

Example backtrace:

Traceback (most recent call last):
  File "main.py", line 1, in <module>
    import ppb
  File "/home/sun/skc/prog/.venv/lib/python3.7/site-packages/ppb/__init__.py", line 7, in <module>
    from ppb.assets import Circle
  File "/home/sun/skc/prog/.venv/lib/python3.7/site-packages/ppb/assets.py", line 1, in <module>
    import sdl2.ext
  File "/home/sun/skc/prog/.venv/lib/python3.7/site-packages/sdl2/__init__.py", line 5, in <module>
    from .audio import *
  File "/home/sun/skc/prog/.venv/lib/python3.7/site-packages/sdl2/audio.py", line 175, in <module>
    SDL_DequeueAudio = _bind("SDL_DequeueAudio", [SDL_AudioDeviceID, c_void_p, Uint32], Uint32)
  File "/home/sun/skc/prog/.venv/lib/python3.7/site-packages/sdl2/dll.py", line 150, in bind_function
    raise ValueError(e % (funcname, self._libfile, libver))
ValueError: Could not find function 'SDL_DequeueAudio' in libSDL2-2.0.so.0 (SDL2 2.0.4)

[AstraLuma]

This version of SDL is in Ubuntu 16.04.

EDIT: This user is on Mint 18.3, which was released in November 2017 and is based on Ubuntu 16.04.

[AstraLuma]

https://repology.org/project/sdl2/versions might be helpful.

[a-hurst]

At present, the minimum version of SDL2 supported is 2.0.5, but I think 2.0.4 should be able to be supported with a little extra work provided that none of the high-level wrapper functions in the sdl2.ext module depend on anything added in 2.0.5.

Basically, py-sdl2's backwards-compatibility mechanism works by marking recently-introduced functions with the version they were introduced in, and then raising an informative exception when a function is called that requires a newer SDL2 version by comparing the required version to the installed version. Theoretically, all that's needed here should be flagging functions added in 2.0.5 so 2.0.4 doesn't try to load them. Also, for even older versions, I should probably have it fail with an informative exception rather than an opaque one like that.

[AstraLuma]

Yeah, unfortunately, PPB has had a few people come in with Ubuntu 16.04 systems, and we expect Enterprise Fun with our target audience as we grow. (16.04 is supported upstream until next year.)

Like I said, we could also "fix" this issue by adding Linux builds to pysdl2-dll, which means that software wouldn't be reliant on distro SDL versions for anything.

[a-hurst]

Yeah, that would be the ideal solution, since it means you don't have to worry too much about end-users being on an older version can make better use of new SDL2 features as they're added without risking backwards compatibility. I'm working on making the "DLL too old" exception more informative right now since that should be fixed anyway, and then I'll look and see how much effort bringing back 2.0.4 support should be (it might only take a few minutes if I'm lucky), but I'll revisit the pysdl2-dll Linux wheels soon too (I have a virtual conference next week to prep for, so I won't be able to dive deep into it until after that)

[AstraLuma]

I already volunteered to do Linux DLLs, since I do a lot of CI/Pipelines, but I'll be happy to have collaborators.

[a-hurst]

Right, if you're willing to take the lead on that I'd be happy to facilitate in any way I can! Is there anything you need from me on that front?

I realize I still need to send you the mostly-complete build script for all the Linux SDL2 libraries + dependencies that I mentioned earlier, IIRC the thing I got stuck on was that the version of WebP (one of the dependencies for SDL2_image) included in the official SDL2_image repo didn't want to build from source due to some issue with autoconf or something, but I think everything else downloads and compiles.

[AstraLuma]

Yeah, that'll help a lot! One of the things I have to figure out is which libraries to bundle and which to assume exist on the system, and then how to build each of them on manylinux.

[a-hurst]

Okay, here it is! It's basically just the getsdl2.py script found in py-sdl2's .ci folder, but it's been modified to also build the dependencies included in the "external" folders of the source for the mixer/ttf/image libraries.

started playing around with it again yesterday and couldn't get past the libtool issue I'd run into before, so I created a fresh CentOS 7 VM (basis of the manylinux2014 image) and tried running the script there instead. This time, I think the issue is that the build process isn't configured correctly to look for headers and libraries in the ./sdlprefix folder I'm installing all of the libraries and headers to, since it's complaining that zlib isn't installed when building libpng even though zlib.h and libz are clearly present in the sdlprefix. Hopefully you know more about this stuff than I do and can get it all building properly.

import os
import sys
import shutil
import tarfile
import subprocess as sub
from zipfile import ZipFile 
from distutils.util import get_platform
import subprocess as sub

try:
    from urllib.request import urlopen # Python 3.x
except ImportError:
    from urllib2 import urlopen # Python 2


libraries = ['SDL2', 'SDL2_image']#, 'SDL2_mixer', 'SDL2_ttf', 'SDL2_gfx']

sdl2_urls = {
    'SDL2': 'https://www.libsdl.org/release/SDL2-{0}{1}',
    'SDL2_mixer': 'https://www.libsdl.org/projects/SDL_mixer/release/SDL2_mixer-{0}{1}',
    'SDL2_ttf': 'https://www.libsdl.org/projects/SDL_ttf/release/SDL2_ttf-{0}{1}',
    'SDL2_image': 'https://www.libsdl.org/projects/SDL_image/release/SDL2_image-{0}{1}',
    'SDL2_gfx': 'https://github.com/a-hurst/sdl2gfx-builds/releases/download/{0}/SDL2_gfx-{0}{1}'
}

libversions = {
    '2.0.10': {
        'SDL2': '2.0.10',
        'SDL2_mixer': '2.0.4',
        'SDL2_ttf': '2.0.15',
        'SDL2_image': '2.0.5',
        'SDL2_gfx': '1.0.4'
    },
    '2.0.9': {
        'SDL2': '2.0.9',
        'SDL2_mixer': '2.0.4',
        'SDL2_ttf': '2.0.14',
        'SDL2_image': '2.0.4',
        'SDL2_gfx': '1.0.4'
    },
    '2.0.8': {
        'SDL2': '2.0.8',
        'SDL2_mixer': '2.0.2',
        'SDL2_ttf': '2.0.14',
        'SDL2_image': '2.0.3',
        'SDL2_gfx': '1.0.4'
    },
    '2.0.7': {
        'SDL2': '2.0.7',
        'SDL2_mixer': '2.0.2',
        'SDL2_ttf': '2.0.14',
        'SDL2_image': '2.0.2',
        'SDL2_gfx': '1.0.4'
    },
    '2.0.6': {
        'SDL2': '2.0.6',
        'SDL2_mixer': '2.0.1',
        'SDL2_ttf': '2.0.14',
        'SDL2_image': '2.0.1',
        'SDL2_gfx': '1.0.4'
    },
    '2.0.5': {
        'SDL2': '2.0.5',
        'SDL2_mixer': '2.0.1',
        'SDL2_ttf': '2.0.14',
        'SDL2_image': '2.0.1',
        'SDL2_gfx': '1.0.4'
    }
}


def getDLLs(platform_name, version):
    
    dlldir = os.path.join('dlls')
    for d in [dlldir, 'temp']:
        if os.path.isdir(d):
            shutil.rmtree(d)
        os.mkdir(d)
    
    if 'macosx' in platform_name:
        
        for lib in libraries:
            
            mountpoint = '/tmp/' + lib
            dllname = lib + '.framework'
            dllpath = os.path.join(mountpoint, dllname)
            dlloutpath = os.path.join(dlldir, dllname)
            
            # Download disk image containing library
            libversion = libversions[version][lib]
            dmg = urlopen(sdl2_urls[lib].format(libversion, '.dmg'))
            outpath = os.path.join('temp', lib + '.dmg')
            with open(outpath, 'wb') as out:
                out.write(dmg.read())
            
            # Mount image, extract framework, then unmount
            sub.check_call(['hdiutil', 'attach', outpath, '-mountpoint', mountpoint])
            shutil.copytree(dllpath, dlloutpath, symlinks=True)
            sub.call(['hdiutil', 'unmount', mountpoint])

    elif platform_name in ['win32', 'win-amd64']:
        
        suffix = '-win32-x64.zip' if platform_name == 'win-amd64' else '-win32-x86.zip'
        
        for lib in libraries:
            
            # Download zip archive containing library
            libversion = libversions[version][lib]
            dllzip = urlopen(sdl2_urls[lib].format(libversion, suffix))
            outpath = os.path.join('temp', lib + '.zip')
            with open(outpath, 'wb') as out:
                out.write(dllzip.read())
            
            # Extract dlls and license files from archive
            with ZipFile(outpath, 'r') as z:
                for name in z.namelist():
                    if name[-4:] == '.dll':
                        z.extract(name, dlldir)
                        
    else:

        suffix = '.tar.gz' # source code
        gfxsrc = 'http://www.ferzkopp.net/Software/SDL2_gfx/SDL2_gfx-{0}.tar.gz'
        basedir = os.getcwd()

        libdir = os.path.join(basedir, 'sdlprefix')
        if os.path.isdir(libdir):
            shutil.rmtree(libdir)
        os.mkdir(libdir)

        for lib in libraries:

            libversion = libversions[version][lib]
            print('\n======= Downloading {0} {1} =======\n'.format(lib, libversion))
            
            # Download tar archive containing source
            liburl = sdl2_urls[lib].format(libversion, suffix)
            if lib == 'SDL2_gfx':
                liburl = gfxsrc.format(libversion)
            outpath = os.path.join('temp', lib + suffix)
            if not os.path.exists(outpath):
                srctar = urlopen(liburl)
                with open(outpath, 'wb') as out:
                    out.write(srctar.read())
            
            # Extract source from archive
            sourcepath = os.path.join('temp', lib + '-' + libversion)
            with tarfile.open(outpath, 'r:gz') as z:
                z.extractall(path='temp')

            pkgconfig_dir = os.path.join(libdir, 'lib', 'pkgconfig')
            buildenv = os.environ
            if os.path.exists(pkgconfig_dir):
                buildenv['PKG_CONFIG_PATH'] = pkgconfig_dir

            # Build any dependencies
            buildcmds = [
                ['./configure', '--prefix={0}'.format(libdir)],
                ['make'],
                ['make', 'install']
            ]
            ignore = ['libvorbisidec', 'libwebp'] # vorbisidec only needed for special non-standard builds
            autoreconf = ['libwebp']
            extra_args = {
                'opusfile': ['--disable-http']
            }
            ext_dir = os.path.join(sourcepath, 'external')
            if os.path.exists(ext_dir):
                dependencies = os.listdir(ext_dir)
                dependencies.sort(reverse=True) # so zlib gets built first
                for dep in dependencies:
                    dep_path = os.path.join(ext_dir, dep)
                    if not os.path.isdir(dep_path):
                        continue
                    depname, depversion = dep.split('-')
                    if depname in ignore:
                            continue
                    print('======= Compiling {0} dependency {1} =======\n'.format(lib, dep))
                    os.chdir(dep_path)
                    #if depname in autoreconf:
                    #    p = sub.Popen(['libtoolize'], stdout=sys.stdout, stderr=sys.stderr)
                    #    p.communicate()
                    #    if p.returncode != 0:
                    #        raise RuntimeError("Error building {0}".format(dep))
                    for cmd in buildcmds:
                        if cmd[0] == './configure' and depname in extra_args.keys():
                            cmd = cmd + extra_args[depname]
                        p = sub.Popen(cmd, stdout=sys.stdout, stderr=sys.stderr, env=buildenv)
                        p.communicate()
                        if p.returncode != 0:
                            raise RuntimeError("Error building {0}".format(dep))
                    print('\n======= {0} built sucessfully =======\n'.format(dep))
                    os.chdir(basedir)

            # Build the library
            buildcmds = [
                ['./configure', '--prefix={0}'.format(libdir)],
                ['make'],
                ['make', 'install']
            ]
            print('======= Compiling {0} {1} =======\n'.format(lib, libversion))
            os.chdir(sourcepath)
            if lib == "SDL2_mixer":
                p = sub.Popen(['autoreconf', '-vfi'], stdout=sys.stdout, stderr=sys.stderr)
                p.communicate()
                if p.returncode != 0:
                    raise RuntimeError("Error building {0}".format(lib))
            for cmd in buildcmds:
                p = sub.Popen(cmd, stdout=sys.stdout, stderr=sys.stderr, env=buildenv)
                p.communicate()
                if p.returncode != 0:
                    raise RuntimeError("Error building {0}".format(lib))
            os.chdir(basedir)

            # Copy built library to dll folder and reset working dir
            print('\n======= {0} {1} built sucessfully =======\n'.format(lib, libversion))
            for f in os.listdir(os.path.join(libdir, 'lib')):
                if f == "lib{0}.so".format(lib):
                    fpath = os.path.join(libdir, 'lib', f)
                    if os.path.islink(fpath):
                        fpath = os.path.realpath(fpath)
                    shutil.copy(fpath, dlldir)

        print("Installed binaries:")
        print(os.listdir(dlldir))
            

if __name__ == '__main__':
    getDLLs(get_platform(), '2.0.10')

[a-hurst]

As for the dependencies needed for SDL2 and what we can assume will be available on a given system, I don't think we need to worry too much about that: going by the official libsdl.org binaries for macOS and Windows there are no dependency libraries bundled with SDL2 itself, and since it only dynamically links to libraries if they're available on the host system, we should be able to build SDL2 with the headers for essentially every audio/video/input back-end that it supports and assume that end-users will have at least one of each installed (i.e. we won't need to bundle ALSA or JACK to make sure SDL2 has a valid audio back-end).

The main dependency libraries we'll want to include are the dependencies for ttf/mixer/image, which are also provided in the official macOS/Windows binaries. Thankfully the dependency libraries for each module are provided in the "external" folders of their source code releases, so we thankfully don't need to gather those ourselves.

[a-hurst]

Anyway, I'm not going to be available much over the next few days as I have to finish some data analysis and prepare a poster for a virtual conference next week, but if there's anything else I can quickly clarify or help with let me know and I'll try to briefly respond!

[ell1e]

Please make SDL2 binaries an optional extra package. Shipping all the libraries you linked has licensing implications, since e.g. SDL_Image, SDL_mixer, and also SDL_TTF use lots of external stuff under less liberal licensing than zlib. This is bad for applications that use SDL2 but never planned to use any of the extension libraries. It's great to have this easier for people who want it, but IMHO it shouldn't be the only way to use PySDL2.

Edit: just to be more specific:

Thankfully the dependency libraries for each module are provided in the "external" folders

This is exactly the problem, these have licensing that often is more restrictive, comparatively speaking, than zlib (e.g. requires crediting etc). So these aren't good to be forced to pull into an application if there weren't any plans to use them.

[a-hurst]

Hi @ell1e, right now things are already divided into the pysdl2 package (just the ctypes bindings) and the optional separatepysdl2-dll package containing the binaries for SDL2 + addons for the given target platform. Are you talking about separating the pysdl2-dll package into two separate packages, with one only containing SDL2 and the other containing the addon modules + dependencies?

[ell1e]

Ah no, nevermind then. I didn't know it was separated out already, I was thinking of the scenario of using a common installer packager to package a python app with site-packages as-is into an .exe or Android .apk where if one wants more granularity than all SDL2 libs prebuilt or none prebuilt I think it's fair to manually add dlls. It's just usually a pain to remove site-packages again, so if it hadn't been separated out already that would have been a potential issue, but it's usually not difficult to manually add arbitrary files (like more dlls) so that's fine.

[a-hurst]

Closing, as I believe the original issue was fixed in this commit: 92fee41

Also, with manylinux wheels this should be much less of an issue.