Pyodide, system installs.

.github/workflows/test-valgrind.yml

@@ -64,6 +64,7 @@ jobs:
             log(f'{os.getcwd()=}')
 
             log('Installing valgrind.')
+            run(f'sudo apt update')
             run(f'sudo apt install valgrind')
             run(f'valgrind --version')
 

docs/installation.rst

@@ -6,21 +6,33 @@ Installation
 =============
 
 Requirements
----------------
+---------------------------------------------------------
 
 All the examples below assume that you are running inside a Python virtual
 environment. See: https://docs.python.org/3/library/venv.html for details.
+We also assume that `pip` is up to date.
+
+For example:
+
+* Windows::
 
-For example::
+    py -m venv pymupdf-venv
+    .\pymupdf-venv\Scripts\activate
+    python -m pip install --upgrade pip
+
+* Linux, MacOS::
 
     python -m venv pymupdf-venv
     . pymupdf-venv/bin/activate
+    python -m pip install --upgrade pip
 
 
+Installation
+---------------------------------------------------------
+
 PyMuPDF should be installed using pip with::
 
-  python -m pip install --upgrade pip
-  python -m pip install --upgrade pymupdf
+  pip install --upgrade pymupdf
 
 This will install from a Python wheel if one is available for your platform.
 
@@ -31,31 +43,19 @@ Installation when a suitable wheel is not available
 If a suitable Python wheel is not available, pip will automatically build from
 source using a Python sdist.
 
-**This requires C/C++ development tools and SWIG to be installed**:
-
-* On Unix-style systems such as Linux, OpenBSD and FreeBSD,
-  use the system package manager to install SWIG.
-
-  * For example on Debian Linux, do: `sudo apt install swig`
+**This requires C/C++ development tools to be installed**:
 
 * On Windows:
 
-  * Install Visual Studio 2019. If not installed in a standard location, set
+  *
+    Install Visual Studio 2019. If not installed in a standard location, set
     environmental variable `PYMUPDF_SETUP_DEVENV` to the location of the
     `devenv.com` binary.
-
-    * Having other installed versions of Visual Studio, for example Visual
-      Studio 2022, can cause problems because one can end up with MuPDF and
-      PyMuPDF code being compiled with different compiler versions.
-
-  * Install SWIG by following the instructions at:
-    https://swig.org/Doc4.0/Windows.html#Windows_installation
 
-* On MacOS, install MacPorts using the instructions at:
-  https://www.macports.org/install.php
-
-  * Then install SWIG with: `sudo port install swig`
-  * You may also need: `sudo port install swig-python`
+  *
+    Having other installed versions of Visual Studio, for example Visual Studio
+    2022, can cause problems because one can end up with MuPDF and PyMuPDF code
+    being compiled with different compiler versions.
 
 As of `PyMuPDF-1.20.0`, the required MuPDF source code is already in the
 sdist and is automatically built into PyMuPDF.
@@ -84,81 +84,81 @@ There are no **mandatory** external dependencies. However, some optional feature
 .. note:: You can install these additional components at any time -- before or after installing PyMuPDF. PyMuPDF will detect their presence during import or when the respective functions are being used.
 
 
-Installation from source without using an sdist
----------------------------------------------------------
+Build and install from local PyMuPDF checkout and optional local MuPDF checkout
+-------------------------------------------------------------------------------
+
+* Install C/C++ development tools as described above.
+
+* Enter a Python venv and update pip, as described above.
 
-* First get a PyMuPDF source tree:
+* Get a PyMuPDF source tree:
 
-  * Clone the git repository at https://github.com/pymupdf/PyMuPDF,
-    for example::
+  * Clone the PyMuPDF git repository::
 
       git clone https://github.com/pymupdf/PyMuPDF.git
 
   * Or download and extract a `.zip` or `.tar.gz` source release from
     https://github.com/pymupdf/PyMuPDF/releases.
 
-* Install C/C++ development tools and SWIG as described above.
 
 * Build and install PyMuPDF::
 
-    cd PyMuPDF && python setup.py install
+    cd PyMuPDF && pip install .
 
   This will automatically download a specific hard-coded MuPDF source release,
   and build it into PyMuPDF.
-
-.. note:: When running Python scripts that use PyMuPDF, make sure that the
-  current directory is not the `PyMuPDF/` directory.
 
-  Otherwise, confusingly, Python will attempt to import `fitz` from the local
-  `fitz/` directory, which will fail because it only contains source files.
 
-* Building for different Python versions in same PyMuPDF tree:
+Build and install PyMuPDF using a local MuPDF source tree:
 
-  *
-    PyMuPDF will build for the version of Python that runs `setup.py`. So for
-    example on Windows one can build different versions by using `py -3.9 ` or
-    `py -3.10-32`.
+* Clone the MuPDF git repository::
 
+    git clone --recursive https://ghostscript.com:/home/git/mupdf.git
 
-Running tests
----------------------------------------------------------
+*
+  Build PyMuPDF, specifying the location of the local MuPDF tree with the
+  environmental variables `PYMUPDF_SETUP_MUPDF_BUILD`::
 
-Having a PyMuPDF tree available allows one to run PyMuPDF's `pytest` test
-suite::
+    cd PyMuPDF && PYMUPDF_SETUP_MUPDF_BUILD=../mupdf pip install .
 
-    pip install pytest fontTools
-    pytest PyMuPDF/tests
 
+Building for different Python versions in same PyMuPDF tree:
 
-Building and testing with git checkouts of PyMuPDF and MuPDF
-------------------------------------------------------------------------------------------------------------------
+*
+  PyMuPDF will build for the version of Python that is being used to run
+  `pip`. To run `pip` with a specific Python version, use `python -m pip`
+  instead of `pip`.
 
-Things to do:
+  So for example on Windows one can build different versions with::
 
-* Install C/C++ development tools and SWIG as described above.
-* Get PyMuPDF.
-* Get MuPDF.
-* Create a Python virtual environment.
-* Build PyMuPDF with environmental variable `PYMUPDF_SETUP_MUPDF_BUILD` set
-  to the path of the local MuPDF checkout.
-* Run PyMuPDF tests.
+    cd PyMuPDF && py -3.9 -m pip install .
 
-For example::
+  or::
 
-    git clone https://github.com/pymupdf/PyMuPDF.git
-    git clone -b 1.23.x --recursive https://ghostscript.com:/home/git/mupdf.git
-    python -m venv pymupdf-venv
-    . pymupdf-venv/bin/activate
-    cd PyMuPDF
-    PYMUPDF_SETUP_MUPDF_BUILD=../mupdf python setup.py install
-    cd ..
-    pip install pytest fontTools
-    pytest PyMuPDF
+    cd PyMuPDF && py -3.10-32 -m pip install .
 
 
-Using a non-default MuPDF
+.. note:: When running Python scripts that use PyMuPDF, make sure that the
+  current directory is not the `PyMuPDF/` directory.
+
+  Otherwise, confusingly, Python will attempt to import `fitz` from the local
+  `fitz/` directory, which will fail because it only contains source files.
+
+
+Running tests
 ---------------------------------------------------------
 
+Having a PyMuPDF tree available allows one to run PyMuPDF's `pytest` test
+suite::
+
+  pip install pytest fontTools
+  pytest PyMuPDF/tests
+
+
+
+Notes about using a non-default MuPDF
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
 Using a non-default build of MuPDF by setting environmental variable
 `PYMUPDF_SETUP_MUPDF_BUILD` can cause various things to go wrong and so is
 not generally supported:

pipcl.py

@@ -1222,6 +1222,7 @@ def build_extension(
         prerequisites_swig=None,
         prerequisites_compile=None,
         prerequisites_link=None,
+        infer_swig_includes=True,
         ):
     '''
     Builds a Python extension module using SWIG. Works on Windows, Linux, MacOS
@@ -1297,7 +1298,13 @@ def build_extension(
 
                 On non-Windows we use cc's -MF and -MF args to generate dynamic
                 dependencies so this is not usually required.
-
+        infer_swig_includes:
+            If true, we extract `-I<path>` and `-I <path>` args from
+            `compile_extra` (also `/I` on windows) and use them with swig so
+            that it can see the same header files as C/C++. This is useful
+            when using enviromment variables such as `CC` and `CXX` to set
+            `compile_extra.
+
     Returns the leafname of the generated library file within `outdir`, e.g.
     `_{name}.so` on Unix or `_{name}.cp311-win_amd64.pyd` on Windows.
     '''
@@ -1312,6 +1319,22 @@ def build_extension(
     os.makedirs( outdir, exist_ok=True)
 
     # Run SWIG.
+
+    if infer_swig_includes:
+        # Extract include flags from `compiler_extra`.
+        swig_includes_extra = ''
+        compiler_extra_items = compiler_extra.split()
+        i = 0
+        while i < len(compiler_extra_items):
+            item = compiler_extra_items[i]
+            # Swig doesn't seem to like a space after `I`.
+            if item == '-I' or (windows() and item == '/I'):
+                swig_includes_extra += f' -I{compiler_extra_items[i+1]}'
+                i += 1
+            elif item.startswith('-I') or (windows() and item.startswith('/I')):
+                swig_includes_extra += f' -I{compiler_extra_items[i][2:]}'
+            i += 1
+        swig_includes_extra = swig_includes_extra.strip()
     deps_path = f'{path_cpp}.d'
     prerequisites_swig2 = _get_prerequisites( deps_path)
     run_if(
@@ -1325,6 +1348,7 @@ def build_extension(
                 -o {path_cpp}
                 -MD -MF {deps_path}
                 {includes_text}
+                {swig_includes_extra}
                 {path_i}
             '''
             ,
@@ -1499,10 +1523,10 @@ def build_extension(
                         -o {path_so}
                         {compiler_extra}
                         {libpaths_text}
-                        {libs_text}
-                        {rpath_flag}
                         {linker_extra}
                         {pythonflags.ldflags}
+                        {libs_text}
+                        {rpath_flag}
                     '''
         run_if(
                 command,
@@ -1823,7 +1847,13 @@ def macos_patch( library, *sublibraries):
         name = name.split('\n')
         assert len(name) == 2 and name[0] == f'{sublibrary}:', f'{name=}'
         name = name[1]
-        command += f' -change {name} @rpath/{os.path.basename(name)}'
+        # strip trailing so_name.
+        leaf = os.path.basename(name)
+        m = re.match('^(.+[.]((so)|(dylib)))[0-9.]*$', leaf)
+        assert m
+        _log(f'Changing {leaf=} to {m.group(1)}')
+        leaf = m.group(1)
+        command += f' -change {name} @rpath/{leaf}'
     command += f' {library}'
     _log( f'Running: {command}')
     subprocess.run( command, shell=1, check=1)
@@ -1914,7 +1944,6 @@ def run_if( command, out, *prerequisites, verbose=True):
         pipcl.py: run_if(): Not running command because up to date: 'run_if_test_out'
     '''    
     doit = False
-
     if not doit:
         out_mtime = _fs_mtime( out)
         if out_mtime == 0:
@@ -1927,9 +1956,15 @@ def run_if( command, out, *prerequisites, verbose=True):
     else:
         cmd = None
     if command != cmd:
-        doit = 'Command has changed'
+        if cmd is None:
+            doit = 'No previous command stored'
+        else:
+            doit = f'Command has changed'
+            if 0:
+                doit += f': {cmd!r} => {command!r}'
 
     if not doit:
+        # See whether any prerequisites are newer than target.
         def _make_prerequisites(p):
             if isinstance( p, (list, tuple)):
                 return list(p)

scripts/gh_release.py

@@ -353,8 +353,8 @@ def build_pyodide_wheel():
     # Disable libcrypto because not available in Pyodide.
     env_extra['HAVE_LIBCRYPTO'] = 'no'
 
-    # Set OS=wasm-mt for MuPDF build.
-    env_extra['OS'] = 'wasm-mt'
+    # Tell MuPDF to build for Pyodide.
+    env_extra['OS'] = 'pyodide'
 
     # Build PyMuPDF as a single wheel without a separate PyMuPDFb
     # wheel.
@@ -492,7 +492,7 @@ def pyodide_setup(clean=False):
         command += f' && {sys.executable} -m venv {venv_pyodide}'
     command += f' && . {venv_pyodide}/bin/activate'
     command += f' && echo "### running pip install ..."'
-    command += f' && python -m pip install --upgrade pip wheel pyodide-build'
+    command += f' && python -m pip install --upgrade pip wheel pyodide-build==0.23.4'
 
     # Run emsdk install scripts and enter emsdk environment.
     #