Expose modules in Flint trunk

[edgarcosta]

Is there an easy way to expose modules not yet available in Flint 3.0.0?

Related, would it be possible for one to provide a fat version of pyflint where we include flint instead of linking against the system version?

[oscarbenjamin]

Is there an easy way to expose modules not yet available in Flint 3.0.0?

I don't understand hat this means. Can you give an example of a module that is not yet available?

The wheels as provided on PyPI already include Flint and will not link against the system version. If you build python-flint yourself from source then you'll need to arrange to bundle Flint into the wheel if that is what you want. This is handled by cibuildwheel and auditwheel etc for the wheels that are on PyPI.

[edgarcosta]

The module acb_theta.

Thanks for the clarification.
If that is the case why not have flint as a submodule, and build directly from it?

[oscarbenjamin]

If you build against current flint trunk then you should have acb_theta.h available which would mean you could write Cython code to call the functions there. Otherwise you could use ctypes.

why not have flint as a submodule, and build directly from it?

I don't understand what you mean by this. Are you familiar with how the current python-flint code is organised? What exactly might you suggest to do differently?

[edgarcosta]

I am sorry, I am certainly struggling to understand how python-flint code is organized.

I looked at some of your scripts in bin, and I see in bin/build_dependencies_unix.sh

curl -O -L https://www.flintlib.org/flint-$FLINTVER.tar.gz
tar xf flint-$FLINTVER.tar.gz
cd flint-$FLINTVER
  ./bootstrap.sh
  ./configure --disable-static
  make -j
  sudo make install

and similarly in bin/build_dependencies_unix.sh but now
curl -O -L https://github.com/flintlib/flint/releases/download/v$FLINTVER/flint-$FLINTVER.tar.gz

From this, I conclude, perhaps wrongly, that python-flint is always behind flintlib/flint, as it must wait for a new release.

Furthermore, from the README,

First install FLINT 3.0.0. Note that as of python-flint 0.5.0 only this exact version of FLINT will work. 

I get the idea that I cannot even try to build it against a newer version of FLINT.

Getting back to your second question.

What exactly might you suggest to do differently?

First, let me start by saying that I have zero experience in deploying wheels.
I was proposing something like what I did in the past, pycontrolledreduction or pyhdme, where I put the libraries that I was wrapping as a submodule in the python wrapper.
In particular, when the user does something like

sage -pip install  git+https://github.com/edgarcosta/pyhdme.git

it builds the dependent library on the fly.
The advantage that I see in this, is that one can keep the branch master up to date with the trunk of flint.
Furthermore, it enables users to access the latest version by doing,

pip install  git+https://github.com/flintlib/python-flint.git

as long as they could already build flint.

Does this make sense?

For some transparency, my ultimate goal is, on a short-term basis, to expose my collaborators to the module acb_theta, who might not be comfortable with C code or building Flint and its dependencies by themselves.

[oscarbenjamin]

It is possible to build python-flint against a system of install of Flint but it needs to be at least Flint 3.0.0. The README instruction was written before Flint 3.0.1 was released and so at the time it was true that only the exact 3.0.0 version would work. This was because the changes to accommodate 3.0.0 meant that python-flint's Cython code was not compatible with Flint 2.9.0 any more. The README should be updated now to say at least Flint >= 3.0.0 if not Flint >= 3.0.1 (since 3.0.0 is not tested any more).

It would have been possible for python-flint to maintain compatibility with Flint 2.9.0 and also 3.0.0+ but it seemed like a bunch of work for relatively little gain. Going forwards I would like to keep Flint 3.0.0 (or perhaps 3.0.1) as a minimum supported version for python-flint if it is not too difficult to do so. Ideally then it should be possible to build python-flint against whatever system install of Flint is being used at least for Flint >= 3.0.1 which would hopefully mean that in a few years it is reasonable to expect that someone can build the latest release of python-flint from source against e.g. distro-provided packages of Flint, GMP etc or equally a local build of Flint trunk if that is what is being used.

The build scripts that download Flint etc currently use these versions:

python-flint/bin/build_variables.sh

Lines 21 to 24 in e5da576

	# These are the actual dependencies used (at least by default):
	GMPVER=6.3.0
	MPFRVER=4.1.0
	FLINTVER=3.0.1

Other versions can be used but these are the versions that are tested in CI and that are used for the wheels that are distributed to PyPI. The Flint version has kept up to date very quickly after a release (or release candidate) of Flint but testing is not done with Flint trunk directly.

When the next release of Flint is available I would expect to split the CI jobs so that there are some that test against 3.0.1 and others that test e.g. 3.1.0. I expect that the wheels that are built for PyPI will always use and bundle the latest version of Flint though. Probably just for Linux we can add some build from source jobs for a matrix of Flint versions (perhaps also Flint trunk) here:

python-flint/.github/workflows/buildwheel.yml

Lines 121 to 137 in e5da576

	test_pip_vcs_sdist:
	name: pip install ${{ matrix.target }} on ${{ matrix.python-version }}
	runs-on: ubuntu-22.04
	strategy:
	fail-fast: false
	matrix:
	python-version: ['3.11', '3.12']
	# '.' means install from git checkout
	# 'python-flint' means install from PyPI sdist
	target: ['.', 'python-flint']
	steps:
	- uses: actions/checkout@v3
	- uses: actions/setup-python@v4
	with:
	python-version: ${{ matrix.python-version }}
	- run: bin/pip_install_ubuntu.sh ${{ matrix.target }}
	- run: python -m flint.test --verbose

Building Flint on the fly is something that I have explored using meson (see gh-52). Probably though we would always want something like pip install python-flint to use the latest release of Flint rather than Flint trunk though. It might be possible to have an extra like pip install python-flint[trunk] but I'm not sure exactly how to make that work.

For something like acb_theta I would ideally want to make it conditional so that python-flint can still be compatible with Flint 3.0.1. There are a few options for how to do that that might be suitable depending on how exactly we would want to expose the acb_theta functionality in python-flint e.g. it could be an extension module that is only built conditionally or something like that.

my ultimate goal is, on a short-term basis, to expose my collaborators to the module acb_theta, who might not be comfortable with C code or building Flint and its dependencies by themselves.

Part of the purpose of python-flint is to make Flint's functionality more accessible like this but we achieve that by bundling everything so that the user can do pip install python-flint and get prebuilt binaries for the latest release of Flint bundled in the wheel. When the next release of Flint comes out this is what we will do.

In the meantime though the relevant Cython code would need to be added to python-flint so that it can expose the acb_theta functionality.

[oscarbenjamin]

After gh-119 there is now CI testing against Flint's main branch. That PR also shows one way to use compile time checks to handle differences between Flint versions.