Install sage_conf using flit instead of setuptools #29854

mkoeppe · 2020-06-13T04:30:21Z

So far this is just an illustration of what can be done with flit (#29846) and what cannot:

Both non-src and src layouts are supported (for the latter, see WIP: allow src dir pypa/flit#260).
Symlinks of directories or files to somewhere outside of the root are not supported.

Not intended for merging at this point.

Depends on #29846

Component: build

Branch/Commit: u/mkoeppe/install_sage_conf_using_flit @ 7ea5fab

Issue created by migration from https://trac.sagemath.org/ticket/29854

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-06-13T04:30:50Z

Branch: u/mkoeppe/install_sage_conf_using_flit

mkoeppe · 2020-06-13T04:31:24Z

Last 10 new commits:

`dacea8b`	`WIP: Install sage-env-config with sage_conf`
`1ee054c`	`Fixup`
`5f7b1cc`	`Merge branch 't/21559/change-src-bin-installation' into t/29850/install_sage_env_config_with_sage_conf`
`dfb4400`	`build/pkgs/pytoml: New`
`11b6003`	`build/pkgs/flit, flit_core: New`
`20e2d59`	`Add dependencies`
`721e8fd`	`build/bin/sage-pip-install: For PEP 517 packages, do not try to uninstall first`
`5e7155e`	`build/pkgs/pytoml/spkg-install.in: New`
`8f58336`	`Merge branch 't/29846/add_packages_flit_and_dependencies_flit_core__pytoml' into t/29850/install_sage_env_config_with_sage_conf`
`fd83992`	`build/pkgs/sage_conf: Install with flit instead of setuptools`

mkoeppe · 2020-06-13T04:31:24Z

Commit: fd83992

mkoeppe · 2020-06-13T04:31:24Z

Work Issues: Rebase so it is not on top of #29850

mkoeppe · 2020-06-13T19:27:03Z

comment:3

This branch does not work because it is trying to use a symlink for sage_conf.py into the SAGE_ROOT/src directory.

[sage_conf-none] Processing /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/sage_conf/src
[sage_conf-none]   Created temporary directory: /private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-build-w7fopzwn
[sage_conf-none]   Added file:///Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/sage_conf/src to build tracker '/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-tracker-jmerw27d'
[sage_conf-none]     Created temporary directory: /private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-modern-metadata-dxzef0bb
[sage_conf-none]     Preparing wheel metadata: started
[sage_conf-none]     Running command /Users/mkoeppe/s/sage/sage-rebasing/local/bin/python3 /Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmpb7arkmg5
[sage_conf-none]     Traceback (most recent call last):
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
[sage_conf-none]         main()
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
[sage_conf-none]         json_out['return_val'] = hook(**hook_input['kwargs'])
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
[sage_conf-none]         return hook(metadata_directory, config_settings)
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit_core/buildapi.py", line 28, in prepare_metadata_for_build_wheel
[sage_conf-none]         module = Module(ini_info.module, os.getcwd())
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit_core/common.py", line 55, in __init__
[sage_conf-none]         raise ValueError("No file/folder found for module {}".format(name))
[sage_conf-none]     ValueError: No file/folder found for module sage_conf
[sage_conf-none]     Preparing wheel metadata: finished with status 'error'
[sage_conf-none] ERROR: Command errored out with exit status 1: /Users/mkoeppe/s/sage/sage-rebasing/local/bin/python3 /Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmpb7arkmg5 Check the logs for full command output.

sagetrac-git · 2020-06-13T19:41:28Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`e537d48`	`build/pkgs/sage_conf: Install with flit instead of setuptools`

sagetrac-git · 2020-06-13T19:41:28Z

Changed commit from fd83992 to e537d48

mkoeppe · 2020-06-13T19:41:44Z

Changed work issues from Rebase so it is not on top of #29850 to none

mkoeppe · 2020-06-13T19:41:44Z

Dependencies: #29846

mkoeppe · 2020-06-13T19:43:26Z

comment:6

Rebased on top of only #29846 (flit)

sagetrac-git · 2020-06-13T20:03:04Z

Branch pushed to git repo; I updated commit sha1. New commits:

`27cf73b`	`build/pkgs/sage_conf: Switch to src layout where src is a symlink (also does not work)`
`4d20d4a`	`build/pkgs/sage_conf: Back to sage_conf.py[.in] as a file here, not a symlink (works)`

sagetrac-git · 2020-06-13T20:03:04Z

Changed commit from e537d48 to 4d20d4a

mkoeppe · 2020-06-13T20:55:35Z

comment:10

Both with this version and the version in 27cf73b, also:

flit build 
Fetching list of valid trove classifiers                                                                                                                                                                      I-flit.validate
Version number normalised: '9.2.beta0' -> '9.2b0' (see PEP 440)                                                                                                                                         W-flit_core.versionno
Found 4 files tracked in git                                                                                                                                                                                     I-flit.sdist
Traceback (most recent call last):
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/bin/flit", line 8, in <module>
    sys.exit(main())
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/__init__.py", line 171, in main
    gen_setup_py=args.setup_py)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/build.py", line 46, in main
    sdist_file = sb.build(dist_dir, gen_setup_py=gen_setup_py)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/sdist.py", line 223, in build
    return Path(super().build(str(target_dir), gen_setup_py=gen_setup_py))
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit_core/sdist.py", line 181, in build
    files_to_add = self.apply_includes_excludes(self.select_files())
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit_core/sdist.py", line 155, in apply_includes_excludes
    .format(", ".join(missing_crucial)))
Exception: Crucial files were excluded from the sdist: src/sage_conf.py

(because sage_conf.py is generated and not tracked in git)

sagetrac-git · 2020-06-13T21:05:26Z

Branch pushed to git repo; I updated commit sha1. New commits:

`7ea5fab`	`build/pkgs/sage_conf/src/pyproject.toml: Add tool.flit.sdist section`

sagetrac-git · 2020-06-13T21:05:26Z

Changed commit from 4d20d4a to 7ea5fab

mkoeppe · 2020-06-13T21:16:48Z

comment:12

This fixes the sdist.

Same change on top of 27cf73b creates a tar file with a packaged symlink:

 lrwxr-xr-x       0/0             0 sage_conf-9.2b0/src --> ../../../../src

and gives an error afterwards:

Built sdist: dist/sage_conf-9.2b0.tar.gz                                                                                                                                                                    I-flit_core.sdist
Traceback (most recent call last):
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/bin/flit", line 8, in <module>
    sys.exit(main())
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/__init__.py", line 171, in main
    gen_setup_py=args.setup_py)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/build.py", line 51, in main
    with unpacked_tarball(sdist_file) as tmpdir:
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/build.py", line 24, in unpacked_tarball
    tf.extractall(tmpdir)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tarfile.py", line 2000, in extractall
    numeric_owner=numeric_owner)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tarfile.py", line 2042, in extract
    numeric_owner=numeric_owner)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tarfile.py", line 2104, in _extract_member
    os.makedirs(upperdirs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmp6jcs8l04/sage_conf-9.2b0/src'

mkoeppe · 2020-06-14T05:00:45Z

comment:13

A big part of the issues with symlinks comes, of course, from the behavior of pip, which copies the local source directory to a temporary location, breaking symlinks in the process. Various unresolved pip issues talk about finding a solution for local builds, most recently pypa/pip#7555 (earlier issues: pypa/pip#2195 and within, pypa/pip#7882).

mkoeppe · 2021-02-13T20:51:01Z

comment:15

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

mkoeppe added this to the sage-9.2 milestone Jun 13, 2020

mkoeppe added c: build labels Jun 13, 2020

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Install sage_conf using flit~~ Install sage_conf using flit instead of setuptools Jun 13, 2020

This comment has been minimized.

Sign in to view

mkoeppe modified the milestones: sage-9.2, sage-9.3 Jun 14, 2020

mkoeppe modified the milestones: sage-9.3, sage-9.4 Feb 13, 2021

mkoeppe removed this from the sage-9.4 milestone Jul 19, 2021

mkoeppe added this to the sage-9.5 milestone Jul 19, 2021

mkoeppe modified the milestones: sage-9.5, sage-9.6 Dec 14, 2021

mkoeppe removed this from the sage-9.6 milestone Jan 23, 2022

mkoeppe added the wishlist item label Jan 23, 2022

mkoeppe closed this as not planned Won't fix, can't repro, duplicate, stale Dec 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Install sage_conf using flit instead of setuptools #29854

Install sage_conf using flit instead of setuptools #29854

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

sagetrac-git mannequin commented Jun 13, 2020

sagetrac-git mannequin commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

sagetrac-git mannequin commented Jun 13, 2020

sagetrac-git mannequin commented Jun 13, 2020

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Jun 13, 2020

sagetrac-git mannequin commented Jun 13, 2020

sagetrac-git mannequin commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 14, 2020

mkoeppe commented Feb 13, 2021

Install sage_conf using flit instead of setuptools #29854

Install sage_conf using flit instead of setuptools #29854

Comments

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

sagetrac-git mannequin commented Jun 13, 2020

sagetrac-git mannequin commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

sagetrac-git mannequin commented Jun 13, 2020

sagetrac-git mannequin commented Jun 13, 2020

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Jun 13, 2020

sagetrac-git mannequin commented Jun 13, 2020

sagetrac-git mannequin commented Jun 13, 2020

mkoeppe commented Jun 13, 2020

mkoeppe commented Jun 14, 2020

mkoeppe commented Feb 13, 2021