Do not stage .pc files in src/lib/, clean old generated *.pc files at 'make distclean', fix 'permission denied' errors #29082

dimpase · 2020-01-26T12:40:13Z

#29003 introduced 2-stage installation of generated *.pc files,
via src/lib/pkgconfig- but did not provide a way to clean them up at all. And this is a problem (cf. e.g. #29071).

Also, in some installations, repeated installations this code leads to 'permission denied' errors when trying to overwrite read-only files:

  [openblas-0.3.6.p0]   Wrote /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-minimal/local/var/tmp/sage/build/openblas-0.3.6.p0/inst/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-minimal/local/lib/pkgconfig/blas.pc
  [openblas-0.3.6.p0]   Wrote /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-minimal/local/var/tmp/sage/build/openblas-0.3.6.p0/inst/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-minimal/local/lib/pkgconfig/cblas.pc
  [openblas-0.3.6.p0]   Wrote /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-minimal/local/var/tmp/sage/build/openblas-0.3.6.p0/inst/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-minimal/local/lib/pkgconfig/lapack.pc
  [openblas-0.3.6.p0]   
  [openblas-0.3.6.p0]   real	8m40.259s
  [openblas-0.3.6.p0]   user	45m2.739s
  [openblas-0.3.6.p0]   sys	5m48.754s
  [openblas-0.3.6.p0]   Copying package files from temporary location /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-minimal/local/var/tmp/sage/build/openblas-0.3.6.p0/inst to /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-minimal/local
  [openblas-0.3.6.p0]   cp: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-minimal/local/./lib/pkgconfig/cblas.pc: Permission denied
  [openblas-0.3.6.p0]   cp: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-minimal/local/./lib/pkgconfig/blas.pc: Permission denied
  [openblas-0.3.6.p0]   cp: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-minimal/local/./lib/pkgconfig/lapack.pc: Permission denied

This ticket revises the installation as follows:

configure no longer creates .pc files in the build tree
Instead it creates rules in make/Makefile that create the .pc files in SAGE_LOCAL.
Before pc files are installed into SAGE_LOCAL, the targets are removed, to avoid permission problems

This ticket does not solve everything. There may still be problems when switching from a system library to an spkg-provided library. We will use the follow-up ticket #29387 (Complete solution for installing the generated *.pc files) to address this problem.

CC: @embray @mkoeppe @jhpalmieri @vbraun @kiwifb

Component: build: configure

Author: Matthias Koeppe

Branch/Commit: 329843b

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29082

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-01-28T00:43:23Z

comment:2

My suggestion would be to rewrite this as a script package instead of putting things into src/. This stuff really does not belong into src/.

mkoeppe · 2020-01-28T01:00:14Z

comment:3

Added #29071 as a dependency as it touches the same files.

mkoeppe · 2020-01-28T01:00:14Z

Dependencies: #29071

mkoeppe · 2020-01-28T01:06:32Z

comment:5

(deleted a comment that was intended for #29071.)

mkoeppe · 2020-01-28T01:19:20Z

comment:6

(deleted a comment that was intended for #29071.)

dimpase · 2020-01-28T09:03:08Z

comment:7

Replying to @mkoeppe:

My suggestion would be to rewrite this as a script package instead of putting things into src/. This stuff really does not belong into src/.

Perhaps we should just create var/ next to src/ and put these things there?

mkoeppe · 2020-01-28T12:22:24Z

comment:8

No, I don't think so. These are configure-generated files of sage-the-distribution - like build/make/Makefile. There's no need for a new top level.

Let me take care of this ticket after #29071 is done.

mkoeppe · 2020-01-29T00:31:17Z

Changed dependencies from #29071 to #29051, #29071, #29084

mkoeppe · 2020-01-29T02:02:08Z

Author: Matthias Koeppe

mkoeppe · 2020-01-29T04:19:06Z

Branch: u/mkoeppe/move__pc_file_from_src__to_build___clean_generated___pc_files_at__make_distclean_

mkoeppe · 2020-01-29T04:26:34Z

Last 10 new commits:

`683a926`	`make installing *.pc files conditional`
`a0acbcb`	`correct quoting of m4 index variable`
`5ee6844`	`ensure gfortran is available`
`56541ae`	`wrap AC_FC_FUNC so that it does not throw an error without Fortran`
`9d1770a`	`revert #29025, remove useless [ALL] section`
`dba7aef`	`revert from "libraries=" back to section-specific "BLAH_libs="`
`3a4524e`	`Merge remote-tracking branch 'trac/public/packages/numpy/no_DEFAULT_and_ALL_numpy_site_cfg' into openblaspcfix`
`06f46eb`	`Add blas to fflas_ffpack linked libraries so that openblas is picked up on Arch`
`5f16988`	`Merge branch 'u/arojas/make_fflas_ffpack_detect_and_use_system_openblas_on_arch' of git://trac.sagemath.org/sage into t/29082/move__pc_file_from_src__to_build___clean_generated___pc_files_at__make_distclean_`
`2a01db1`	`Move all BLAS PC file installation logic to new script package sage_sage_system_blas_facade`

mkoeppe · 2020-01-29T04:26:34Z

Commit: 2a01db1

mkoeppe · 2020-01-29T04:27:05Z

comment:13

Branch is on top of #29051, #29071, #29084.
This is a first version, haven't tested much yet.

mkoeppe · 2020-01-29T04:31:04Z

comment:14

Tests run at https://github.com/mkoeppe/sage/actions/runs/32449719

mkoeppe · 2020-01-30T01:37:22Z

Changed dependencies from #29051, #29071, #29084 to #29051, #29071, #29084, #26130, #29056

mkoeppe · 2020-01-31T15:59:35Z

comment:16

I will redo this on top of the hot fix - #29121.

mkoeppe · 2020-03-22T02:13:23Z

comment:56

Error with gsl.pc for local-homebrew-macos-standard - https://github.com/mkoeppe/sage/runs/524361197

mkoeppe · 2020-03-22T02:15:04Z

comment:57

also ubuntu-bionic-standard (sagelib and cvxopt)

sagetrac-git · 2020-03-22T02:59:08Z

Branch pushed to git repo; I updated commit sha1. New commits:

`2a38a16`	`build/pkgs/gsl/spkg-configure.m4: Quoting fix`

sagetrac-git · 2020-03-22T02:59:08Z

Changed commit from f5b4f62 to 2a38a16

mkoeppe · 2020-03-22T02:59:45Z

comment:60

Tests run at https://github.com/mkoeppe/sage/actions/runs/60588138

dimpase · 2020-03-22T09:10:02Z

Reviewer: Dima Pasechnik

dimpase · 2020-03-22T09:10:02Z

comment:62

OK, this does the needed job. Perhaps the make rules could be made less explicit, but OK.

mkoeppe · 2020-03-22T09:27:10Z

comment:63

Thanks!

vbraun · 2020-03-23T23:22:26Z

comment:65

Make distclean deletes src/lib/pkgconfig/.gitignore

sagetrac-git · 2020-03-23T23:26:54Z

Branch pushed to git repo; I updated commit sha1. New commits:

`329843b`	`Ignore /src/lib/pkgconfig using top-level .gitignore`

sagetrac-git · 2020-03-23T23:26:54Z

Changed commit from 2a38a16 to 329843b

dimpase · 2020-03-24T03:32:11Z

comment:68

ok. sorry for overlooking this.

mkoeppe · 2020-03-24T03:33:21Z

comment:69

Thanks

vbraun · 2020-03-29T00:24:26Z

Changed branch from u/mkoeppe/move__pc_file_from_src__to_build___clean_generated___pc_files_at__make_distclean_ to 329843b

dimpase added this to the sage-9.1 milestone Jan 26, 2020

dimpase added c: build: configure labels Jan 26, 2020

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~clean generated *.pc files~~ Move .pc file from src/ to build/, clean generated *.pc files at 'make distclean' Jan 28, 2020

mkoeppe added s: needs work and removed s: needs info labels Jan 28, 2020

This comment has been minimized.

Sign in to view

mkoeppe added p: major / 3 and removed p: blocker / 1 labels Jan 31, 2020

This comment has been minimized.

Sign in to view

mkoeppe added s: needs work and removed s: needs review labels Mar 22, 2020

mkoeppe added s: needs review and removed s: needs work labels Mar 22, 2020

This comment has been minimized.

Sign in to view

dimpase added s: positive review and removed s: needs review labels Mar 22, 2020

mkoeppe changed the title ~~Move .pc file from src/ to build/, clean generated *.pc files at 'make distclean', fix 'permission denied' errors~~ Do not stage .pc files in src/lib/, clean old generated *.pc files at 'make distclean', fix 'permission denied' errors Mar 22, 2020

vbraun added s: needs work and removed s: positive review labels Mar 23, 2020

mkoeppe added s: needs review and removed s: needs work labels Mar 23, 2020

dimpase added s: positive review and removed s: needs review labels Mar 24, 2020

vbraun removed the s: positive review label Mar 29, 2020

vbraun closed this as completed in d7db290 Mar 29, 2020

Do not stage .pc files in src/lib/, clean old generated *.pc files at 'make distclean', fix 'permission denied' errors #29082

Do not stage .pc files in src/lib/, clean old generated *.pc files at 'make distclean', fix 'permission denied' errors #29082

Comments

dimpase commented Jan 26, 2020

mkoeppe commented Jan 28, 2020

mkoeppe commented Jan 28, 2020

mkoeppe commented Jan 28, 2020

This comment has been minimized.

mkoeppe commented Jan 28, 2020

mkoeppe commented Jan 28, 2020

dimpase commented Jan 28, 2020

This comment has been minimized.

mkoeppe commented Jan 28, 2020

mkoeppe commented Jan 29, 2020

mkoeppe commented Jan 29, 2020

mkoeppe commented Jan 29, 2020

This comment has been minimized.

mkoeppe commented Jan 29, 2020

mkoeppe commented Jan 29, 2020

mkoeppe commented Jan 29, 2020

mkoeppe commented Jan 29, 2020

mkoeppe commented Jan 30, 2020

mkoeppe commented Jan 31, 2020

This comment has been minimized.

mkoeppe commented Mar 22, 2020

mkoeppe commented Mar 22, 2020

sagetrac-git mannequin commented Mar 22, 2020

sagetrac-git mannequin commented Mar 22, 2020

mkoeppe commented Mar 22, 2020

This comment has been minimized.

dimpase commented Mar 22, 2020

dimpase commented Mar 22, 2020

mkoeppe commented Mar 22, 2020

vbraun commented Mar 23, 2020

sagetrac-git mannequin commented Mar 23, 2020

sagetrac-git mannequin commented Mar 23, 2020

dimpase commented Mar 24, 2020

mkoeppe commented Mar 24, 2020

vbraun commented Mar 29, 2020