Make Sage relocatable

[culler]

The relocate-once.py paradigm for installing Sage is annoying, inconvenient and unnecessary. Why not get rid of it?

Currently the bash script that starts sage checks for the
script relocate-once.py and if it is found, runs it. The script deletes itself when it finishes.

What the relocation script does is to replace every instance of a certain path containing a uuid by a new destination path for a long list of binary and text files. The substitution looks like this:

SearchAndReplace(ROOT_PATH, '/var/lib/buildbot/slave/binary_pkg/build/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas391y', DESTINATION)

A simple comnon-sense replacement for this paradigm which would make Sage relocatable is the following. There are
three parts.

1. For the DESTINATION use '/var/tmp/sage/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas391y

2. Run the relocation script at build time, before packaging Sage, so the user doesn't have to watch the sausage being made.

3. Modify the sage bash script so that it first creates a symlink:

   /var/tmp/sage/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas391y -> $SAGE_ROOT

   when it starts up, before it actually runs sage. The bash script would need to be created at build time from a template, since the command for creating the symlink contains the release uuid.

CC: @vbraun @dimpase @kcrisman @slel @NathanDunfield

Component: relocation

Author: Marc Culler

Issue created by migration from https://trac.sagemath.org/ticket/31076

[culler]

Attachment: relocate.patch.gz

[culler]

comment:2

I am attaching a small patch that implements this suggestion as a proof of concept.
It works - I was able to relocate sage and have it still work.
There is one tiny issue which would need to be addressed to finish this - because of the symlink,
when you start sage it warns that the SAGE_ROOT variable is being overwritten. Those warnings
should be suppressed, at least in the case where the known symlink is being replaced by its
canonical path.

[mkoeppe]

comment:3

We could add something like this as a configure option such as ./configure --enable-relocatable (to set SAGE_LOCAL to /var/tmp/sage-$UUID) - then relocatability would not have to rely on https://github.com/sagemath/binary-pkg at all.
The UUID would be generated by configure and stored in a file in $prefix/var/lib/sage/,
and the sage script and also the build scripts would regenerate the symlink $SAGE_LOCAL -> $prefix whenever needed.

[vbraun]

comment:4

What about multiuser systems, now only a single person can use the binary? Not saying that this use case needs to be 100% supported, just needs to be discussed.

You would also need to check file ownership, otherwise it becomes trivial to trick other users into running your binaries.

Really we should be pushing users towards containers, especially since rootless containers are now possible.

[culler]

comment:5

Thanks for considering this ticket.

I am writing another patch which handles this slightly differently. I think it should deal with the multi-user situation. Here is the idea. The buildbot installs a RUNPATH containing a uuid. That uuid should be viewed as an identifier for the build. When that build is installed, i.e. when its relocate-once.py is run, the path gets replaced with a path in /var/tmp (because /var/tmp is persistent across reboots) which contains a different uuid that identifies the installation as opposed to the build. When sage starts up, the installation-specific path becomes a symlink pointing to the installation root, as deduced from the args passed to the startup script. The symlink should only be writable by its owner. The startup script should ignore errors that occur because the symlink already exists and is not writable. I think that would allow a system administrator to install sage and also to relocate it if needed. Other users would be able to use that installation, but they could not relocate it or redirect it to a nefarious secret installation.

[mkoeppe]

comment:6

Replying to @vbraun:

What about multiuser systems, now only a single person can use the binary? Not saying that this use case needs to be 100% supported, just needs to be discussed.

Yes, this is certainly a constraint that would need to be documented.

[culler]

comment:7

Here is one more issue that occurs to me, but seems not to be too problematic. If someone wants to rebuild some component of sage, then the build should use the installation-specific RUNPATH. I am planning to save that path inside the installation as a (write-protected) 1-line bash script which can be sourced by the startup script. That will also make it available to any build script.

[culler]

comment:8

Attachment: relocate2.patch.gz

Here is the new version of the patch (relocate2.patch), which I think should deal with the multi-user situation.

To test:

$ tar xvfj path/to/sage-9.2-Ubuntu_20.04-x86_64.tar.bz2
$ cd SageMath ; patch -p1 < path/to/relocate2.patch ; cd ..
$ SageMath/sage
$ mv SageMath Sage-9.2
$ Sage-9.2/sage

[vbraun]

comment:9

I can give write permissions to anyone, thats not safe

[culler]

comment:10

Of course. Any superuser can give write permission on anything to anyone. And any user can create world-writable files or directories and world-executable programs. What does that have to do with anything? That is inherent to unix. The question is what happens when sage is installed for use by all users by a special sage user (not the root user) and the sage user follows the default installation procedure which does not give write permission on the symlink to any other user.

(The symlink is the only new thing here so any security issue thst does not involve the symlink is already present, independent from this proposal.)

[culler]

comment:11

Hi Volker, I think I now see what you are getting at. The scheme that I was proposing requires that /var/tmp/sage be writable. Even though the symlink itself is not writable, this allows malicious Mallory to rename /var/tmp/sage and replace it with a nefarious substitute. Thanks for pointing that out.

So this is what I will do instead. I will check if a user is installing sage below his or her home directory and if so put the symlink in the user's .sage directory. If not, I will revert to the old behavior so multi-user installations will not be relocatble. How is that?

[mkoeppe]

comment:12

You could get around the problem by changing /var/tmp/sage/%s to /var/tmp/sage-%s. /var/tmp is sticky and so, once created by a user, it cannot be removed or moved by another user

[vbraun]

comment:13

The only way without race conditions is:

• create /var/tmp/$DIR, ignoring EEXISTS
• check that I am the owner of /var/tmp/$DIR
  You should also assume that /var/tmp/ is periodically cleaned so this needs to run during each startup.

[culler]

Attachment: relocate3.patch.gz

[culler]

comment:14

The latest attachment, relocate3.patch, attempts to handle this as simply as possible without using /var/tmp.

It does not check whether sage is being installed in a user's home directory. Instead, it simply asks the user whether to make sage relocatable, and warns against doing that for a multi-user installation. (I actually like to install sage outside of my home directory, in another directory that I own, because installing it in my home directory makes find and locate difficult for me to use.)

If the user answers yes, then the symlink is used as before, but it lives in ~/.sage/locations instead of in /var/tmp/sage. A user can have more than one sage installation (as I usually do).

If the user answers no, then the relocate-once.py script is run as it currently is, hard-wiring all of the paths to the installation directory, once and for all.

[mkoeppe]

comment:15

I don't know, I was interested in the earlier iteration of this idea exactly because it would allow to avoid patching the distributed binaries - which would open a path to codesigned binaries, which is in particular relevant for macOS.

[vbraun]

comment:16

Expanding ~ is a shell convention and is not done on the POSIX API level. While you can find binaries in the path that way, the paths to shared libraries and hardcoded data files will point to the old location. Thats why a symlink in ~/.sage/locations cannot help. If it works you coincidentally have libraries and data files in the hardcoded location.

[culler]

comment:17

Replying to @mkoeppe:

I don't know, I was interested in the earlier iteration of this idea exactly because it would allow to avoid patching the distributed binaries - which would open a path to codesigned binaries, which is in particular relevant for macOS.

Code signing for macOS basically only makes sense for bundles - either Application bundles or Framework bundles. Apple's version of the rpath, which you can see with otool -L, can be made relative to the path of a bundle. Instead of using the relocate-once.py script when installing sage you could either use the install-name-tool command after the build or you could have the clang linker install a relative path when it does the linking.

In the case of sage, I think the most natural thing to do would probably be to make sage be a Framework which is intended to be installed in /Library/Frameworks from a .pkg file. If you were to do that then you could use an absolute path like /Library/Frameworks/Sage.framework/Versions/9.2 as the install library path. The current SageMath directory could be placed in the bundle as a Resources directory or as a subdirectory of one. The .pkg file could also be setup to install a symlink in /usr/local/bin pointing to the sage bash script.

[culler]

comment:18

Replying to @vbraun:

Expanding ~ is a shell convention and is not done on the POSIX API level. While you can find binaries in the path that way, the paths to shared libraries and hardcoded data files will point to the old location. Thats why a symlink in ~/.sage/locations cannot help. If it works you coincidentally have libraries and data files in the hardcoded location.

The modified relocate-once.py script does not expand ~. It uses the $HOME environment variable to find ~/.sage. If that environment variable does not exist, the installation aborts. The path that gets used as the runpath (i.e. the path to the symlink which gets set by the sage script) is an absolute path, for me anyway, since $HOME is an absolute path by default on Ubuntu.

I don't really understand the complaint about using the shell. As shipped, sage is a bash script. How could a user run that without having a bash-compatible shell?

Also, my patch does work. You should try it. I do not coincidentally have libraries and data files in the hardcoded location. The libraries and data files are in the hardcoded location by design. That is because the hardcoded location is a symlink to the actual location of the libraries and data files, and it automatically gets updated by the sage script whenever it is run. The directory containing the sage script is the root of the sage installation so the sage script knows how to update the symlink.

[culler]

comment:20

Here is my attempt to summarize what we have learned here.

First, it has been demonstrated that sage can be made relocatable by having relocate-once.py substitute a path to a symlink for the runpath created by the buildbot. All three attached patches do this successfully. The remaining issue is to decide what the pathname of the symlink should be. Should the filename of the symlink include the uuid associated to the build or a new uuid associated to a specific installation? Should the symlink be in a /var/tmp directory or in a ~/.sage directory?

There were security concerns raised about using /var/tmp but those seem to have disappeared once we remembered that /var/tmp is sticky. There may be other issues related to using /var/tmp, such as race conditions and the possibility that /var/tmp gets deleted even though it does persist through reboots.

An advantage of putting the symlink in /var/tmp and using the build uuid in the filename is that it allows the runpaths to be created when a sage release is packaged. But a problem with this is that the first user to install that release of sage on a given system will become the owner of the symlink. On a single user system that is not a problem. On a multi-user system it is a problem unless the system administrator is the first user to install sage.

Using an installation uuid is a partial solution, since each user who installs a given sage release would have a different uuid associated to their particular installation. But in that case it would make more sense to put the symlink in the user's $HOME/.sage directory.

Based on this, I suggest the following:

1. Distribute sage with the runpath set to /var/tmp/sage-.

2. If the relocate-once.py install script is run with effective uid 0, it should not change the runpath. It should just create the symlink and make it owned by root. The symlink does not need to change unless the system administrator moves sage, so race conditions are not an issue. If /var/tmp gets deleted or if sage is moved, the system administrator can fix sage by just changing or recreating the symlink, e.g. by running a relocation script provided for that purpose with the sage package. (This could possibly be the same relocate-once.py script which is preserved by moving, not removing, itself after it runs successfully. But it could also be written by the relocate-once.py script). A multi-user sage should refuse to run if the /var/tmp symlink is not owned by root.

3. If the install script is not being run with effective uid 0, assume that sage is being installed for use only by the user who is running the script. In that case, rewrite the runpaths to be a path in the user's ~/.sage directory, and update the symlink dynamically as is done in relocate3.patch. The uuid may as well be associated to a specific installation, as is done in that patch, because the user might well be a developer who wants to have more than one installation of the same sage release.

[culler]

comment:21

A warning about testing the patches attached here. On my Ubuntu system I have installed the python-is-python3 package, so the relocate-once.py script is run with python3. It does not work with python2 because it uses input. So if you do not have python-is-python3 you need to change the hashbang in that script to use python3.

[mkoeppe]

comment:22

I like this plan.

Changes to the build path set by the binary distribution and to the relocate-once.py script will have to be done with a pull request to
https://github.com/sagemath/binary-pkg, which generates that file.

I would suggest to use the present ticket for changes to Sage that implements the symlink manipulation that needs to be done after relocate-once.py is deleted, see comment 3 above.

[mkoeppe]

comment:23

sagemath/binary-pkg#27

[mkoeppe]

comment:24

Marc, some comments on your pull request.

1. What really needs the relocation trickery is not SAGE_ROOT, but rather SAGE_LOCAL (which defaults to SAGE_ROOT/local but can be configured to arbitrary paths). Let me call the symlink for that SAGE_LOCAL_SYMLINK. SAGE_ROOT itself holds very little state and can be easily "relocated" by just rerunning ./configure --prefix=SAGE_LOCAL_SYMLINK.

2. Because the SAGE_LOCAL prefix hierarchy contains all the files hardcoding SAGE_LOCAL, this is also where runpath.sh should be put, not SAGE_ROOT. I would suggest to use the location SAGE_LOCAL/var/lib/sage/.

3. In current Sage, after completing the build from source (=installation), SAGE_LOCAL is fully self-contained and SAGE_ROOT from this point of view is optional. In particular, invoking SAGE_LOCAL/bin/sage should work without having to go through the SAGE_ROOT/sage script. Hence, start up logic ("This sage installation has been moved!") should be done by a script that is installed in SAGE_LOCAL - such as src/bin/sage or src/bin/sage-env.

[vbraun]

comment:25

systemd-tempfiles deletes files in /var/tmp after 30 days (atime) on Fedora. I don't have a list of all the expiration dates on different distros but anything set up for multiuser access is going to clean tempfiles periodically since users tend to create them, often without noticing.

[culler]

comment:26

RE systemd-tempfiles

For an owner of a personal computer running linux this is not an issue because
if the link is missing it will be automatically recreated the next time that sage is started. It is also not an issue for "private" installations since the symlink is not in /var/tmp.

So this really only matters for true multi-user systems with a real administrator and multiple users who do not have administrator privileges. And in fact it only matters in the subcase when the administrator wants to install a common sage for all users instead of making each user install their own private version. In that subcase perhaps the best solution is to just install a non-relocatable hard-wired sage as is currently being distributed.

Yes, it might be a bit more convenient for the administrator of a big system to be able to move rather than reinstall sage. But that is probably a rare event and anyway I know from my own experience that nobody minds creating extra work for system administrators.

[culler]

comment:27

Replying to @mkoeppe:

Thank you for the comments, and the explanation. Although it is a different topic, I was confused by why there are multiple copies of some scripts, such as sage-env. It seems that the one which is actually used is src/bin/sage-env but there is an identical copy of that file named local/bin/sage-env which does not get used.

There is no problem with moving runpath.sh to local/var/lib/sage. But I can't tell if
there are any other changes that would be needed based on these comments. It sounds like the top level sage script could be eliminated entirely (and maybe should be for the standalone macOS app) but I don't think you are suggesting doing this in the PR.

Are you suggesting any changes besides moving runpath.sh?

[dimpase]

comment:28

systemd is not what everyone runs. Assuming it is available on every Linux box is too optimistic.

[mkoeppe]

comment:29

Replying to @culler:

I was confused by why there are multiple copies of some scripts, such as sage-env. It seems that the one which is actually used is src/bin/sage-env but there is an identical copy of that file named local/bin/sage-env which does not get used.

local (SAGE_LOCAL) is the installation tree.
If you invoke local/bin/sage directly, then also local/bin/sage-env is used, not src/bin/sage-env.
For application deployment, it is best to ignore SAGE_ROOT and the sage script there
and to use SAGE_LOCAL only. SAGE_ROOT is only needed for build tasks such as installing optional packages; and for source introspection.

[mkoeppe]

comment:30

Replying to @culler:

Are you suggesting any changes besides moving runpath.sh?

Yes, I would suggest to move the new relocation logic (or even all relocation logic if possible) from the top-level script sage into the script src/bin/sage (installed as local/bin/sage).

[culler]

comment:31

Replying to @dimpase:

systemd is not what everyone runs. Assuming it is available on every Linux box is too optimistic.

I think you misunderstood the comments.

Nobody was assuming that systemd would be available and certainly this proposal would not depend on it. The issue is that the relocation scheme does use a symlink that is located in /var/tmp and that link would periodically get deleted by systemd-tempfiles on those systems which do use systemd. The systemd was not a dependency, it was just a potential troublemaker.

My response was that this does not matter because it would only mean that when a user starts up sage they might be asked to type a password in order to recreate the link. (Or, if it was a private sage installation, the link would be in the user's home directory so it would not get deleted when /var/tmp is cleaned up.)

The only situation where recreating the link would be problematic is when the user does not have admin permissions on the system. I was suggesting that the current non-relocatable installation would be more appropriate in that case.

[mkoeppe]

comment:32

Replying to @mkoeppe:

Replying to @culler:

Are you suggesting any changes besides moving runpath.sh?

Yes, I would suggest to move the new relocation logic (or even all relocation logic if possible) from the top-level script sage into the script src/bin/sage (installed as local/bin/sage).

Wondering if we should use the existing hook src/bin/sage-location for this. See #31270