Support: Figure out why script's "current directory" fails to be read

[azriel91]

Problem Description

Hiya, I need help with figuring out why a script task works as my local user, but does not work when I run it through a CI runner.

I have a conformance task in a workspace repository root's Makefile.toml, which contains a script.

[tasks.conformance]
script = [
'''
echo hi
'''
]

The actual script content doesn't matter — when I run this as myself, I get the hi. My CI runner is my laptop, installed as its own gitlab-runner user.

When I run cargo make --no-workspace conformance as gitlab-runner, it fails with:

[cargo-make] INFO - cargo-make 0.10.5
[cargo-make] INFO - Using Build File: Makefile.toml
[cargo-make] INFO - Task: conformance
[cargo-make] INFO - Setting Up Env.
[cargo-make] INFO - Running Task: init
[cargo-make] INFO - Running Task: conformance
[cargo-make] ERROR - Error while executing command, unable to extract exit code.
[cargo-make] WARN - Build Failed.

Modifying cargo-make a little, I get this bit of information:

# ... elided (same as above)
[cargo-make] INFO - Running Task: conformance
[cargo-make] ERROR - Err(ScriptError { info: IOError(Error { repr: Os { code: 13, message: "Permission denied" } }) })
[cargo-make] WARN - Build Failed.

So the runner apparently hits "Permission denied" before executing the script. I traced this to [email protected]: runner.rs#L92, current_dir() is returning Err (EACCES).

Based on the Rust docs, source code, googling, and checking file permissions (I checked every parent directory along the working dir to see that it's ugo r+x), the more likely reason for the error is the working directory "does not exist", or is incorrect.

So, I tried figuring out what the working directory is.

$ cargo make --no-workspace -v conformance 2>&1 | grep -F 'orking direc'
[cargo-make] DEBUG - Changing working directory to: .
[cargo-make] DEBUG - Working directory changed to: .

Cargo make successfully changes it to ., and my mini experiment, compiled and run from the same repository successfully gets the current_dir():

use std::env;
use std::env::current_dir;
use std::path::Path;

fn main() {
    // let here = Path::new(".").canonicalize().unwrap();
    let here = Path::new(".");

    assert!(env::set_current_dir(&here).is_ok());
    println!("Successfully changed working directory to {}", here.display());
    match current_dir() {
        Ok(path) => println!("{}", path.display()),
        e @ Err(..) => println!("{:?}", e),
    };
}

Output:

Successfully changed working directory to .
/home/gitlab-runner/builds/81b47ec8/0/azriel91/autexousious

This is where I ran out of ideas 😭

[azriel91]

Not sure if this helps, Shell / Rust version:

Shell:
GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Cargo:
cargo 0.26.0-nightly (1d6dfea44 2018-01-26)
release: 0.26.0
commit-hash: 1d6dfea44f97199d5d5c177c7dadcde393eaff9a
commit-date: 2018-01-26

Rustc:
rustc 1.26.0-nightly (2789b067d 2018-03-06)
binary: rustc
commit-hash: 2789b067da2ac921b86199bde21dd231ace1da39
commit-date: 2018-03-06
host: x86_64-unknown-linux-gnu
release: 1.26.0-nightly
LLVM version: 6.0

[sagiegurari]

I'll try to reproduce. Not sure if it will be that simple.

[sagiegurari]

can you explain more about the setup?
what happens if the project is under the runner home directory and not your user home directory?

[azriel91]

what happens if the project is under the runner home directory and not your user home directory?

yeap the problem occurred when the project is under the runner home directory — /home/gitlab-runner/builds/81b47ec8/0/azriel91/autexousious (ah, should have been clear which user I meant when I said "user")

---

... and of course when I tried to reproduce the issue today, the script task worked — using exactly the same directory and user as yesterday.

I don't think it's worth trying to reproduce this since it doesn't even happen on the same machine (within 24 hours), restarting must have fixed it.

I'll close this since it's transient and can't be reproduced. sad happy face 😶

Thank you for trying though, it's much appreciated 🙃

---

can you explain more about the setup?

Pretty much install gitlab-runner on linux and then point it at a workspace project with a Makefile.toml in the root. I'm not sure how that installation

Just for the record, these snippets may be useful in the future if it ever comes up again

Denied:

#! /bin/bash
mkdir test_denied
cd test_denied

cargo init --lib member
cat > Cargo.toml <<EOF
[workspace]
members = ["member"]
EOF

cat > Makefile.toml <<EOF
[tasks.conformance]
script = ["echo hi"]
EOF

cargo make --no-workspace conformance

Permissions:

#! /bin/bash
parts=(${PWD//\// })

cd /
for part in ${parts[@]}; do
    echo $(stat -c "%A %U" .) $(pwd)
    cd $part
done

echo $(stat -c "%A %U" .) $(pwd)

apt history didn't show anything related to gitlab-runner around 2018-03-17 NZT (when the issue happened):

$ rg -C3 -F 'gitlab-runner' /var/log/apt/
/var/log/apt/term.log
280-Log ended: 2018-03-10  11:37:26
281-
282-Log started: 2018-03-10  13:06:31
283:Selecting previously unselected package gitlab-runner.
(Reading database ... 253478 files and directories currently installed.)
285:Preparing to unpack .../gitlab-runner_10.5.0_amd64.deb ...
286:Unpacking gitlab-runner (10.5.0) ...
287:Setting up gitlab-runner (10.5.0) ...
288:GitLab Runner: creating gitlab-runner...
289:gitlab-runner: Service is not running.
290-gitlab-ci-multi-runner: Service is not running.
291-Log ended: 2018-03-10  13:06:35
292-

/var/log/apt/history.log
73-End-Date: 2018-03-10  11:37:26
74-
75-Start-Date: 2018-03-10  13:06:31
76:Commandline: apt install gitlab-runner
77-Requested-By: azriel (1000)
78:Install: gitlab-runner:amd64 (10.5.0)
79-End-Date: 2018-03-10  13:06:35
80-
81-Start-Date: 2018-03-10  16:52:37

[azriel91]

Hey, this happened twice since I closed it, I'll reopen in case it's a cargo-make issue, but shall investigate it again now. I haven't got solid clues to what's causing it, but it may be to do with rust nightly-2018-03-16 (latest nightly breaks some cargo subcommands, including cargo-make).

It's happened on both my regular azriel user and gitlab-runner so I'm not going down the "gitlab-runner may be broken" path yet, but we'll see.

[sagiegurari]

Latest cargo and rust are both broken.
I use nightly of first of February

[sagiegurari]

cargo doesn't support sub commands critical issue: rust-lang/cargo#5212
rust nightly compilation fails critical issue: rust-lang/rust#49113

So i suggest you use rustup and default it to nightly of 2018-02-01 which works good.

[azriel91]

Yeaps I've juust tried nightly-2018-02-01. It compiles and runs (I can run --version), but the permissions issue on current_dir() still happens when running a task containing a script. Hoping I find a proper lead soon; I haven't yet

[sagiegurari]

ok, please keep me updated. if you find an issue in cargo-make itself and would like to create a PR, I would much appreciate it.

[azriel91]

Traced it! Literally, straced it. Thanks to @zomerfaun, who suggested using the tool.

There's a bug in run_script when multiple users run tasks that contain scripts.

• run_script edits the script and places the edited script in a directory in /tmp/run_script
• /tmp has the sticky bit set, so a second user is unable to write to that directory, even if writing to a different file within the directory.

Since my laptop is used by me as well as gitlab-runner, when I run the conformance task as myself, the gitlab-runner can no longer run it (and the other way around, if I push a commit before conformance myself).

Fix is to add random characters to the temporary directory itself. Perhaps we should make it a temporary directory, either using tempdir or tempfile (which actually contains tempdir, just pending Stebalien/tempfile#43 to release it).

[sagiegurari]

you mean if I do /tmp/run_script/123/somescript.sh where 123 is random number everytime, it should resolve it?
Sounds like a rust bug to me.

[azriel91]

nope, like

• user1 creates /tmp/run_script/
• user1 creates /tmp/run_script/123.sh
• user2 tries to create /tmp/run_script/234.sh, you get permission denied, because /tmp/run_script/ belongs to user1

[sagiegurari]

ok, i got it. maybe I could add username in there /tmp/user/run_script/123.sh
that should resolve it I hope

[azriel91]

Yeaps that should 👍

[sagiegurari]

I'll handle it now

[sagiegurari]

I'm going to publish a new version that would ensure the minimal version of run_script has the fix.
however if you reinstall cargo-make it should take the latest run_script which has the fix so you could verify it.
you can make sure that when installing, it downloads run_script version 0.1.14

[sagiegurari]

new version published. can you please verify?

[azriel91]

Yeaps it works! 🎉 Thank you!

[sagiegurari]

Thank you for finding the root cause 👍