Problem with hard way in docker #26
Comments
|
Can you turn on debug mode and provide full logs? (Also, I see you're using a custom seccomp profile; is it the one from Dangerzone?) |
|
Hi!, yes seccomp is from Dangerzone Thanks! |
EtiennePerot
added a commit
that referenced
this issue
Oct 5, 2024
|
Thanks. Sadly I didn't really add enough debug information to the function (only the tool) when running in debug mode, whoops. In 46aa572 I have submitted a patch that adds this debug information. Please re-run it. This just adds debug information so I don't expect it to fix the problem, just to provide more complete logs. Looking at the Linux documentation for
|
|
Hi @EtiennePerot , first thanks for you time. |
|
Thanks, this at least isolates the issue. Can you run the other commands in the docker container, without
|
|
Hi, I was running docker without this command :) |
|
Hello, I am encountering a similar error message when hitting the button to run code after having set up the function, I suspect this is how OP started to look at the issue too.
Based on the previous conversations I have tried the same steps, and ended up with the same issues as OP. Getting to the last step, I have added the option
So adding |
|
Thanks all. Still unclear as to what could be causing this, but one possible avenue is that the container may not be running as UID 0. I see $ docker run --rm -it ubuntu cat /proc/self/status | grep Uid
Uid: 0 0 0 0However if I set a specific user, the output is more similar: $ docker run --rm -it --user=nobody ubuntu cat /proc/self/status | grep Uid
Uid: 65534 65534 65534 65534Do you perhaps have a I wrote the following script to print a whole lot of debug info that may be relevant. It is meant to run outside of a container, on the machine that is running the Docker daemon. Save it as a shell script (make sure to check the #!/bin/bash
set -x
# Edit this to match how you run Open WebUI, but:
# - Remove the `--restart always` option
# - Remove the `-d` (or `--detach`) option
# - Remove the `--name` option
# - Add `--rm -i`
# - If you have /home/ubuntu/seccomp.gvisor.json present, then replace `--security-opt=seccomp=unconfined` with `--security-opt=seccomp=/home/ubuntu/seccomp.gvisor.json`
docker_run=(
sudo
docker run
--mount type=bind,source=/sys/fs/cgroup,target=/sys/fs/cgroup,readonly=false
--security-opt=seccomp=unconfined
--security-opt=apparmor=unconfined
--security-opt=label=type:container_engine_t
-p 3000:8080
-e ENABLE_SIGNUP=false -e ENABLE_ADMIN_CHAT_ACCESS=false -e ENABLE_LOGIN_FORM=true -e DEFAULT_USER_ROLE=user -e OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true -e WEBUI_URL="https://xxxxxxx/" -e MICROSOFT_REDIRECT_URI="XXXXX" -e ENABLE_OAUTH_SIGNUP=true -e MICROSOFT_CLIENT_ID=XXXX -e MICROSOFT_CLIENT_SECRET=XXX -e MICROSOFT_CLIENT_TENANT_ID=XXX -e OPENAI_API_KEY=XX
-v open-webui:/app/backend/data
--rm -i
ghcr.io/open-webui/open-webui:main
)
header() {
set +x
echo '' >&2
echo "========" "$@" "========" >&2
set -x
}
header 'uname:'
uname -a
header 'Environment (user):'
env
header 'Environment (sudo):'
sudo env
header 'Docker command (user):'
which docker
if file "$(which docker)" | grep -i ascii; then
cat "$(which docker)"
fi
docker info
header 'Docker command (sudo):'
sudo which docker
if sudo file "$(sudo which docker)" | grep -i ascii; then
sudo cat "$(sudo which docker)"
fi
sudo docker info
header "Docker daemon processes:"
ps aux | grep dockerd
header "Docker daemon process status:"
for p in $(pidof dockerd); do
sudo cat "/proc/$p/status"
done
useful_configs=(
/etc/docker/daemon.json
/etc/subuid
/etc/subgid
/etc/lsb-release
/etc/os-release
"$HOME/.config/docker/daemon.json"
)
for f in "${useful_configs[@]}"; do
header "$f outside container:"
sudo cat "$f"
done
header 'Namespaces outside container:'
ls -l /proc/self/ns
header 'Namespaces outside container (sudo):'
sudo ls -l /proc/self/ns
header 'Namespaces inside container:'
"${docker_run[@]}" ls -l /proc/self/ns
useful_files=(
/proc/self/status
/proc/self/mountinfo
/proc/self/uid_map
/proc/sys/kernel/unprivileged_userns_clone
/proc/sys/kernel/unprivileged_userns_apparmor_policy
/proc/sys/user/max_user_namespaces
)
for f in "${useful_files[@]}"; do
header "$f outside container:"
cat "$f"
header "$f outside container with sudo:"
sudo cat "$f"
header "$f inside container:"
"${docker_run[@]}" cat "$f"
done
header 'Hello world in container with strace:'
strace -ff "${docker_run[@]}" echo 'hello world'
header 'In-container process as observed from outside:'
"${docker_run[@]}" bash -c 'sleep 14; echo asdfasdf' &
docker_pid="$!"
sleep 6
contained_psaux="$(sudo ps aux | grep -P 'sleep 14; echo asdfasdf' | grep -v grep)"
echo "$contained_psaux"
for p in $(echo "$contained_psaux" | awk '{print $2}'); do
sudo cat "/proc/$p/status"
sudo cat "/proc/$p/uid_map"
sudo cat "/proc/$p/gid_map"
sudo ls -l "/proc/$p/ns"
done
wait "$docker_pid"
header 'capsh outside container:'
capsh --print
header 'capsh outside container with sudo:'
sudo capsh --print
header 'capsh inside container:'
"${docker_run[@]}" bash -c 'apt-get update && apt-get install -y libcap2-bin && capsh --print'
header 'Downloading gVisor:'
wget -q "https://storage.googleapis.com/gvisor/releases/release/latest/$(uname -m)/runsc" -O "$HOME/runsc"
chmod 555 "$HOME/runsc"
header 'gVisor in rootless mode outside container:'
"$HOME/runsc" --rootless --network=none --ignore-cgroups=true --debug-log=/dev/stderr --host-settings=check do echo hi
header 'gVisor in rootful mode outside container:'
sudo "$HOME/runsc" --network=none --ignore-cgroups=true --debug-log=/dev/stderr --host-settings=check do echo hi
rm -f "$HOME/runsc"
header 'gVisor in rootless mode in container:'
"${docker_run[@]}" bash -c 'apt-get update && apt-get install -y wget && wget -q "https://storage.googleapis.com/gvisor/releases/release/latest/$(uname -m)/runsc" -O "$HOME/runsc" && chmod 555 "$HOME/runsc" && "$HOME/runsc" --rootless --network=none --ignore-cgroups=true --debug-log=/dev/stderr --host-settings=check do echo hi'
header 'gVisor in rootful mode in container:'
"${docker_run[@]}" bash -c 'apt-get update && apt-get install -y wget && wget -q "https://storage.googleapis.com/gvisor/releases/release/latest/$(uname -m)/runsc" -O "$HOME/runsc" && chmod 555 "$HOME/runsc" && "$HOME/runsc" --network=none --ignore-cgroups=true --debug-log=/dev/stderr --host-settings=check do echo hi'
header 'unshare with strace inside container:'
"${docker_run[@]}" bash -c 'apt-get update && apt-get install -y strace && strace -ff unshare --map-root-user cat /proc/self/status'
header 'Code execution outside container:'
wget -q 'https://raw.githubusercontent.com/EtiennePerot/safe-code-execution/refs/heads/master/open-webui/tools/run_code.py' -O /tmp/run_code.py
CODE_EVAL_VALVE_OVERRIDE_REQUIRE_RESOURCE_LIMITING=false python3 /tmp/run_code.py --use_sample_code --debug
header 'Code execution outside container with sudo:'
CODE_EVAL_VALVE_OVERRIDE_REQUIRE_RESOURCE_LIMITING=false sudo python3 /tmp/run_code.py --use_sample_code --debug
rm -f /tmp/run_code.py
header 'Code execution inside container:'
"${docker_run[@]}" bash -c 'apt-get update && apt-get install -y wget && wget -q "https://raw.githubusercontent.com/EtiennePerot/safe-code-execution/refs/heads/master/open-webui/tools/run_code.py" -O /tmp/run_code.py && CODE_EVAL_VALVE_OVERRIDE_REQUIRE_RESOURCE_LIMITING=false python3 /tmp/run_code.py --use_sample_code --debug' |
|
this is my running result on ubuntu 24.04.1
I already followed the document of docker to setup rootless mode to bypass the new ubuntu kernel limitations, but it seems that /proc is still not properly mounted. |
|
Thanks. I updated the script above with more files and after adding the |
|
Please have a look. But one thing to point out, before I set up rootless mode, I already experienced the exact same problem described by the other two users previously, so I look up online and someone point out a certain webpage saying that ubuntu 24.04 has increased the limit on docker and kernel itself, and someone on stack exchange suggested setting up rootless mode, so I followed the docker documentation to setup rootless mode. Here is the output |
|
Thanks @benhaotang for re-running the script. I recently updated the code (again) to no longer check for procfs being obstructed, because the latest version of But other than this procfs issue, I am still confused at why the error writing to
Can you link to the specific page? I'm curious what this limit in the Ubuntu 24.04 kernel is, even if that's not the issue causing this. Also, if I could reproduce this myself, it might be faster for me to debug. For example, if you are running all of this in a virtual machine, you could take a snapshot of the virtual machine (after removing any sensitive files) and upload it somewhere. Alternatively, I can try setting it up on a fresh Ubuntu install. Are you using Ubuntu 24.04 server, Ubuntu 24.04 desktop, or some Ubuntu 24.04 cloud image? I ask because they come with different default kernel configurations, so this might be relevant when trying to reproduce this issue. I added (If the output is too large, you can attach it as a text file to the GitHub comment.) |
|
Hi, I was not able to find the stack exchange webpage I was referring to, (I remember is about mounting /proc on ubuntu 24.04, but searching on google and my browser history turns out nothing:( strange and sorry), but I found out that docker's official website also talks about similar things when setting up rootless mode as of now:
I am using 24.04 desktop, I use it as my home machine and server, I just got this machine on prime day so it's a pretty fresh and new install of ubuntu so far. I re-run your script in both default mode and rootless mode, please find the log for both: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Running inside docker this command echo 'print("Hello world!")' | python3 run_code.py
I get echo 'print("Hello world!")' | python3 run_code.py
Event: {'type': 'status', 'data': {'status': 'in_progress', 'description': 'Checking for updates...', 'done': False}}
Event: {'type': 'status', 'data': {'status': 'in_progress', 'description': 'Checking for updates...', 'done': False}}
Event: {'type': 'status', 'data': {'status': 'in_progress', 'description': 'Checking messages for code blocks...', 'done': False}}
Event: {'type': 'status', 'data': {'status': 'in_progress', 'description': 'Checking messages for code blocks...', 'done': False}}
Event: {'type': 'status', 'data': {'status': 'in_progress', 'description': 'Checking if environment supports sandboxing...', 'done': False}}
Event: {'type': 'status', 'data': {'status': 'in_progress', 'description': 'Checking if environment supports sandboxing...', 'done': False}}
Event: {'type': 'status', 'data': {'status': 'in_progress', 'description': 'Initializing sandbox configuration...', 'done': False}}
Event: {'type': 'status', 'data': {'status': 'in_progress', 'description': 'Initializing sandbox configuration...', 'done': False}}
Event: {'type': 'status', 'data': {'status': 'in_progress', 'description': 'Running Python code in gVisor sandbox...', 'done': False}}
Event: {'type': 'status', 'data': {'status': 'in_progress', 'description': 'Running Python code in gVisor sandbox...', 'done': False}}
Event: {'type': 'status', 'data': {'status': 'error', 'description': 'Sandbox exception: OSError: OSError in write_uid_map (#1): [Errno 1] Operation not permitted (other attempts: OSError in write_uid_map (#2): [Errno 1] Operation not permitted; OSError in write_uid_map (#3): [Errno 1] Operation not permitted); unshare_user (euid=0 egid=0 pid=166 do_resource_limiting=True initial_cgroup_name=None codeeval_cgroup_name=None controllers=None)', 'done': True}}
{"status": "SANDBOX_ERROR", "output": "Sandbox exception: OSError: OSError in write_uid_map (#1): [Errno 1] Operation not permitted (other attempts: OSError in write_uid_map (#2): [Errno 1] Operation not permitted; OSError in write_uid_map (#3): [Errno 1] Operation not permitted); unshare_user (euid=0 egid=0 pid=166 do_resource_limiting=True initial_cgroup_name=None codeeval_cgroup_name=None controllers=None)"}
General information
Kernel information: Linux ip-172-31-2-39 6.8.0-1016-aws use an external library #17-Ubuntu SMP Mon Sep 2 13:48:07 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Runtime: Docker
If running in Docker:
docker runcommand: sudo docker run --security-opt seccomp=/home/ubuntu/seccomp.gvisor.json --mount type=bind,source=/sys/fs/cgroup,target=/sys/fs/cgroup,readonly=false --mount type=bind,source=/proc2,target=/proc2,readonly=false --security-opt apparmor=unconfined --security-opt label=type:container_engine_t -d -p 3000:8080 -e ENABLE_SIGNUP=false -e ENABLE_ADMIN_CHAT_ACCESS=false -e ENABLE_LOGIN_FORM=true -e DEFAULT_USER_ROLE=user -e OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true -e WEBUI_URL="https://XXXXXXX" -e MICROSOFT_REDIRECT_URI="XXXXX" -e ENABLE_OAUTH_SIGNUP=true -e MICROSOFT_CLIENT_ID=XXXX -e MICROSOFT_CLIENT_SECRET=XXX -e MICROSOFT_CLIENT_TENANT_ID=XXX -e OPENAI_API_KEY=XX -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:maindocker inspect openwebui_container_name_hereto find out]Debug logs
Additional context
Thanks!
The text was updated successfully, but these errors were encountered: