Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cant make Criu dump work on a GCP node #1230

Closed
mikosa opened this issue Oct 13, 2020 · 7 comments
Closed

Cant make Criu dump work on a GCP node #1230

mikosa opened this issue Oct 13, 2020 · 7 comments

Comments

@mikosa
Copy link

mikosa commented Oct 13, 2020

Hi I tried different linux flavors on GCP, and cant make CRIU work

For example if i try an ubuntu machine:
image

bootstrap code:

wget https://launchpad.net/ubuntu/+archive/primary/+sourcefiles/criu/3.14-1/criu_3.14.orig.tar.xz
tar xvfJ criu_3.14.orig.tar.xz

sudo apt-get install -y build-essential asciidoc debhelper-compat dh-python libbsd-dev libcap-dev libgnutls28-dev libnet1-dev libnl-3-dev libprotobuf-c-dev libprotobuf-dev pkg-config protobuf-c-compiler protobuf-compiler python3-all xmlto --no-install-recommends

cd criu-3.14
sudo make install

then I get the mount error

image

I also tried to install another version of criu 3.13, same error.

If I try a debian machine, I get similar problem.

Is there a problem with my bootstrap script?
How can i make it work?
@adrianreber
Copy link
Member

Which process are you trying to dump? I have never seen this error before myself or a report of it. We would need a full log file of the dump. Can you run it with -v4 and add the output here. Please do not use a screenshot, just copy the text.

@mikosa
Copy link
Author

mikosa commented Oct 13, 2020 

mike@gke-cluster-7-default-pool-94f35828-n541:~/criu-3.14$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS             
  NAMES
mike@gke-cluster-7-default-pool-94f35828-n541:~/criu-3.14$ sudo docker run -d grafana/grafana
Unable to find image 'grafana/grafana:latest' locally
latest: Pulling from grafana/grafana
df20fa9351a1: Pull complete 
5f101b91128e: Pull complete 
06b22e47f660: Pull complete 
e8278dc77f35: Pull complete 
4ac3c61790c9: Pull complete 
e5505e948ec3: Pull complete 
Digest: sha256:733842cca5bd9bcab1eb795da264863a8245402ff3ac8ff17e274334bb32c692
Status: Downloaded newer image for grafana/grafana:latest
ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32
mike@gke-cluster-7-default-pool-94f35828-n541:~/criu-3.14$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS             
  NAMES
ccacf2ccb758        grafana/grafana     "/run.sh"           10 seconds ago      Up 6 seconds        3000/tcp          
  nifty_visvesvaraya
mike@gke-cluster-7-default-pool-94f35828-n541:~/criu-3.14$ sudo docker top ccacf2ccb758
UID                 PID                 PPID                C                   STIME               TTY               
  TIME                CMD
472                 392365              392337              2                   11:53               ?                 
  00:00:00            grafana-server --homepath=/usr/share/grafana --config=/etc/grafana/grafana.ini --packaging=docke
r cfg:default.log.mode=console cfg:default.paths.data=/var/lib/grafana cfg:default.paths.logs=/var/log/grafana cfg:def
ault.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning
mike@gke-cluster-7-default-pool-94f35828-n541:~/criu-3.14$ sudo criu dump -t 392365 -v4
(00.000005) Version: 3.14 (gitid 0)
(00.000147) Running on gke-cluster-7-default-pool-94f35828-n541 Linux 5.4.0-1024-gcp #24~18.04.1-Ubuntu SMP Sun Sep 6 03:37:03 UTC 2020 x86_64
(00.000202) Loaded kdat cache from /run/criu.kdat
(00.000238) ========================================
(00.000271) Dumping processes (pid: 392365)
(00.000304) ========================================
(00.000632) rlimit: RLIMIT_NOFILE unlimited for self
(00.001916) Running pre-dump scripts
(00.002013) irmap: Searching irmap cache in work dir
(00.002078) No irmap-cache image
(00.002116) irmap: Searching irmap cache in parent
(00.002193) irmap: No irmap cache
(00.002248) cpu: x86_family 6 x86_vendor_id GenuineIntel x86_model_id Intel(R) Xeon(R) CPU @ 2.30GHz
(00.002289) cpu: fpu: xfeatures_mask 0x5 xsave_size 832 xsave_size_max 832 xsaves_size 0
(00.002347) cpu: fpu: x87 floating point registers     xstate_offsets      0 / 0      xstate_sizes    160 / 160   
(00.002399) cpu: fpu: AVX registers                    xstate_offsets    576 / 576    xstate_sizes    256 / 256   
(00.002434) cpu: fpu:1 fxsr:1 xsave:1 xsaveopt:1 xsavec:0 xgetbv1:0 xsaves:0
(00.002612) cg-prop: Parsing controller "cpu"
(00.002678) cg-prop:    Strategy "replace"
(00.002768) cg-prop:    Property "cpu.shares"
(00.002850) cg-prop:    Property "cpu.cfs_period_us"
(00.002912) cg-prop:    Property "cpu.cfs_quota_us"
(00.002974) cg-prop:    Property "cpu.rt_period_us"
(00.003069) cg-prop:    Property "cpu.rt_runtime_us"
(00.003132) cg-prop: Parsing controller "memory"
(00.003182) cg-prop:    Strategy "replace"
(00.003247) cg-prop:    Property "memory.limit_in_bytes"
(00.003333) cg-prop:    Property "memory.memsw.limit_in_bytes"
(00.003401) cg-prop:    Property "memory.swappiness"
(00.003483) cg-prop:    Property "memory.soft_limit_in_bytes"
(00.003568) cg-prop:    Property "memory.move_charge_at_immigrate"
(00.003647) cg-prop:    Property "memory.oom_control"
(00.003729) cg-prop:    Property "memory.use_hierarchy"
(00.003808) cg-prop:    Property "memory.kmem.limit_in_bytes"
(00.003955) cg-prop:    Property "memory.kmem.tcp.limit_in_bytes"
(00.004033) cg-prop: Parsing controller "cpuset"
(00.004182) cg-prop:    Strategy "replace"
(00.004296) cg-prop:    Property "cpuset.cpus"
(00.004373) cg-prop:    Property "cpuset.mems"
(00.004455) cg-prop:    Property "cpuset.memory_migrate"
(00.004531) cg-prop:    Property "cpuset.cpu_exclusive"
(00.004611) cg-prop:    Property "cpuset.mem_exclusive"
(00.004692) cg-prop:    Property "cpuset.mem_hardwall"
(00.004775) cg-prop:    Property "cpuset.memory_spread_page"
(00.004860) cg-prop:    Property "cpuset.memory_spread_slab"
(00.004936) cg-prop:    Property "cpuset.sched_load_balance"
(00.005014) cg-prop:    Property "cpuset.sched_relax_domain_level"
(00.005118) cg-prop: Parsing controller "blkio"
(00.005154) cg-prop:    Strategy "replace"
(00.005230) cg-prop:    Property "blkio.weight"
(00.005306) cg-prop: Parsing controller "freezer"
(00.005353) cg-prop:    Strategy "replace"
(00.005436) cg-prop: Parsing controller "perf_event"
(00.005469) cg-prop:    Strategy "replace"
(00.005555) cg-prop: Parsing controller "net_cls"
(00.005590) cg-prop:    Strategy "replace"
(00.005666) cg-prop:    Property "net_cls.classid"
(00.005741) cg-prop: Parsing controller "net_prio"
(00.005787) cg-prop:    Strategy "replace"
(00.005855) cg-prop:    Property "net_prio.ifpriomap"
(00.005945) cg-prop: Parsing controller "pids"
(00.005978) cg-prop:    Strategy "replace"
(00.006059) cg-prop:    Property "pids.max"
(00.006144) cg-prop: Parsing controller "devices"
(00.006189) cg-prop:    Strategy "replace"
(00.006266) cg-prop:    Property "devices.list"
(00.006409) Preparing image inventory (version 1)
(00.006484) Add pid ns 1 pid 393502
(00.006554) Add net ns 2 pid 393502
(00.006602) Add ipc ns 3 pid 393502
(00.006641) Add uts ns 4 pid 393502
(00.006694) Add time ns 5 pid 393502
(00.006747) Add mnt ns 6 pid 393502
(00.006787) Add user ns 7 pid 393502
(00.006835) Add cgroup ns 8 pid 393502
(00.006896) cg: Dumping cgroups for 393502
(00.006961) cg:  `- New css ID 1
(00.007013) cg:     `- [blkio] -> [/user.slice] [0]
(00.007072) cg:     `- [cpu,cpuacct] -> [/user.slice] [0]
(00.007106) cg:     `- [cpuset] -> [/] [0]
(00.007154) cg:     `- [devices] -> [/user.slice] [0]
(00.007201) cg:     `- [freezer] -> [/] [0]
(00.007234) cg:     `- [hugetlb] -> [/] [0]
(00.007281) cg:     `- [memory] -> [/user.slice] [0]
(00.007314) cg:     `- [name=systemd] -> [/user.slice/user-1002.slice/session-10.scope] [0]
(00.007351) cg:     `- [net_cls,net_prio] -> [/] [0]
(00.007398) cg:     `- [perf_event] -> [/] [0]
(00.007436) cg:     `- [pids] -> [/user.slice/user-1002.slice/session-10.scope] [0]
(00.007470) cg:     `- [rdma] -> [/] [0]
(00.007518) cg: Set 1 is criu one
(00.007568) Detected cgroup V1 freezer
(00.007727) Seized task 392365, state 1
(00.007795) seccomp: Collected tid_real 392365 mode 0x2
(00.007963)     Seizing 392365's 392412 thread
(00.008156) seccomp: Collected tid_real 392412 mode 0x2
(00.008218)     Seizing 392365's 392413 thread
(00.008338) seccomp: Collected tid_real 392413 mode 0x2
(00.008398)     Seizing 392365's 392414 thread
(00.008543) seccomp: Collected tid_real 392414 mode 0x2
(00.008589)     Seizing 392365's 392415 thread
(00.008701) seccomp: Collected tid_real 392415 mode 0x2
(00.008754)     Seizing 392365's 392422 thread
(00.008879) seccomp: Collected tid_real 392422 mode 0x2
(00.008941)     Seizing 392365's 392425 thread
(00.009068) seccomp: Collected tid_real 392425 mode 0x2
(00.009132)     Seizing 392365's 392426 thread
(00.009230) seccomp: Collected tid_real 392426 mode 0x2
(00.009334) Collected (3 attempts, 0 in_progress)
(00.009438) Collected (4 attempts, 0 in_progress)
(00.009509) Collected 392365 in 1 state
(00.009573) Will take pid namespace in the image
(00.009622) Add pid ns 9 pid 392365
(00.009674) Will take net namespace in the image
(00.009698) Add net ns 10 pid 392365
(00.009709) Will take ipc namespace in the image
(00.009714) Add ipc ns 11 pid 392365
(00.009756) Will take uts namespace in the image
(00.009779) Add uts ns 12 pid 392365
(00.009793) Will take mnt namespace in the image
(00.009831) Add mnt ns 13 pid 392365
(00.009847) Lock network
(00.009853) Running network-lock scripts
iptables-restore: invalid option -- 'w'
ip6tables-restore: invalid option -- 'w'
(00.014038)     type overlay source overlay mnt_id 3624 s_dev 0x1b0 / @ ./ flags 0x280000 options lowerdir=/var/lib/docker/overlay2/l/M2I7ZOC6HZQ2FBY36LX7LZV64H:/var/lib/docker/overlay2/l/WAJLYKXZV32WSRCWLZZHDJX3FB:/var/lib/docker/overlay2/l/ETAWUFFOJEGBNJZNO47WIIHFV5:/var/lib/docker/overlay2/l/UMOCT6OFQDT6GBIZLK5TLZMV3V:/var/lib/docker/overlay2/l/O5AT2JV2LEHM37WR4MYNJYRSYO:/var/lib/docker/overlay2/l/77BTK52Y6AJ6RNGRCMO7BNPZHW:/var/lib/docker/overlay2/l/MG6ITAOLVILTUENLGII3KE7RBH,upperdir=/var/lib/docker/overlay2/fd5004d5487c67c21807c7561ec9db57ae0d73667dea1cddb0a04c1d8e10ddd7/diff,workdir=/var/lib/docker/overlay2/fd5004d5487c67c21807c7561ec9db57ae0d73667dea1cddb0a04c1d8e10ddd7/work,xino=off
(00.014285)     type proc source proc mnt_id 3625 s_dev 0x1b2 / @ ./proc flags 0x20000e options 
(00.014357)     type tmpfs source tmpfs mnt_id 3626 s_dev 0x1b3 / @ ./dev flags 0x1000002 options size=65536k,mode=755
(00.014424)     type devpts source devpts mnt_id 3627 s_dev 0x1b4 / @ ./dev/pts flags 0x20000a options gid=5,mode=620,ptmxmode=666
(00.014464)     type sysfs source sysfs mnt_id 3628 s_dev 0x1b5 / @ ./sys flags 0x20000f options 
(00.014582)     type tmpfs source tmpfs mnt_id 3629 s_dev 0x1b6 / @ ./sys/fs/cgroup flags 0x20000f options mode=755
(00.014674)     type cgroup source cgroup mnt_id 3630 s_dev 0x1e /docker/ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32 @ ./sys/fs/cgroup/systemd flags 0x28000f options xattr,name=systemd
(00.014689)     type cgroup source cgroup mnt_id 3631 s_dev 0x21 /docker/ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32 @ ./sys/fs/cgroup/cpu,cpuacct flags 0x28000f options cpu,cpuacct
(00.014751)     type cgroup source cgroup mnt_id 3632 s_dev 0x22 /docker/ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32 @ ./sys/fs/cgroup/net_cls,net_prio flags 0x28000f options net_cls,net_prio
(00.014782)     type cgroup source cgroup mnt_id 3633 s_dev 0x23 /docker/ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32 @ ./sys/fs/cgroup/perf_event flags 0x28000f options perf_event
(00.014827)     type cgroup source cgroup mnt_id 3634 s_dev 0x24 / @ ./sys/fs/cgroup/rdma flags 0x28000f options rdma
(00.014866)     type cgroup source cgroup mnt_id 3635 s_dev 0x25 /docker/ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32 @ ./sys/fs/cgroup/freezer flags 0x28000f options freezer
(00.014916)     type cgroup source cgroup mnt_id 3636 s_dev 0x26 /docker/ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32 @ ./sys/fs/cgroup/blkio flags 0x28000f options blkio
(00.014929)     type cgroup source cgroup mnt_id 3637 s_dev 0x27 /docker/ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32 @ ./sys/fs/cgroup/devices flags 0x28000f options devices
(00.014973)     type cgroup source cgroup mnt_id 3638 s_dev 0x28 /docker/ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32 @ ./sys/fs/cgroup/memory flags 0x28000f options memory
(00.015018)     type cgroup source cgroup mnt_id 3639 s_dev 0x29 /docker/ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32 @ ./sys/fs/cgroup/pids flags 0x28000f options pids
(00.015034)     type cgroup source cgroup mnt_id 3640 s_dev 0x2a /docker/ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32 @ ./sys/fs/cgroup/cpuset flags 0x28000f options cpuset
(00.015116)     type cgroup source cgroup mnt_id 3641 s_dev 0x2b /docker/ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32 @ ./sys/fs/cgroup/hugetlb flags 0x28000f options hugetlb
(00.015181)     type mqueue source mqueue mnt_id 3642 s_dev 0x1b1 / @ ./dev/mqueue flags 0x20000e options 
(00.015227)     type tmpfs source shm mnt_id 3643 s_dev 0x1b7 / @ ./dev/shm flags 0x20000e options size=65536k
(00.015240)     type ext4 source /dev/sda1 mnt_id 3644 s_dev 0x800001 /var/lib/docker/containers/ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32/resolv.conf @ ./etc/resolv.conf flags 0x200000 options 
(00.015256)     type ext4 source /dev/sda1 mnt_id 3645 s_dev 0x800001 /var/lib/docker/containers/ccacf2ccb758dd734f1026cbc3b1ac37373163c5be91bd707dc090a11906ef32/hostname @ ./etc/hostname flags 0x200000 options 
(00.015301)     type ext4 source /dev/sda1 mnt_id 3646 s_dev 0x800001 /var/lib/docker/containers/ccacf2ccb758dd734f102
6cbc3b1ac37373163c5be91bd707dc090a11906ef32/hosts @ ./etc/hosts flags 0x200000 options 
(00.015331)     type proc source proc mnt_id 3281 s_dev 0x1b2 /bus @ ./proc/bus flags 0x200001 options 
(00.015342)     type proc source proc mnt_id 3282 s_dev 0x1b2 /fs @ ./proc/fs flags 0x200001 options 
(00.015383)     type proc source proc mnt_id 3283 s_dev 0x1b2 /irq @ ./proc/irq flags 0x200001 options 
(00.015412)     type proc source proc mnt_id 3284 s_dev 0x1b2 /sys @ ./proc/sys flags 0x200001 options 
(00.015467)     type proc source proc mnt_id 3285 s_dev 0x1b2 /sysrq-trigger @ ./proc/sysrq-trigger flags 0x200001 options 
(00.015542)     type tmpfs source tmpfs mnt_id 3286 s_dev 0x1b8 / @ ./proc/acpi flags 0x200001 options 
(00.015572)     type tmpfs source tmpfs mnt_id 3287 s_dev 0x1b3 /null @ ./proc/kcore flags 0x1000002 options size=65536k,mode=755
(00.015584)     type tmpfs source tmpfs mnt_id 3288 s_dev 0x1b3 /null @ ./proc/keys flags 0x1000002 options size=65536k,mode=755
(00.015612)     type tmpfs source tmpfs mnt_id 3289 s_dev 0x1b3 /null @ ./proc/timer_list flags 0x1000002 options size=65536k,mode=755
(00.015655)     type tmpfs source tmpfs mnt_id 3290 s_dev 0x1b3 /null @ ./proc/sched_debug flags 0x1000002 options size=65536k,mode=755
(00.015696)     type tmpfs source tmpfs mnt_id 3291 s_dev 0x1b9 / @ ./proc/scsi flags 0x200001 options 
(00.015708)     type tmpfs source tmpfs mnt_id 3292 s_dev 0x1ba / @ ./sys/firmware flags 0x200001 options 
(00.015761) mnt: Building mountpoints tree
(00.015818) mnt:        Building plain mount tree
(00.015874) mnt:                Working on 3292->3628
(00.015916) mnt:                Working on 3291->3625
(00.015926) mnt:                Working on 3290->3625
(00.015936) mnt:                Working on 3289->3625
(00.015946) mnt:                Working on 3288->3625
(00.015988) mnt:                Working on 3287->3625
(00.015998) mnt:                Working on 3286->3625
(00.016008) mnt:                Working on 3285->3625
(00.016018) mnt:                Working on 3284->3625
(00.016027) mnt:                Working on 3283->3625
(00.016037) mnt:                Working on 3282->3625
(00.016083) mnt:                Working on 3281->3625
(00.016094) mnt:                Working on 3646->3624
(00.016103) mnt:                Working on 3645->3624
(00.016113) mnt:                Working on 3644->3624
(00.016123) mnt:                Working on 3643->3626
(00.016169) mnt:                Working on 3642->3626
(00.016180) mnt:                Working on 3641->3629
(00.016205) mnt:                Working on 3640->3629
(00.016215) mnt:                Working on 3639->3629
(00.016252) mnt:                Working on 3638->3629
(00.016262) mnt:                Working on 3637->3629
(00.016271) mnt:                Working on 3636->3629
(00.016302) mnt:                Working on 3635->3629
(00.016312) mnt:                Working on 3634->3629
(00.016343) mnt:                Working on 3633->3629
(00.016353) mnt:                Working on 3632->3629
(00.016388) mnt:                Working on 3631->3629
(00.016415) mnt:                Working on 3630->3629
(00.016425) mnt:                Working on 3629->3628
(00.016460) mnt:                Working on 3628->3624
(00.016471) mnt:                Working on 3627->3626
(00.016480) mnt:                Working on 3626->3624
(00.016490) mnt:                Working on 3625->3624
(00.016521) mnt:                Working on 3624->3280
(00.016531) mnt:        Resorting children of 3624 in mount order
(00.016547) mnt:        Resorting children of 3646 in mount order
(00.016572) mnt:        Resorting children of 3645 in mount order
(00.016580) mnt:        Resorting children of 3644 in mount order
(00.016614) mnt:        Resorting children of 3628 in mount order
(00.016624) mnt:        Resorting children of 3629 in mount order
(00.016634) mnt:        Resorting children of 3641 in mount order
(00.016642) mnt:        Resorting children of 3640 in mount order
(00.016650) mnt:        Resorting children of 3639 in mount order
(00.016675) mnt:        Resorting children of 3638 in mount order
(00.016701) mnt:        Resorting children of 3637 in mount order
(00.016710) mnt:        Resorting children of 3636 in mount order
(00.016717) mnt:        Resorting children of 3635 in mount order
(00.016742) mnt:        Resorting children of 3634 in mount order
(00.016750) mnt:        Resorting children of 3633 in mount order
(00.016758) mnt:        Resorting children of 3632 in mount order
(00.016787) mnt:        Resorting children of 3631 in mount order
(00.016795) mnt:        Resorting children of 3630 in mount order
(00.016803) mnt:        Resorting children of 3292 in mount order
(00.016811) mnt:        Resorting children of 3626 in mount order
(00.016852) mnt:        Resorting children of 3643 in mount order
(00.016863) mnt:        Resorting children of 3642 in mount order
(00.016871) mnt:        Resorting children of 3627 in mount order
(00.016878) mnt:        Resorting children of 3625 in mount order
(00.016888) mnt:        Resorting children of 3291 in mount order
(00.016896) mnt:        Resorting children of 3290 in mount order
(00.016904) mnt:        Resorting children of 3289 in mount order
(00.016942) mnt:        Resorting children of 3288 in mount order
(00.016951) mnt:        Resorting children of 3287 in mount order
(00.016959) mnt:        Resorting children of 3286 in mount order
(00.016967) mnt:        Resorting children of 3285 in mount order
(00.016975) mnt:        Resorting children of 3284 in mount order
(00.016983) mnt:        Resorting children of 3283 in mount order
(00.016991) mnt:        Resorting children of 3282 in mount order
(00.017027) mnt:        Resorting children of 3281 in mount order
(00.017036) mnt: Done:
(00.017041) mnt: [./](3624->3280)
(00.017047) mnt:  [./etc/hosts](3646->3624)
(00.017058) mnt:  <--
(00.017094) mnt:  [./etc/hostname](3645->3624)
(00.017100) mnt:  <--
(00.017105) mnt:  [./etc/resolv.conf](3644->3624)
(00.017110) mnt:  <--
(00.017115) mnt:  [./sys](3628->3624)
(00.017120) mnt:   [./sys/fs/cgroup](3629->3628)
(00.017125) mnt:    [./sys/fs/cgroup/hugetlb](3641->3629)
(00.017130) mnt:    <--
(00.017135) mnt:    [./sys/fs/cgroup/cpuset](3640->3629)
(00.017170) mnt:    <--
(00.017175) mnt:    [./sys/fs/cgroup/pids](3639->3629)
(00.017180) mnt:    <--
(00.017185) mnt:    [./sys/fs/cgroup/memory](3638->3629)
(00.017190) mnt:    <--
(00.017195) mnt:    [./sys/fs/cgroup/devices](3637->3629)
(00.017200) mnt:    <--
(00.017204) mnt:    [./sys/fs/cgroup/blkio](3636->3629)
(00.017209) mnt:    <--
(00.017230) mnt:    [./sys/fs/cgroup/freezer](3635->3629)
(00.017236) mnt:    <--
(00.017241) mnt:    [./sys/fs/cgroup/rdma](3634->3629)
(00.017246) mnt:    <--
(00.017263) mnt:    [./sys/fs/cgroup/perf_event](3633->3629)
(00.017288) mnt:    <--
(00.017305) mnt:    [./sys/fs/cgroup/net_cls,net_prio](3632->3629)
(00.017326) mnt:    <--
(00.017331) mnt:    [./sys/fs/cgroup/cpu,cpuacct](3631->3629)
(00.017336) mnt:    <--
(00.017341) mnt:    [./sys/fs/cgroup/systemd](3630->3629)
(00.017364) mnt:    <--
(00.017371) mnt:   <--
(00.017375) mnt:   [./sys/firmware](3292->3628)
(00.017381) mnt:   <--
(00.017385) mnt:  <--
(00.017390) mnt:  [./dev](3626->3624)
(00.017424) mnt:   [./dev/shm](3643->3626)
(00.017430) mnt:   <--
(00.017435) mnt:   [./dev/mqueue](3642->3626)
(00.017440) mnt:   <--
(00.017444) mnt:   [./dev/pts](3627->3626)
(00.017449) mnt:   <--
(00.017454) mnt:  <--
(00.017459) mnt:  [./proc](3625->3624)
(00.017464) mnt:   [./proc/scsi](3291->3625)
(00.017492) mnt:   <--
(00.017497) mnt:   [./proc/sched_debug](3290->3625)
(00.017502) mnt:   <--
(00.017507) mnt:   [./proc/timer_list](3289->3625)
(00.017536) mnt:   <--
(00.017541) mnt:   [./proc/keys](3288->3625)
(00.017546) mnt:   <--
(00.017551) mnt:   [./proc/kcore](3287->3625)
(00.017556) mnt:   <--
(00.017581) mnt:   [./proc/acpi](3286->3625)
(00.017587) mnt:   <--
(00.017591) mnt:   [./proc/sysrq-trigger](3285->3625)
(00.017603) mnt:   <--
(00.017608) mnt:   [./proc/sys](3284->3625)
(00.017613) mnt:   <--
(00.017634) mnt:   [./proc/irq](3283->3625)
(00.017639) mnt:   <--
(00.017655) mnt:   [./proc/fs](3282->3625)
(00.017660) mnt:   <--
(00.017681) mnt:   [./proc/bus](3281->3625)
(00.017686) mnt:   <--
(00.017691) mnt:  <--
(00.017696) mnt: <--
(00.017716) mnt: Inspecting sharing on 3292 shared_id 0 master_id 0 (@./sys/firmware)
(00.017750) mnt: Inspecting sharing on 3291 shared_id 0 master_id 0 (@./proc/scsi)
(00.017756) mnt: Inspecting sharing on 3290 shared_id 0 master_id 0 (@./proc/sched_debug)
(00.017777) mnt:        The mount 3289 is bind for 3290 (@./proc/timer_list -> @./proc/sched_debug)
(00.017787) mnt:        The mount 3288 is bind for 3290 (@./proc/keys -> @./proc/sched_debug)
(00.017795) mnt:        The mount 3287 is bind for 3290 (@./proc/kcore -> @./proc/sched_debug)
(00.017831) mnt:        The mount 3626 is bind for 3290 (@./dev -> @./proc/sched_debug)
(00.017840) mnt: Inspecting sharing on 3289 shared_id 0 master_id 0 (@./proc/timer_list)
(00.017845) mnt: Inspecting sharing on 3288 shared_id 0 master_id 0 (@./proc/keys)
(00.017850) mnt: Inspecting sharing on 3287 shared_id 0 master_id 0 (@./proc/kcore)
(00.017855) mnt: Inspecting sharing on 3286 shared_id 0 master_id 0 (@./proc/acpi)
(00.017861) mnt: Inspecting sharing on 3285 shared_id 0 master_id 0 (@./proc/sysrq-trigger)
(00.017866) mnt:        The mount 3284 is bind for 3285 (@./proc/sys -> @./proc/sysrq-trigger)
(00.017898) mnt:        The mount 3283 is bind for 3285 (@./proc/irq -> @./proc/sysrq-trigger)
(00.017907) mnt:        The mount 3282 is bind for 3285 (@./proc/fs -> @./proc/sysrq-trigger)
(00.017922) mnt:        The mount 3281 is bind for 3285 (@./proc/bus -> @./proc/sysrq-trigger)
(00.017948) mnt:        The mount 3625 is bind for 3285 (@./proc -> @./proc/sysrq-trigger)
(00.017957) mnt: Inspecting sharing on 3284 shared_id 0 master_id 0 (@./proc/sys)
(00.017962) mnt: Inspecting sharing on 3283 shared_id 0 master_id 0 (@./proc/irq)
(00.017998) mnt: Inspecting sharing on 3282 shared_id 0 master_id 0 (@./proc/fs)
(00.018005) mnt: Inspecting sharing on 3281 shared_id 0 master_id 0 (@./proc/bus)
(00.018010) mnt: Inspecting sharing on 3646 shared_id 0 master_id 0 (@./etc/hosts)
(00.018015) mnt:        The mount 3645 is bind for 3646 (@./etc/hostname -> @./etc/hosts)
(00.018023) mnt:        The mount 3644 is bind for 3646 (@./etc/resolv.conf -> @./etc/hosts)
(00.018031) mnt: Inspecting sharing on 3645 shared_id 0 master_id 0 (@./etc/hostname)
(00.018037) mnt: Inspecting sharing on 3644 shared_id 0 master_id 0 (@./etc/resolv.conf)
(00.018042) mnt: Inspecting sharing on 3643 shared_id 0 master_id 0 (@./dev/shm)
(00.018047) mnt: Inspecting sharing on 3642 shared_id 0 master_id 0 (@./dev/mqueue)
(00.018094) mnt: Inspecting sharing on 3641 shared_id 0 master_id 25 (@./sys/fs/cgroup/hugetlb)
(00.018101) Error (criu/mount.c:938): mnt: Mount 3641 ./sys/fs/cgroup/hugetlb (master_id: 25 shared_id: 0) has unreachable sharing. Try --enable-external-masters.
(00.018137) Unlock network
(00.018173) Running network-unlock scripts
iptables-restore: invalid option -- 'w'
ip6tables-restore: invalid option -- 'w'
(00.022522) Unfreezing tasks into 1
(00.022600)     Unseizing 392365 into 1
(00.022887) Error (criu/cr-dump.c:1764): Dumping FAILED.

@adrianreber
Copy link
Member

It seems you are trying to checkpoint a process out of a container. This will not work easily. If you resolve the mount problem, you will hit many other problems.

What is your goal? Understanding you goal might help to find another solution for your problem.

Why aren't you checkpointing the whole container?

@mikosa
Copy link
Author

mikosa commented Oct 13, 2020

what do u mean? I am getting the pid of the container itself and trying to dump it

mike@gke-cluster-7-default-pool-94f35828-n541:~/criu-3.14$ sudo docker run -d grafana/grafana
9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb
mike@gke-cluster-7-default-pool-94f35828-n541:~/criu-3.14$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
9dae705cf1cd        grafana/grafana     "/run.sh"           8 seconds ago       Up 6 seconds        3000/tcp            festive_colden
mike@gke-cluster-7-default-pool-94f35828-n541:~/criu-3.14$ sudo docker inspect 9dae705cf1cd
[
    {
        "Id": "9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb",
        "Created": "2020-10-13T12:29:20.830279419Z",
        "Path": "/run.sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 430247,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-10-13T12:29:21.429685864Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
---
mike@gke-cluster-7-default-pool-94f35828-n541:~/criu-3.14$ sudo criu dump -t 430247
iptables-restore: invalid option -- 'w'
ip6tables-restore: invalid option -- 'w'
Error (criu/mount.c:938): mnt: Mount 3641 ./sys/fs/cgroup/hugetlb (master_id: 25 shared_id: 0) has unreachable sharing. Try --enable-external-masters.
iptables-restore: invalid option -- 'w'
ip6tables-restore: invalid option -- 'w'
Error (criu/cr-dump.c:1764): Dumping FAILED.
mike@gke-cluster-7-default-pool-94f35828-n541:~/criu-3.14$

@adrianreber
Copy link
Member

what do u mean? I am getting the pid of the container itself and trying to dump it

You cannot (easily) checkpoint a process out of a container without telling CRIU all the details about the resources which are only available in the container. Your first problem is a mount point which is just visible in the container and which CRIU cannot find outside of the container. No one has tried to checkpoint a process out of a running container for a long time as far as I know.

@mikosa
Copy link
Author

mikosa commented Oct 13, 2020

I am trying to checkpoint the whole container actually

I tried to use docker with similar errors

mike@gke-cluster-7-default-pool-94f35828-n541:~/criu-3.14$ sudo docker checkpoint create 9dae705cf1cd  test
Error response from daemon: Cannot checkpoint container 9dae705cf1cd: runc did not terminate sucessfully: criu failed: type NOTIFY errno 0 path= /run/containerd/io.containerd.runtime.v1.linux/moby/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65
e7539bd554038eb/criu-dump.log: unknown

mike@gke-cluster-7-default-pool-94f35828-n541:~/criu-3.14$ cat /run/containerd/io.containerd.runtime.v1.linux/moby/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb/criu-dump.log
(00.000000) Unable to get $HOME directory, local configuration file will not be used.(00.000031) Version: 3.14 (gitid 0)
(00.000037) Running on gke-cluster-7-default-pool-94f35828-n541 Linux 5.4.0-1024-gcp #24~18.04.1-Ubuntu SMP Sun Sep 6 03:37:03 UTC 2020 x86_64
(00.000039) Overwriting RPC settings with values from /etc/criu/runc.conf
(00.000052) Loaded kdat cache from /run/criu.kdat
(00.000126) ========================================
(00.000130) Dumping processes (pid: 430247)
(00.000132) ========================================
(00.000141) rlimit: RLIMIT_NOFILE unlimited for self
(00.000150) Running pre-dump scripts
(00.000153)     RPC
(00.000294) irmap: Searching irmap cache in work dir
(00.000306) No irmap-cache image
(00.000310) irmap: Searching irmap cache in parent
(00.000315) irmap: No irmap cache
(00.000351) cpu: x86_family 6 x86_vendor_id GenuineIntel x86_model_id Intel(R) Xeon(R) CPU @ 2.30GHz
(00.000363) cpu: fpu: xfeatures_mask 0x5 xsave_size 832 xsave_size_max 832 xsaves_size 0
(00.000368) cpu: fpu: x87 floating point registers     xstate_offsets      0 / 0      xstate_sizes    160 / 160   
(00.000371) cpu: fpu: AVX registers                    xstate_offsets    576 / 576    xstate_sizes    256 / 256   
(00.000374) cpu: fpu:1 fxsr:1 xsave:1 xsaveopt:1 xsavec:0 xgetbv1:0 xsaves:0
(00.000484) cg-prop: Parsing controller "cpu"
(00.000488) cg-prop:    Strategy "replace"
(00.000490) cg-prop:    Property "cpu.shares"
(00.000493) cg-prop:    Property "cpu.cfs_period_us"
(00.000495) cg-prop:    Property "cpu.cfs_quota_us"
(00.000497) cg-prop:    Property "cpu.rt_period_us"
(00.000499) cg-prop:    Property "cpu.rt_runtime_us"
(00.000501) cg-prop: Parsing controller "memory"
(00.000503) cg-prop:    Strategy "replace"
(00.000505) cg-prop:    Property "memory.limit_in_bytes"
(00.000507) cg-prop:    Property "memory.memsw.limit_in_bytes"
(00.000509) cg-prop:    Property "memory.swappiness"
(00.000511) cg-prop:    Property "memory.soft_limit_in_bytes"
(00.000513) cg-prop:    Property "memory.move_charge_at_immigrate"
(00.000515) cg-prop:    Property "memory.oom_control"
(00.000518) cg-prop:    Property "memory.use_hierarchy"
(00.000520) cg-prop:    Property "memory.kmem.limit_in_bytes"
(00.000522) cg-prop:    Property "memory.kmem.tcp.limit_in_bytes"
(00.000524) cg-prop: Parsing controller "cpuset"
(00.000526) cg-prop:    Strategy "replace"
(00.000528) cg-prop:    Property "cpuset.cpus"
(00.000530) cg-prop:    Property "cpuset.mems"
(00.000532) cg-prop:    Property "cpuset.memory_migrate"
(00.000534) cg-prop:    Property "cpuset.cpu_exclusive"
(00.000536) cg-prop:    Property "cpuset.mem_exclusive"
(00.000538) cg-prop:    Property "cpuset.mem_hardwall"
(00.000540) cg-prop:    Property "cpuset.memory_spread_page"
(00.000542) cg-prop:    Property "cpuset.memory_spread_slab"
(00.000544) cg-prop:    Property "cpuset.sched_load_balance"
(00.000546) cg-prop:    Property "cpuset.sched_relax_domain_level"
(00.000548) cg-prop: Parsing controller "blkio"
(00.000551) cg-prop:    Strategy "replace"
(00.000553) cg-prop:    Property "blkio.weight"
(00.000555) cg-prop: Parsing controller "freezer"
(00.000557) cg-prop:    Strategy "replace"
(00.000559) cg-prop: Parsing controller "perf_event"
(00.000561) cg-prop:    Strategy "replace"
(00.000563) cg-prop: Parsing controller "net_cls"
(00.000566) cg-prop:    Strategy "replace"
(00.000568) cg-prop:    Property "net_cls.classid"
(00.000570) cg-prop: Parsing controller "net_prio"
(00.000572) cg-prop:    Strategy "replace"
(00.000574) cg-prop:    Property "net_prio.ifpriomap"
(00.000576) cg-prop: Parsing controller "pids"
(00.000578) cg-prop:    Strategy "replace"
(00.000580) cg-prop:    Property "pids.max"
(00.000582) cg-prop: Parsing controller "devices"
(00.000584) cg-prop:    Strategy "replace"
(00.000586) cg-prop:    Property "devices.list"
(00.000623) Preparing image inventory (version 1)
(00.000649) Add pid ns 1 pid 441962
(00.000658) Add net ns 2 pid 441962
(00.000665) Add ipc ns 3 pid 441962
(00.000672) Add uts ns 4 pid 441962
(00.000677) Add time ns 5 pid 441962
(00.000684) Add mnt ns 6 pid 441962
(00.000690) Add user ns 7 pid 441962
(00.000696) Add cgroup ns 8 pid 441962
(00.000745) cg: Dumping cgroups for 441962
(00.000769) cg:  `- New css ID 1
(00.000772) cg:     `- [blkio] -> [/system.slice/containerd.service] [0]
(00.000774) cg:     `- [cpu,cpuacct] -> [/system.slice/containerd.service] [0]
(00.000776) cg:     `- [cpuset] -> [/] [0]
(00.000779) cg:     `- [devices] -> [/system.slice/containerd.service] [0]
(00.000781) cg:     `- [freezer] -> [/] [0]
(00.000783) cg:     `- [hugetlb] -> [/] [0]
(00.000785) cg:     `- [memory] -> [/system.slice/containerd.service] [0]
(00.000787) cg:     `- [name=systemd] -> [/system.slice/containerd.service] [0]
(00.000789) cg:     `- [net_cls,net_prio] -> [/] [0]
(00.000791) cg:     `- [perf_event] -> [/] [0]
(00.000793) cg:     `- [pids] -> [/system.slice/containerd.service] [0]
(00.000795) cg:     `- [rdma] -> [/] [0]
(00.000797) cg: Set 1 is criu one
(00.000830) Detected cgroup V1 freezer
(00.000832) freezing processes: 100000 attempts with 100 ms steps
(00.000846) freezer.state=FROZEN
(00.000898) SEIZE 430247: success
(00.000918) SEIZE 430300: success
(00.000937) SEIZE 430301: success
(00.000994) SEIZE 430302: success
(00.001020) SEIZE 430303: success
(00.001039) SEIZE 430307: success
(00.001048) SEIZE 430308: success
(00.001187) Seized task 430247, state 1
(00.001192) seccomp: Collected tid_real 430247 mode 0x2
(00.001209)     Seizing 430247's 430300 thread
(00.001247) seccomp: Collected tid_real 430300 mode 0x2
(00.001250)     Seizing 430247's 430301 thread
(00.001286) seccomp: Collected tid_real 430301 mode 0x2
(00.001289)     Seizing 430247's 430302 thread
(00.001324) seccomp: Collected tid_real 430302 mode 0x2
(00.001327)     Seizing 430247's 430303 thread
(00.001362) seccomp: Collected tid_real 430303 mode 0x2
(00.001364)     Seizing 430247's 430307 thread
(00.001400) seccomp: Collected tid_real 430307 mode 0x2
(00.001402)     Seizing 430247's 430308 thread
(00.001437) seccomp: Collected tid_real 430308 mode 0x2
(00.001455) Collected (-1 attempts, 0 in_progress)
(00.001495) Collected (0 attempts, 0 in_progress)
(00.001511) Collected 430247 in 1 state
(00.001527) Will take pid namespace in the image
(00.001530) Add pid ns 9 pid 430247
(00.001536) Will take net namespace in the image
(00.001538) Add net ns 10 pid 430247
(00.001545) Will take ipc namespace in the image
(00.001547) Add ipc ns 11 pid 430247
(00.001553) Will take uts namespace in the image
(00.001555) Add uts ns 12 pid 430247
(00.001564) Will take mnt namespace in the image
(00.001567) Add mnt ns 13 pid 430247
(00.001576) Lock network
(00.001579) Running network-lock scripts
(00.001581)     RPC
iptables-restore: invalid option -- 'w'
ip6tables-restore: invalid option -- 'w'
(00.005259)     type overlay source overlay mnt_id 3624 s_dev 0x1b0 / @ ./ flags 0x280000 options lowerdir=/var/lib/docker/overlay2/l/Q4UT5DMWS35NCX4BJBO3QKAFEJ:/var/lib/docker/overlay2/l/WAJLYKXZV32WSRCWLZZHDJX3FB:/var/lib/docker/overlay2/l/ETAWUFFOJEGBNJZNO47WIIHFV5:/var/lib/docker/overlay2/l/UMOCT6OFQDT6GBIZLK5TLZMV3V:/var/lib/docker/overlay2/l/O5AT2JV2LEHM37WR4MYNJYRSYO:/var/lib/docker/overlay2/l/77BTK52Y6AJ6RNGRCMO7BNPZHW:/var/lib/docker/overlay2/l/MG6ITAOLVILTUENLGII3KE7RBH,upperdir=/var/lib/docker/overlay2/a12977c43cf2aa78e1be2ffb93c640448b8931499d78c4b2774228e5777a98df/diff,workdir=/var/lib/docker/overlay2/a12977c43cf2aa78e1be2ffb93c640448b8931499d78c4b2774228e5777a98df/work,xino=off
(00.005277)     type proc source proc mnt_id 3625 s_dev 0x1b2 / @ ./proc flags 0x20000e options 
(00.005284)     type tmpfs source tmpfs mnt_id 3626 s_dev 0x1b3 / @ ./dev flags 0x1000002 options size=65536k,mode=755
(00.005292)     type devpts source devpts mnt_id 3627 s_dev 0x1b4 / @ ./dev/pts flags 0x20000a options gid=5,mode=620,ptmxmode=666
(00.005305)     type sysfs source sysfs mnt_id 3628 s_dev 0x1b5 / @ ./sys flags 0x20000f options 
(00.005341)     type tmpfs source tmpfs mnt_id 3629 s_dev 0x1b6 / @ ./sys/fs/cgroup flags 0x20000f options mode=755
(00.005350)     type cgroup source cgroup mnt_id 3630 s_dev 0x1e /docker/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb @ ./sys/fs/cgroup/systemd flags 0x28000f options xattr,name=systemd
(00.005363)     type cgroup source cgroup mnt_id 3631 s_dev 0x21 /docker/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb @ ./sys/fs/cgroup/cpu,cpuacct flags 0x28000f options cpu,cpuacct
(00.005370)     type cgroup source cgroup mnt_id 3632 s_dev 0x22 /docker/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb @ ./sys/fs/cgroup/net_cls,net_prio flags 0x28000f options net_cls,net_prio
(00.005375)     type cgroup source cgroup mnt_id 3633 s_dev 0x23 /docker/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb @ ./sys/fs/cgroup/perf_event flags 0x28000f options perf_event
(00.005381)     type cgroup source cgroup mnt_id 3634 s_dev 0x24 / @ ./sys/fs/cgroup/rdma flags 0x28000f options rdma
(00.005412)     type cgroup source cgroup mnt_id 3635 s_dev 0x25 /docker/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e
(00.005419)     type cgroup source cgroup mnt_id 3636 s_dev 0x26 /docker/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb @ ./sys/fs/cgroup/blkio flags 0x28000f options blkio
(00.005425)     type cgroup source cgroup mnt_id 3637 s_dev 0x27 /docker/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb @ ./sys/fs/cgroup/devices flags 0x28000f options devices
(00.005430)     type cgroup source cgroup mnt_id 3638 s_dev 0x28 /docker/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb @ ./sys/fs/cgroup/memory flags 0x28000f options memory
(00.005435)     type cgroup source cgroup mnt_id 3639 s_dev 0x29 /docker/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb @ ./sys/fs/cgroup/pids flags 0x28000f options pids
(00.005441)     type cgroup source cgroup mnt_id 3640 s_dev 0x2a /docker/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb @ ./sys/fs/cgroup/cpuset flags 0x28000f options cpuset
(00.005468)     type cgroup source cgroup mnt_id 3641 s_dev 0x2b /docker/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb @ ./sys/fs/cgroup/hugetlb flags 0x28000f options hugetlb
(00.005475)     type mqueue source mqueue mnt_id 3642 s_dev 0x1b1 / @ ./dev/mqueue flags 0x20000e options 
(00.005481)     type tmpfs source shm mnt_id 3643 s_dev 0x1b7 / @ ./dev/shm flags 0x20000e options size=65536k
(00.005487)     type ext4 source /dev/sda1 mnt_id 3644 s_dev 0x800001 /var/lib/docker/containers/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb/resolv.conf @ ./etc/resolv.conf flags 0x200000 options 
(00.005492)     type ext4 source /dev/sda1 mnt_id 3645 s_dev 0x800001 /var/lib/docker/containers/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb/hostname @ ./etc/hostname flags 0x200000 options 
(00.005497)     type ext4 source /dev/sda1 mnt_id 3646 s_dev 0x800001 /var/lib/docker/containers/9dae705cf1cde021f899a621823fc1c7f4935e00d7c59af65e7539bd554038eb/hosts @ ./etc/hosts flags 0x200000 options 
(00.005504)     type proc source proc mnt_id 3281 s_dev 0x1b2 /bus @ ./proc/bus flags 0x200001 options 
(00.005509)     type proc source proc mnt_id 3282 s_dev 0x1b2 /fs @ ./proc/fs flags 0x200001 options 
(00.005514)     type proc source proc mnt_id 3283 s_dev 0x1b2 /irq @ ./proc/irq flags 0x200001 options 
(00.005518)     type proc source proc mnt_id 3284 s_dev 0x1b2 /sys @ ./proc/sys flags 0x200001 options 
(00.005535)     type proc source proc mnt_id 3285 s_dev 0x1b2 /sysrq-trigger @ ./proc/sysrq-trigger flags 0x200001 options 
(00.005540)     type tmpfs source tmpfs mnt_id 3286 s_dev 0x1b8 / @ ./proc/acpi flags 0x200001 options 
(00.005547)     type tmpfs source tmpfs mnt_id 3287 s_dev 0x1b3 /null @ ./proc/kcore flags 0x1000002 options size=65536k,mode=755
(00.005571)     type tmpfs source tmpfs mnt_id 3288 s_dev 0x1b3 /null @ ./proc/keys flags 0x1000002 options size=65536k,mode=755
(00.005576)     type tmpfs source tmpfs mnt_id 3289 s_dev 0x1b3 /null @ ./proc/timer_list flags 0x1000002 options size=65536k,mode=755
(00.005581)     type tmpfs source tmpfs mnt_id 3290 s_dev 0x1b3 /null @ ./proc/sched_debug flags 0x1000002 options size=65536k,mode=755
(00.005586)     type tmpfs source tmpfs mnt_id 3291 s_dev 0x1b9 / @ ./proc/scsi flags 0x200001 options 
(00.005594)     type tmpfs source tmpfs mnt_id 3292 s_dev 0x1ba / @ ./sys/firmware flags 0x200001 options 
(00.005601) mnt: Building mountpoints tree
(00.005603) mnt:        Building plain mount tree
(00.005605) mnt:                Working on 3292->3628
(00.005608) mnt:                Working on 3291->3625
(00.005610) mnt:                Working on 3290->3625
(00.005612) mnt:                Working on 3289->3625
(00.005614) mnt:                Working on 3288->3625
(00.005616) mnt:                Working on 3287->3625
(00.005618) mnt:                Working on 3286->3625
(00.005620) mnt:                Working on 3285->3625
(00.005622) mnt:                Working on 3284->3625
(00.005624) mnt:                Working on 3283->3625
(00.005626) mnt:                Working on 3282->3625
(00.005628) mnt:                Working on 3281->3625
(00.005630) mnt:                Working on 3646->3624
(00.005632) mnt:                Working on 3645->3624
(00.005634) mnt:                Working on 3644->3624
(00.005636) mnt:                Working on 3643->3626
(00.005638) mnt:                Working on 3642->3626
(00.005640) mnt:                Working on 3641->3629
(00.005642) mnt:                Working on 3640->3629
(00.005644) mnt:                Working on 3639->3629
(00.005646) mnt:                Working on 3638->3629
(00.005648) mnt:                Working on 3637->3629
(00.005650) mnt:                Working on 3636->3629
(00.005652) mnt:                Working on 3635->3629
(00.005654) mnt:                Working on 3634->3629
(00.005656) mnt:                Working on 3633->3629
(00.005658) mnt:                Working on 3632->3629
(00.005660) mnt:                Working on 3631->3629
(00.005662) mnt:                Working on 3630->3629
(00.005664) mnt:                Working on 3629->3628
(00.005666) mnt:                Working on 3628->3624
(00.005668) mnt:                Working on 3627->3626
(00.005670) mnt:                Working on 3626->3624
(00.005672) mnt:                Working on 3625->3624
(00.005674) mnt:                Working on 3624->3280
(00.005677) mnt:        Resorting children of 3624 in mount order
(00.005679) mnt:        Resorting children of 3646 in mount order
(00.005682) mnt:        Resorting children of 3645 in mount order
(00.005683) mnt:        Resorting children of 3644 in mount order
(00.005685) mnt:        Resorting children of 3628 in mount order
(00.005688) mnt:        Resorting children of 3629 in mount order
(00.005692) mnt:        Resorting children of 3641 in mount order
(00.005694) mnt:        Resorting children of 3640 in mount order
(00.005695) mnt:        Resorting children of 3639 in mount order
(00.005697) mnt:        Resorting children of 3638 in mount order
(00.005699) mnt:        Resorting children of 3637 in mount order
(00.005701) mnt:        Resorting children of 3636 in mount order
(00.005703) mnt:        Resorting children of 3635 in mount order
(00.005705) mnt:        Resorting children of 3634 in mount order
(00.005707) mnt:        Resorting children of 3633 in mount order
(00.005709) mnt:        Resorting children of 3632 in mount order
(00.005711) mnt:        Resorting children of 3631 in mount order
(00.005713) mnt:        Resorting children of 3630 in mount order
(00.005714) mnt:        Resorting children of 3292 in mount order
(00.005716) mnt:        Resorting children of 3626 in mount order
(00.005718) mnt:        Resorting children of 3643 in mount order
(00.005720) mnt:        Resorting children of 3642 in mount order
(00.005722) mnt:        Resorting children of 3627 in mount order
(00.005724) mnt:        Resorting children of 3625 in mount order
(00.005728) mnt:        Resorting children of 3291 in mount order
(00.005730) mnt:        Resorting children of 3290 in mount order
(00.005731) mnt:        Resorting children of 3289 in mount order
(00.005733) mnt:        Resorting children of 3288 in mount order
(00.005735) mnt:        Resorting children of 3287 in mount order
(00.005737) mnt:        Resorting children of 3286 in mount order
(00.005745) mnt:        Resorting children of 3285 in mount order
(00.005747) mnt:        Resorting children of 3284 in mount order
(00.005749) mnt:        Resorting children of 3283 in mount order
(00.005751) mnt:        Resorting children of 3282 in mount order
(00.005752) mnt:        Resorting children of 3281 in mount order
(00.005754) mnt: Done:
(00.005756) mnt: [./](3624->3280)
(00.005759) mnt:  [./etc/hosts](3646->3624)
(00.005761) mnt:  <--
(00.005763) mnt:  [./etc/hostname](3645->3624)
(00.005765) mnt:  <--
(00.005767) mnt:  [./etc/resolv.conf](3644->3624)
(00.005769) mnt:  <--
(00.005771) mnt:  [./sys](3628->3624)
(00.005773) mnt:   [./sys/fs/cgroup](3629->3628)
(00.005778) mnt:    [./sys/fs/cgroup/hugetlb](3641->3629)
(00.005780) mnt:    <--
(00.005782) mnt:    [./sys/fs/cgroup/cpuset](3640->3629)
(00.005784) mnt:    <--
(00.005786) mnt:    [./sys/fs/cgroup/pids](3639->3629)
(00.005788) mnt:    <--
(00.005790) mnt:    [./sys/fs/cgroup/memory](3638->3629)
(00.005792) mnt:    <--
(00.005794) mnt:    [./sys/fs/cgroup/devices](3637->3629)
(00.005796) mnt:    <--
(00.005798) mnt:    [./sys/fs/cgroup/blkio](3636->3629)
(00.005800) mnt:    <--
(00.005801) mnt:    [./sys/fs/cgroup/freezer](3635->3629)
(00.005803) mnt:    <--
(00.005805) mnt:    [./sys/fs/cgroup/rdma](3634->3629)
(00.005807) mnt:    <--
(00.005809) mnt:    [./sys/fs/cgroup/perf_event](3633->3629)
(00.005811) mnt:    <--
(00.005813) mnt:    [./sys/fs/cgroup/net_cls,net_prio](3632->3629)
(00.005815) mnt:    <--
(00.005817) mnt:    [./sys/fs/cgroup/cpu,cpuacct](3631->3629)
(00.005819) mnt:    <--
(00.005821) mnt:    [./sys/fs/cgroup/systemd](3630->3629)
(00.005823) mnt:    <--
(00.005825) mnt:   <--
(00.005827) mnt:   [./sys/firmware](3292->3628)
(00.005829) mnt:   <--
(00.005831) mnt:  <--
(00.005833) mnt:  [./dev](3626->3624)
(00.005835) mnt:   [./dev/shm](3643->3626)
(00.005837) mnt:   <--
(00.005839) mnt:   [./dev/mqueue](3642->3626)
(00.005841) mnt:   <--
(00.005843) mnt:   [./dev/pts](3627->3626)
(00.005845) mnt:   <--
(00.005847) mnt:  <--
(00.005848) mnt:  [./proc](3625->3624)
(00.005851) mnt:   [./proc/scsi](3291->3625)
(00.005853) mnt:   <--
(00.005855) mnt:   [./proc/sched_debug](3290->3625)
(00.005857) mnt:   <--
(00.005858) mnt:   [./proc/timer_list](3289->3625)
(00.005861) mnt:   <--
(00.005862) mnt:   [./proc/keys](3288->3625)
(00.005864) mnt:   <--
(00.005866) mnt:   [./proc/kcore](3287->3625)
(00.005868) mnt:   <--
(00.005870) mnt:   [./proc/acpi](3286->3625)
(00.005872) mnt:   <--
(00.005874) mnt:   [./proc/sysrq-trigger](3285->3625)
(00.005876) mnt:   <--
(00.005878) mnt:   [./proc/sys](3284->3625)
(00.005880) mnt:   <--
(00.005882) mnt:   [./proc/irq](3283->3625)
(00.005884) mnt:   <--
(00.005886) mnt:   [./proc/fs](3282->3625)
(00.005888) mnt:   <--
(00.005890) mnt:   [./proc/bus](3281->3625)
(00.005892) mnt:   <--
(00.005894) mnt:  <--
(00.005896) mnt: <--
(00.005903) mnt: Found /dev/null mapping for ./proc/sched_debug mountpoint
(00.005905) mnt: Found /dev/null mapping for ./proc/timer_list mountpoint
(00.005908) mnt: Found /dev/null mapping for ./proc/keys mountpoint
(00.005910) mnt: Found /dev/null mapping for ./proc/kcore mountpoint
(00.005914) mnt: Found /etc/hosts mapping for ./etc/hosts mountpoint
(00.005917) mnt: Found /etc/hostname mapping for ./etc/hostname mountpoint
(00.005919) mnt: Found /etc/resolv.conf mapping for ./etc/resolv.conf mountpoint
(00.005922) mnt: Found /sys/fs/cgroup/hugetlb mapping for ./sys/fs/cgroup/hugetlb mountpoint
(00.005924) mnt: Found /sys/fs/cgroup/cpuset mapping for ./sys/fs/cgroup/cpuset mountpoint
(00.005926) mnt: Found /sys/fs/cgroup/pids mapping for ./sys/fs/cgroup/pids mountpoint
(00.005929) mnt: Found /sys/fs/cgroup/memory mapping for ./sys/fs/cgroup/memory mountpoint
(00.005931) mnt: Found /sys/fs/cgroup/devices mapping for ./sys/fs/cgroup/devices mountpoint
(00.005933) mnt: Found /sys/fs/cgroup/blkio mapping for ./sys/fs/cgroup/blkio mountpoint
(00.005935) mnt: Found /sys/fs/cgroup/freezer mapping for ./sys/fs/cgroup/freezer mountpoint
(00.005938) mnt: Found /sys/fs/cgroup/rdma mapping for ./sys/fs/cgroup/rdma mountpoint
(00.005940) mnt: Found /sys/fs/cgroup/perf_event mapping for ./sys/fs/cgroup/perf_event mountpoint
(00.005942) mnt: Found /sys/fs/cgroup/net_cls,net_prio mapping for ./sys/fs/cgroup/net_cls,net_prio mountpoint
(00.005945) mnt: Found /sys/fs/cgroup/cpu,cpuacct mapping for ./sys/fs/cgroup/cpu,cpuacct mountpoint
(00.005947) mnt: Found /sys/fs/cgroup/systemd mapping for ./sys/fs/cgroup/systemd mountpoint
(00.005968) mnt: Inspecting sharing on 3292 shared_id 0 master_id 0 (@./sys/firmware)
(00.005971) mnt: Inspecting sharing on 3291 shared_id 0 master_id 0 (@./proc/scsi)
(00.005973) mnt: Inspecting sharing on 3290 shared_id 0 master_id 0 (@./proc/sched_debug)
(00.005978) mnt:        The mount 3289 is bind for 3290 (@./proc/timer_list -> @./proc/sched_debug)
(00.005981) mnt:        The mount 3288 is bind for 3290 (@./proc/keys -> @./proc/sched_debug)
(00.005983) mnt:        The mount 3287 is bind for 3290 (@./proc/kcore -> @./proc/sched_debug)
(00.005985) mnt:        The mount 3626 is bind for 3290 (@./dev -> @./proc/sched_debug)
(00.005987) mnt: Inspecting sharing on 3289 shared_id 0 master_id 0 (@./proc/timer_list)
(00.005990) mnt: Inspecting sharing on 3288 shared_id 0 master_id 0 (@./proc/keys)
(00.005992) mnt: Inspecting sharing on 3287 shared_id 0 master_id 0 (@./proc/kcore)
(00.005994) mnt: Inspecting sharing on 3286 shared_id 0 master_id 0 (@./proc/acpi)
(00.005996) mnt: Inspecting sharing on 3285 shared_id 0 master_id 0 (@./proc/sysrq-trigger)
(00.005998) mnt:        The mount 3284 is bind for 3285 (@./proc/sys -> @./proc/sysrq-trigger)
(00.006000) mnt:        The mount 3283 is bind for 3285 (@./proc/irq -> @./proc/sysrq-trigger)
(00.006002) mnt:        The mount 3282 is bind for 3285 (@./proc/fs -> @./proc/sysrq-trigger)
(00.006005) mnt:        The mount 3281 is bind for 3285 (@./proc/bus -> @./proc/sysrq-trigger)
(00.006007) mnt:        The mount 3625 is bind for 3285 (@./proc -> @./proc/sysrq-trigger)
(00.006009) mnt: Inspecting sharing on 3284 shared_id 0 master_id 0 (@./proc/sys)
(00.006011) mnt: Inspecting sharing on 3283 shared_id 0 master_id 0 (@./proc/irq)
(00.006013) mnt: Inspecting sharing on 3282 shared_id 0 master_id 0 (@./proc/fs)
(00.006015) mnt: Inspecting sharing on 3281 shared_id 0 master_id 0 (@./proc/bus)
(00.006017) mnt: Inspecting sharing on 3646 shared_id 0 master_id 0 (@./etc/hosts)
(00.006019) mnt:        The mount 3645 is bind for 3646 (@./etc/hostname -> @./etc/hosts)
(00.006021) mnt:        The mount 3644 is bind for 3646 (@./etc/resolv.conf -> @./etc/hosts)
(00.006023) mnt: Inspecting sharing on 3645 shared_id 0 master_id 0 (@./etc/hostname)
(00.006026) mnt: Inspecting sharing on 3644 shared_id 0 master_id 0 (@./etc/resolv.conf)
(00.006028) mnt: Inspecting sharing on 3643 shared_id 0 master_id 0 (@./dev/shm)
(00.006030) mnt: Inspecting sharing on 3642 shared_id 0 master_id 0 (@./dev/mqueue)
(00.006032) mnt: Inspecting sharing on 3641 shared_id 0 master_id 25 (@./sys/fs/cgroup/hugetlb)
(00.006034) mnt: Inspecting sharing on 3640 shared_id 0 master_id 24 (@./sys/fs/cgroup/cpuset)
(00.006037) mnt: Inspecting sharing on 3639 shared_id 0 master_id 23 (@./sys/fs/cgroup/pids)
(00.006039) mnt: Inspecting sharing on 3638 shared_id 0 master_id 22 (@./sys/fs/cgroup/memory)
(00.006041) mnt: Inspecting sharing on 3637 shared_id 0 master_id 21 (@./sys/fs/cgroup/devices)
(00.006043) mnt: Inspecting sharing on 3636 shared_id 0 master_id 20 (@./sys/fs/cgroup/blkio)
(00.006046) mnt: Inspecting sharing on 3635 shared_id 0 master_id 19 (@./sys/fs/cgroup/freezer)
(00.006048) mnt: Inspecting sharing on 3634 shared_id 0 master_id 18 (@./sys/fs/cgroup/rdma)
(00.006050) mnt: Inspecting sharing on 3633 shared_id 0 master_id 17 (@./sys/fs/cgroup/perf_event)
(00.006052) mnt: Inspecting sharing on 3632 shared_id 0 master_id 16 (@./sys/fs/cgroup/net_cls,net_prio)
(00.006054) mnt: Inspecting sharing on 3631 shared_id 0 master_id 15 (@./sys/fs/cgroup/cpu,cpuacct)
(00.006057) mnt: Inspecting sharing on 3630 shared_id 0 master_id 11 (@./sys/fs/cgroup/systemd)
(00.006059) mnt: Inspecting sharing on 3629 shared_id 0 master_id 0 (@./sys/fs/cgroup)
(00.006061) mnt: Inspecting sharing on 3628 shared_id 0 master_id 0 (@./sys)
(00.006063) mnt: Inspecting sharing on 3627 shared_id 0 master_id 0 (@./dev/pts)
(00.006065) mnt: Inspecting sharing on 3626 shared_id 0 master_id 0 (@./dev)
(00.006067) mnt: Inspecting sharing on 3625 shared_id 0 master_id 0 (@./proc)
(00.006069) mnt: Inspecting sharing on 3624 shared_id 0 master_id -1 (@./)
(00.006075) Collecting netns 10/430247
(00.006078) Switching to 430247's net for collecting sockets
(00.006181) unix:       Collected: ino 1601260 peer_ino 0 family    1 type    5 state  7 name (null)
(00.006354) inet:       Collected: ino 0x185ff2 family AF_INET    type SOCK_STREAM    port    36242 state TCP_ESTABLISHED  src_addr 172.17.0.2
(00.006363) inet:       Collected: ino 0x185ff9 family AF_INET    type SOCK_STREAM    port    49792 state TCP_ESTABLISHED  src_addr 172.17.0.2
(00.006486) inet:       Collected: ino 0x17cfe4 family AF_INET6   type SOCK_STREAM    port     3000 state TCP_LISTEN       src_addr ::
(00.006558) netlink: Collect netlink sock 0x185bb3
(00.006561) netlink: Collect netlink sock 0x17ce3b
(00.006563) netlink: Collect netlink sock 0x17ce43
(00.006565) netlink: Collect netlink sock 0x186eeb
(00.006567) netlink: Collect netlink sock 0x17ce4a
(00.006569) netlink: Collect netlink sock 0x17ce3c
(00.006571) netlink: Collect netlink sock 0x17ce3f
(00.006573) netlink: Collect netlink sock 0x17ce44
(00.006575) netlink: Collect netlink sock 0x17ce3d
(00.006577) netlink: Collect netlink sock 0x17ce3e
(00.006579) netlink: Collect netlink sock 0x17ce4b
(00.006663) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 430247
(00.006666) seccomp:     Disable filter on tid_rea 430300, will be propagated
(00.006668) seccomp:     Disable filter on tid_rea 430301, will be propagated
(00.006670) seccomp:     Disable filter on tid_rea 430302, will be propagated
(00.006672) seccomp:     Disable filter on tid_rea 430303, will be propagated
(00.006674) seccomp:     Disable filter on tid_rea 430307, will be propagated
(00.006676) seccomp:     Disable filter on tid_rea 430308, will be propagated
(00.006728) ========================================
(00.006730) Dumping task (pid: 430247)
(00.006733) ========================================
(00.006734) Obtaining task stat ... 
(00.006771) 
(00.006774) Collecting mappings (pid: 430247)
(00.006776) ----------------------------------------
(00.007712) Error (criu/files-reg.c:1372): Can't lookup mount=3240 for fd=-3 path=/usr/share/grafana/bin/grafana-server
(00.007728) Error (criu/cr-dump.c:1247): Collect mappings (pid: 430247) failed with -1
(00.007766) Unlock network
(00.007769) Running network-unlock scripts
(00.007771)     RPC
iptables-restore: invalid option -- 'w'
ip6tables-restore: invalid option -- 'w'
(00.010795) Unfreezing tasks into 1
(00.010806)     Unseizing 430247 into 1
(00.010916) Error (criu/cr-dump.c:1764): Dumping FAILED.

@adrianreber
Copy link
Member

Yes, Ubuntu is broken: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1857257

Use another distribution or another Docker graphdriver.

Also see: #860

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@adrianreber @mikosa