Nomad exec driver leaving exector.out task even after allocation being destroyed/removed.

[116davinder]

Nomad version

1.6.3

Operating system and Environment details

Ubuntu 20.04

Issue

Below mentioned nomad processes keep growing for some time, tried nomad system gc / restart nomad client service.

Finally do kill -9 190632 .....

# ps aux | grep nomad                                                                                                                                     
root      190405  6.8  5.4 1921308 53464 ?       Ssl  08:15   4:05 /usr/local/bin/nomad agent -config /etc/nomad                                                                           
root      190434  0.0  0.0   4708    28 pts/1    S+   08:16   0:00 tail -f /var/log/nomad/nomad.log                                                                                        
root      190632  0.0  0.7 1750588 7036 ?        Ssl  08:22   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/76311d7e-2592-2465-f4ca-83d6b3a1b3df/50-sync-db-check/ex
ecutor.out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}                                                                                                                    
root      190637  0.0  0.3 1750588 3320 ?        Ssl  08:22   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/76311d7e-2592-2465-f4ca-83d6b3a1b3df/40-kpropd/executor.
out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}                                                                                                                           
root      190642  0.0  0.8 1750332 8752 ?        Ssl  08:22   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/76311d7e-2592-2465-f4ca-83d6b3a1b3df/30-instantiate-kerb
eros-db/executor.out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}                                                                                                          
root      190694  0.0  0.8 1750588 8380 ?        Ssl  08:22   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/1d43ebd1-14fd-e6e0-21cb-94e9a00965e8/50-sync-db-check/ex
ecutor.out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}                                                                                                                    
root      190699  0.0  1.0 1750588 10836 ?       Ssl  08:22   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/1d43ebd1-14fd-e6e0-21cb-94e9a00965e8/40-kpropd/executor.
out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}                                                                                                                           
root      190704  0.0  1.1 1750588 11132 ?       Ssl  08:22   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/1d43ebd1-14fd-e6e0-21cb-94e9a00965e8/30-instantiate-kerb
eros-db/executor.out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}                                                                                                          
root      190957  0.0  0.8 1750588 8652 ?        Ssl  08:30   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/dd0bc06a-9ac5-a6bf-0be0-2a0f6c949a8d/40-kpropd/executor.
out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}                                                                                                                           
root      190962  0.0  0.6 1750588 6144 ?        Ssl  08:30   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/dd0bc06a-9ac5-a6bf-0be0-2a0f6c949a8d/50-sync-db-check/ex
ecutor.out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}                                                                                                                    
root      191048  0.0  0.9 1750332 9552 ?        Ssl  08:31   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/38c87303-9fa4-8c89-b25b-5457fa2bf17d/40-kpropd/executor.
out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}                                                                                                                           
root      191053  0.0  1.1 1750588 11152 ?       Ssl  08:31   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/38c87303-9fa4-8c89-b25b-5457fa2bf17d/50-sync-db-check/ex
ecutor.out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}                                                                                                                    
root      191158  0.0  0.9 1750588 9184 ?        Ssl  08:32   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/b4660b31-d5c5-38ca-ca9f-ced8499fdab0/50-sync-db-check/ex
ecutor.out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}                                                                                                                    
root      191163  0.0  0.4 1750588 4896 ?        Ssl  08:32   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/b4660b31-d5c5-38ca-ca9f-ced8499fdab0/40-kpropd/executor.
out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}                                                                                                                           
root      191296  0.0  0.8 1750588 8548 ?        Ssl  08:35   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/ac1cc970-c448-6afd-f254-b342cdd20194/50-sync-db-check/ex
ecutor.out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}                                                                                                                    
root      191301  0.0  0.9 1750588 8912 ?        Ssl  08:35   0:00 /usr/local/bin/nomad executor {"LogFile":"/opt/nomad/data/alloc/ac1cc970-c448-6afd-f254-b342cdd20194/40-kpropd/executor.
out","LogLevel":"debug","FSIsolation":true,"CpuTotalTicks":2095}  
….

Reproduction steps

Don't Know how.

Expected Result

above mentioned nomad processed to be automatically deleted after task deletion.

Actual Result

Job file (if appropriate)

Nomad Server logs (if appropriate)

Nothing in special in logs

Nomad Client logs (if appropriate)

Nothing in special in logs

[lgfa29]

Hi @116davinder 👋

I believe this is the same issue as reported in #6980, so I'm going to close this one as duplicate. Feel free to follow the discussion there and 👍.

But let me know if you think this is a different problem and we can reopen it

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.