mounts fail for containerd-driver #13229
Comments
|
Hi @alemonmk! Thanks for providing all the CSI plugin's allocation logs, as those will help narrow down the problem. When Nomad asks the CSI plugin for a volume, it does a "Node Stage Volume" RPC (which succeeds in your logs), followed by a "Nomad Publish Volume" RPC. Most plugins literally just call
I don't understand what it's trying to do with this "lost+found" entity. That's coming from the plugin, not Nomad. It looks like it was added in vultr/vultr-csi@774479d to However, that's just a spurious warning and the plugin goes on to tell us it's published the volume successfully:
But then 4 seconds later we see an "Unpublish Volume" RPC come in, which means Nomad said "nope, this isn't going to work" and unpublished it.
Once we've hit this point, we set some state for the allocation (ref csi_hook.go#L111-L113) and then pass that state to the task driver in the
Note that the Nomad datadir path we see there is mapped to So somewhere between the volume being mounted and us exposing it to the task, we're dropping the state. It could also be a problem with the third-party containerd task driver. In any case, the next thing to do here is to look at the debug-level logs on the client. Can you share those from the time the allocation is placed to when it fails? Aside:
Yes, that's the correct behavior! The access mode and attachment mode are set on the volume only once it's actually attached. We're not attached because the allocation failed. |
I hope you are actually talking about server logs because there's literally nothing about allocations in client log, even on trace level. It probably does not matter much anymore, because there's nothing can be mounted unless it's one of default mounts or host volume: Server log: Wait, what's with that mount source path? At least it has to be in As far as I can read into codes, paths go into syscall pretty much as-is, so I don't even know if who did anything wrong here. |
Nope! Nomad servers are the control plane running raft. Nomad clients are the nodes where workloads are run. This is admittedly a little confusing with the Nomad CLI also being a "client" of sorts. But this is probably made more confusing because you're running the server and the client in the same agent, which we strongly recommend against doing for production. Fortunately, the client logs are included in what you've provided. 😀 Here's the relevant log line:
As you've noted, this looks like a general problem with mounts and not specifically a CSI issue. I've retitled the issue for clarity
Yes, that looks very strange to me. I would expect the path shown to be the absolute path from the Nomad data directory for the task, but this path looks almost as if you were running Nomad as a containerd container. If that's the case, you're "off the edge of the map" and although we'd love to read a write-up of your adventure to get that working once you've done so... it's an entirely unsupported installation. If that's not the case, it looks like there might be an issue with your containerd driver configuration. If you can provide your containerd driver configuration that would be very helpful. The driver is third-party, so I'm also going to tag our good friend @shishir-a412ed over at Roblox who is the primary author of the driver. He might have some insights here as well. |
tgross
changed the title
This log comes from Cluster -> Server -> <node> -> Monitor on the UI. I'm aware it's recommended against though with limited budget and not critical workload it is probably fine.
It's not the case, the bootstrapping effort would be off the charts, and admittedly I'm no expert in container tech. |
|
For the record, I changed the netbox job above to use docker driver and the mount is up. |
|
Thanks @alemonmk. We might need to open an issue with the driver authors here but let me see if I can make a minimal reproduction as well first. |
|
I was able to reproduce this issue on my local Vagrant environment and it looks like the problem is that the containerd driver expects the mount source to be from the host and not the task. As it turns out, this behavior is exactly what's documented for the containerd driver (ref
This is different than other task drivers, which want to isolate the host's filesystem and then end up needing to have a To reproduce, first I reduced the jobspec you provided down to a minimal spec that demonstrates the problem: jobspecjob "example" {
datacenters = ["dc1"]
group "app" {
task "netbox" {
driver = "containerd-driver"
config {
image = "busybox:1"
command = "httpd"
args = ["-v", "-f", "-p", "8001", "-h", "/local"]
mounts = [
{
type = "bind"
source = "${NOMAD_SECRETS_DIR}/extra.py"
target = "/opt/netbox/netbox/config/extra.py"
options = ["rbind", "ro"]
}
]
}
template {
data = "xyzzy"
destination = "${NOMAD_SECRETS_DIR}/extra.py"
}
resources {
cpu = 256
memory = 256
}
}
}
}I run that job and get the same error you're seeing:
And if I remove the bind-mount and/or replace it with the tmpfs mount, everything works as expected. Next I dropped a file into a directory on the host and switched the mounts = [
{
type = "bind"
source = "/srv/data/example.txt"
target = "/opt/netbox/netbox/config/extra.py"
options = ["rbind", "ro"]
}
]Now I can run the job and it works as expected: This still leaves you with a problem with the template rendering though. Template rendering happens before the container is created, so you can't render into the task's pivot root. There are two ways to deal with this:
|
|
Oh right, but the original problem was that you couldn't get CSI to work. 😊 I just double-checked your original jobspec and it looks like you're doing this same bind-mount trick from the CSI volume mount to the location you want the config file. So the same two of solutions apply there as well. I was able to confirm that CSI should work generally with the containerd plugin. This jobspec uses the jobspecjob "example" {
datacenters = ["dc1"]
group "app" {
volume "volume0" {
type = "csi"
source = "test-volume"
attachment_mode = "file-system"
access_mode = "single-node-reader-only"
read_only = true
}
task "netbox" {
driver = "containerd-driver"
config {
image = "busybox:1"
command = "httpd"
args = ["-v", "-f", "-p", "8001", "-h", "/local"]
}
volume_mount {
volume = "volume0"
destination = "${NOMAD_ALLOC_DIR}/volume0"
}
resources {
cpu = 256
memory = 256
}
}
}
} |
|
Thank you for the detailed analysis. So it's all my misunderstanding...
Podman (which I originally wanted to use) also ask for path on host as well.
I believe the config file mount is irrelevant, I tried this and it's failing.
Thanks for the suggestion, though I made things work with |
In the same way with the same error as before? As I noted, the hostpath driver seemed to work fine with containerd for me. So it'd be interesting to see if the Vultr driver is doing something unusual here. Can you provide the client debug logs and plugin logs for this exact scenario without the complication of the template paths?
Those environment variables are for the the paths inside the task's view of the filesystem. For task drivers that have filesystem isolation like |
|
OK, I used the same busybox sleep container above and look further into it by execing into the CSI node container while have no volume_mount stanza to mount the volume. Very interesting finding: From client log (minus 9 hours from time to be consistent to time shown below): I have to do a stop alloc to get new log, but the content is reproducible. From CSI node plugin log: Same good old successful volume publishing here. From CSI node container: From host (also minus 9 hours on the file time): No wonder all the no such file or directory! It actually does not exist! The mount somehow does not propagated to host, but I have no idea what should have been set up on my system but haven't... |
Can you provide the exact jobspec you ran please so that there's no ambiguity for me to try to reproduce?
Ok, that set of logs is mostly server logs but the first error here indicates that when the client tried to claim the volume, the volume did not have free claims available. We don't also see a "could not claim volume" error in the client logs which suggests it was retried successfully. But then 4 seconds later we're unpublishing the volume which suggests the task failed to start. The CSI node plugin log is showing successful responses:
The per-alloc path is what's missing from the host. It probably couldn't hurt if you could share the plugin jobspec as well. Also, I don't think you have debug logging enabled on the client, otherwise I'd expect to see a lot more logs that would really help me figure out what's going on here. As a result, I also can't see the final error message you're seeing that's getting sent up to task event. Can you provide those please? |
|
Disregard the client log last post, I think it was the log when it was restarting the task. Job specs: https://gist.github.com/alemonmk/7365e17ca6b57bbc4ef78311bf9332f9 I get logs from here, and I'm not sure there's a config entry to turn on a more verbose logging. Task never fails since I don't have volume mounted. Mounts in CSI plugin is working. |
|
Hi @tgross Thanks for the detailed analysis and reproduction! IIUC, the bind mount is failing because However, when the job tries to bind mount using I believe this works in docker, because docker is looking for paths relative to task allocation directory. @alemonmk Is there an easy way I can reproduce this in a vagrant VM? a) Install Vultr CSI plugin. Make it up and running and register it with Nomad. I know the job spec for (c), however looking for some instructions around (a) and (b) as I am not super familiar with Vultr CSI plugin. If there is an easier way to reproduce this, please let me know. |
|
I was going through the Some more details in this So, it will work for relative paths like |
I believe this is it: nomad/drivers/docker/driver.go Line 685 in 2697e63 nomad/drivers/docker/driver.go Line 1255 in 2697e63
Please edit and nomad job run
I use Terraform to do this. I don't know anything about vagrant though, apologize. Alternatively, vultr-csi does implement |
|
I believe I have found the problem here despite my limited code reading skills. tl;dr containerd-driver does not set bind mount propagation mode, thus what mounted in CSI node container does not appear in host. First, the CSI mounting points nomad/client/allocrunner/taskrunner/plugin_supervisor_hook.go Lines 155 to 161 in 2b054e3 The mounts defined goes to this function, which we can see BTW this is how docker driver handles it: nomad/drivers/docker/driver.go Lines 1046 to 1063 in 2b054e3 Resulting in containerd mounting /local/csi with propagation mode = rprivate: I'm not sure where I validated this by running csi-vultr-nodes with docker driver. |
|
I have opened issues to containerd-driver, thank you @tgross and @shishir-a412ed for helping me get to the bottom of this with detailed analysis and thinking process. |
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v1.3.1 (2b054e3)
Operating system and Environment details
Vanilla Vultr Debian 11 amd64 image
nomad-driver-containerd 0.9.3
containerd
1.4.13~ds1-1~deb11u1vultr-csi v0.7.0
Issue
I'm trying to run PostgreSQL container with Nomad and use Vultr Block Storage for persistence.
After running CSI driver and have a healthy plugin, I proceeded to add
vultr_block_storageandnomad_volumevia Terraform, the volume showed up in UI. Then I run a PostgreSQL job that has thevolumeandvolume_mountstanzas for the block storage volume, thecontainer-driveralways fails withmounting "<Path to CSI mounted volume path>" to rootfs at "<mount destination>" caused: stat <Path to CSI mounted volume>: no such file or directory.The mounting and unmounting happens in 5 seconds so I'm unable to confirm if everything is correctly set up. I ran the example busybox job from the vultr-csi repository and it fails too. Checked logs from CSI nodes containers, the volume should've been mounted correctly, and the
<Path to CSI mounted volume>provided by Nomad to containerd seem to be correct. I have a few templates that is going to be mounted into the volume but I don't think it would cause any problem.I don't know where to shoot this issue to, I guess I can start here?
Reproduction steps
Expected Result
Task starts.
Actual Result
Log from Vultr CSI driver:
Also after this failed task, Access mode and attachment mode are cleared from the volume for some reason:
Job file (if appropriate)
Nomad Server/Client logs (if appropriate)
Nothing fruitful.
The text was updated successfully, but these errors were encountered: