TaskDriver RPC Error: unknown service hashicorp.nomad.plugins.executor.proto.Executor

[pims]

Hi there!

tldr: I'm trying to write a TaskDriver plugin and seeing this error:

2019-06-16T16:15:33.293-0700 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=9a06aa9b-9810-eff6-8adb-1326d92a93b8 task=my-driver error="rpc error: code = Unknown desc = failed to launch command with executor: rpc error: code = Unimplemented desc = unknown service hashicorp.nomad.plugins.executor.proto.Executor"

Nomad version

Nomad v0.9.3 (c5e8b66)
Single Node, configured as both Client & Server.

Operating system and Environment details

Mac OS Mojave 10.14.4

Issue

When submitting a Job with my new driver, I get the following error:

2019-06-16T16:15:33.293-0700 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=9a06aa9b-9810-eff6-8adb-1326d92a93b8 task=my-task error="rpc error: code = Unknown desc = failed to launch command with executor: rpc error: code = Unimplemented desc = unknown service hashicorp.nomad.plugins.executor.proto.Executor"

The driver is properly registered according to this log statement:

 2019-06-16T16:15:23.365-0700 [INFO ] agent: detected plugin: name=my-driver type=driver plugin_version=0.1.0-dev

The task is started:

2019-06-16T16:15:33.264-0700 [INFO ] client.driver_mgr.nomad-driver-my-driver: starting my-driver task: driver=my-driver @module=my-driver args="[--foo bar --foobar false]" driver_cfg="{Foo:bar foobar:false}" timestamp=2019-06-16T16:15:33.264-0700

The error seems to come from:

executorConfig := &executor.ExecutorConfig{
		LogFile:     pluginLogFile,
		LogLevel:    "debug",
		FSIsolation: capabilities.FSIsolation == drivers.FSIsolationChroot,
	}

	exec, pluginClient, err := executor.CreateExecutor(
		d.logger.With("task_name", handle.Config.Name, "alloc_id", handle.Config.AllocID),
		d.nomadConfig, executorConfig)
	if err != nil {
		return nil, nil, fmt.Errorf("failed to create executor: %v", err)
	}

	user := cfg.User
	if user == "" {
		user = "nobody"
	}

	execCmd := &executor.ExecCommand{
		Cmd:            "/usr/local/bin/my-driver",
		Args:           args,
		Env:            cfg.EnvList(),
		User:           user,
		ResourceLimits: true,
		Resources:      cfg.Resources,
		TaskDir:        cfg.TaskDir().Dir,
		StdoutPath:     cfg.StdoutPath,
		StderrPath:     cfg.StderrPath,
		Mounts:         cfg.Mounts,
		Devices:        cfg.Devices,
	}

	ps, err := exec.Launch(execCmd)
	if err != nil {
		pluginClient.Kill()
		return nil, nil, fmt.Errorf("failed to launch command with executor: %v", err)
	}

Reproduction steps

The custom driver implementation is almost a verbatim copy of the Java TaskDriver, with a few changes to Config and executor.ExecCommand.Cmd

Job file (if appropriate)

see above

Nomad Server logs (if appropriate)

see above

I'm most likely doing something wrong, like not properly registering the driver or misunderstanding the executor logic. Thoughts? Thanks!

[preetapan]

@pims the LXC driver is an existing example of an external driver plugin implementation - https://github.com/hashicorp/nomad-driver-lxc

We are tracking this PR #5726 - which fixes a known issue with external drivers being able to use the executor package in Nomad correctly. You'll need to wait till that PR is merged and then recompile your driver after updating your nomad dependencies.

Will close this for now, feel free to open another issue once you make more progress after updating your code with PR #5726

[pims]

Thanks for the help @preetapan. I'll update the code with #5726.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.