Completion for program with nested subcommands and multiple kinds of arguments

[KiruyaMomochi]

I'm writing a completion generator for spack.

Background

Spack is a package management tool written in Python. It uses argparse to handle arguments. It's command can have nested subcommands, making it very complicated, for example:

spack -v --color always view --dependencies true add     ./ fish
#     ^optional args    ^subcommand              ^subcmd ^two positional args

Each level of command can have its own options, and each option may have its own arguments.
It can also contain more than one positional arguments.

They have already provided script for generating bash completion, but that just can't handle any option with argument. Such an argument corrupts all following completion.

However I prefer fish than bash, so I'm trying to take that as an example, and write one for fish.

Attempt

To make complicated nested subcommands easier to parse, I have defined some functions.

# Get optional argument specifications for given command
# For example, calling 
#     __fish_spack_optspecs_get spack add
# will in turn calls `__fish_spack_add_optspecs`, returning something like 
#     string join \n  "h/help" "l/list-name="
#
# Functions like `__fish_spack_optspecs_get` is generated automatically 
# from spack.
function __fish_spack_optspecs_get
    # Get the function name
    set -l cmd __fish_(string replace -a -- ' ' '_' "$argv")_optspecs
    # Return 1 if optspecs is not defined
    type -q $cmd
    or return 1
    # Run the function to get optspecs
    $cmd
end

# Extract the command from current command line,
# removing all known options and option arguments.
# For example, if command line is
#     spack --color always view --dependencies true add ./
# we will get
#     spack view add ./
function __fish_spack_extract_command
    # Obtain the command name
    set -l line (commandline -opc)
    set -l cmd

    # Recursively remove all options for the command
    # For example:
    # 1. $line == 'spack --color always view --dependencies true add ./ fish'
    # 2. $line == 'view --dependencies true add ./ fish'
    # 3. $line == 'add ./ fish'
    while set -q line[1]
        # Get current command
        set -a cmd $line[1]
        set -e line[1]

        # Get option argument specification of this token
        set -l optspecs (__fish_spack_optspecs_get "$cmd")
        set -q optspecs[1]
        or break

        # Use argparse to remove option and its argument
        argparse -i -s $optspecs -- $line 2>/dev/null
        or break

        # Set line to remaining command line
        set line $argv

        # If command is help, stop
        set -q _flag_help; and return 1

        # If command is a option, stop
        # Q: Or we can ignore it and continue parsing?
        string match -q -- '-*' $argv[1]
        and break
    end
    
    # Return extracted command and remaining command line
    echo $cmd (string match -a -v -- '-*' $line)
end

# Check if current command is the same as given command
# For example, if command line is
#     spack --color always view --dependencies true add
# executing
#     __fish_spack_using_command spack view add
# will return 0, indicating that the command is the same.
function __fish_spack_using_command
    set -l cmd (__fish_spack_extract_command)
    test -z "$cmd"
    and return 1

    test "$cmd" = "$argv"
    and return 0
end

Then I generated completion for all commands from spack's python code, the generated code has 1000+ completes, some look like:

# spack view add
# 2 Positionals: [('path', 'path to file system view directory'), ('spec', 'seed specs of the packages to view')]
# TODO: How to make completion for second positional argument?
# ['-h', '--help'] -> 'help': 0
complete -c spack -n "__fish_spack_using_command spack view add" -s h -l help -d "show this help message and exit"
# ['--projection-file'] -> 'projection_file': None
complete -c spack -n "__fish_spack_using_command spack view add" -l projection-file -r -d "Initialize view using projections from file."
# ['-i', '--ignore-conflicts'] -> 'ignore_conflicts': 0
complete -c spack -n "__fish_spack_using_command spack view add" -s i -l ignore-conflicts

Problem

These completes will runs __fish_spack_using_command over and over again, make completion time very slow.

To solve this, I have considered using a single function to handle all completions, but then I can't take advantage of fish's complete -s/-l feature for argument completion.

During debugging I also find most time are spent on __fish_spack_extract_command, so I have tried to cache its result in a global variable, and return the same value as long as command line does not change. It works well, but I hope there is a better way.

For example, if there is a function that run only once every time user press tab, the problem can be easier to solve.

Related

#7107

[faho]

What you want to do is something like the git completions do - see

fish-shell/share/completions/git.fish

Lines 561 to 578 in 959e17f

	function __fish_git_needs_command
	# Figure out if the current invocation already has a command.
	set -l cmd (commandline -opc)
	set -e cmd[1]
	argparse -s (__fish_git_global_optspecs) -- $cmd 2>/dev/null
	or return 0
	# These flags function as commands, effectively.
	set -q _flag_version; and return 1
	set -q _flag_html_path; and return 1
	set -q _flag_man_path; and return 1
	set -q _flag_info_path; and return 1
	if set -q argv[1]
	# Also print the command, so this can be used to figure out what it is.
	echo $argv[1]
	return 1
	end
	return 0
	end

.

This uses a function that tells argparse to parse the options from the current commandline, which leaves the command intact.

For git it's easier because it only has a few global options, so you can then use argparse --stop-nonopt to make it stop at the first non-option (i.e. the first command). You can then pass that to specific functions depending on the subcommand. It's also simple because e.g. git stash has sub-subcommands, but no options can come between the stash and the sub-subcommand -

fish-shell/share/completions/git.fish

Lines 646 to 658 in 959e17f

	function __fish_git_stash_using_command
	set -l cmd (commandline -opc)
	__fish_git_using_command stash
	or return 2
	# The word after the stash command _must_ be the subcommand
	set cmd $cmd[(contains -i -- "stash" $cmd)..-1]
	set -e cmd[1]
	set -q cmd[1]
	or return 1
	contains -- $cmd[1] $argv
	and return 0
	return 1
	end

.

Really, the only options you need to define for this pass are the ones that

1. are valid before your subcommand (e.g. git --follow log is invalid, it has to be git log --follow)
2. have options - you could use argparse --ignore-unknown to ignore the rest because they are irrelevant

And that is, already, roughly what you are doing. That while loop seems overly complicated to me, I would roughly structure it like

set -l line (commandline -opc)[2..-1] # remove the "spack" because we know we're spack

# the globally valid options are checked here
# you might also want to leave out the `-s` to remove them from the entire commandline,
# if they are valid everywhere
argparse -i -s $optspecs -- $line

# options that are like commands - --help, --version and such
set -q _flag_help; and return 1

set -q argv[1]
or return 1 # no command, so a faulty return

echo $argv[1]

switch $argv[1]
    case view
        argparse -i -s dependencies= -- $argv
        set -q argv[1]
        or return 0 # we have *a command*
        echo -- $argv[1]
        # if this can nest further, we would add another switch here
        # if that becomes too much, you can extract this into another function
        # __fish_spack_extract_view_command
    case some-other-subcommand
        # ...
end

(yes, this is overly complicated, which is why #7107 exists)

[faho]

Question was answered, closing