Manual
- Command line options
- Commands from arguments
- Commands from stdin
- Command and initial arguments on command line
- Reading multiple inputs
- Parallelism
- Dry run
- Debug logging
- Error handling
- Timeout
- Path cache
- Progress bar
- Regular Expression
- Shell Commands
- Bash Function
$ rust-parallel --help
Execute commands in parallel
By Aaron Riekenberg <[email protected]>
https://github.com/aaronriekenberg/rust-parallel
https://crates.io/crates/rust-parallel
Usage: rust-parallel [OPTIONS] [COMMAND_AND_INITIAL_ARGUMENTS]...
Arguments:
[COMMAND_AND_INITIAL_ARGUMENTS]...
Optional command and initial arguments.
If this contains 1 or more ::: delimiters the cartesian product of arguments from all groups are run.
Options:
-d, --discard-output <DISCARD_OUTPUT>
Discard output for commands
Possible values:
- stdout: Redirect stdout for commands to /dev/null
- stderr: Redirect stderr for commands to /dev/null
- all: Redirect stdout and stderr for commands to /dev/null
-i, --input-file <INPUT_FILE>
Input file or - for stdin. Defaults to stdin if no inputs are specified
-j, --jobs <JOBS>
Maximum number of commands to run in parallel, defauts to num cpus
[default: 8]
-0, --null-separator
Use null separator for reading input files instead of newline
-p, --progress-bar
Display progress bar
-r, --regex <REGEX>
Apply regex pattern to inputs
-s, --shell
Use shell mode for running commands.
Each command line is passed to "<shell-path> <shell-argument>" as a single argument.
-t, --timeout-seconds <TIMEOUT_SECONDS>
Timeout seconds for running commands. Defaults to infinite timeout if not specified
--channel-capacity <CHANNEL_CAPACITY>
Input and output channel capacity, defaults to num cpus * 2
[default: 16]
--disable-path-cache
Disable command path cache
--dry-run
Dry run mode
Do not actually run commands just log.
--exit-on-error
Exit on error mode
Exit immediately when a command fails.
--no-run-if-empty
Do not run commands for empty buffered input lines
--shell-path <SHELL_PATH>
Path to shell to use for shell mode
[default: /bin/bash]
--shell-argument <SHELL_ARGUMENT>
Argument to shell for shell mode
[default: -c]
-h, --help
Print help (see a summary with '-h')
-V, --version
Print version
The ::: separator can be used to run the Cartesian Product of command line arguments. This is similar to the ::: behavior in GNU Parallel.
$ rust-parallel echo ::: A B ::: C D ::: E F G
A C E
A C F
A D G
A D F
A D E
A C G
B C E
B C F
B D E
B C G
B D F
B D G
$ rust-parallel echo hello ::: larry curly moe
hello moe
hello curly
hello larry
# run gzip -k on all *.html files in current directory
$ rust-parallel gzip -k ::: *.html
When using commands from arguments, numbered variables {0}, {1}, etc are automatically available based on the number of arguments. {0} will be replaced by the entire input line, and other groups match individual argument groups. This is useful for building more complex command lines. For example:
$ rust-parallel echo group0={0} group1={1} group2={2} group3={3} group2again={2} ::: A B ::: C D ::: E F G
group0=A C E group1=A group2=C group3=E group2again=C
group0=A C G group1=A group2=C group3=G group2again=C
group0=A C F group1=A group2=C group3=F group2again=C
group0=A D G group1=A group2=D group3=G group2again=D
group0=B C F group1=B group2=C group3=F group2again=C
group0=A D F group1=A group2=D group3=F group2again=D
group0=A D E group1=A group2=D group3=E group2again=D
group0=B C E group1=B group2=C group3=E group2again=C
group0=B D E group1=B group2=D group3=E group2again=D
group0=B C G group1=B group2=C group3=G group2again=C
group0=B D F group1=B group2=D group3=F group2again=D
group0=B D G group1=B group2=D group3=G group2again=D
Internally these variables are implemented using an auto-generated regular expression. If a regular expression is manually specified this will override the auto-generated one.
Run complete commands from stdin.
$ cat >./test <<EOL
echo hi
echo there
echo how
echo are
echo you
EOL
$ cat test | rust-parallel
hi
are
you
there
how
Here md5 -s will be prepended to each input line to form a command like md5 -s aal
$ head -100 /usr/share/dict/words | rust-parallel md5 -s | head -10
MD5 ("Aani") = e9b22dd6213c3d29648e8ad7a8642f2f
MD5 ("aardvark") = 88571e5d5e13a4a60f82cea7802f6255
MD5 ("aa") = 4124bc0a9335c27f086f24ba207a4912
MD5 ("aalii") = 0a1ea2a8d75d02ae052f8222e36927a5
MD5 ("a") = 0cc175b9c0f1b6a831c399e269772661
MD5 ("A") = 7fc56270e7a70fa81a5935b72eacbe29
MD5 ("aam") = 35c2d90f7c06b623fe763d0a4e5b7ed9
MD5 ("aal") = ff45e881572ca2c987460932660d320c
MD5 ("Aaronic") = 0390cf1718c4f2d76f770c7c35b40c50
MD5 ("aardwolf") = 66a4a1a2b442e8d218e8e99100069877
By default rust-parallel reads input from stdin only. The -i option can be used 1 or more times to override this behavior. -i - means read from stdin, -i ./test means read from the file ./test:
$ cat >./test <<EOL
foo
bar
baz
EOL
$ head -5 /usr/share/dict/words | rust-parallel -i - -i ./test echo
A
aa
aal
bar
aalii
baz
foo
a
By default the number of parallel jobs to run simulatenously is the number of cpus detected at run time.
This can be override with the -j/--jobs option.
With -j5 all echo commands below run in parallel.
With -j1 all jobs run sequentially.
$ rust-parallel -j5 echo ::: hi there how are you
how
hi
you
there
are
$ rust-parallel -j1 echo ::: hi there how are you
hi
there
how
are
you
Use option --dry-run for dry run mode.
In this mode the commands that would be run are ouput as info level logs.
No commands are actually run - this is useful for testing before running a job.
$ rust-parallel --dry-run echo ::: hi there how are you
2024-05-04T16:08:20.294681Z INFO rust_parallel::command: cmd="/bin/echo",args=["hi"],line=command_line_args:1
2024-05-04T16:08:20.294705Z INFO rust_parallel::command: cmd="/bin/echo",args=["there"],line=command_line_args:2
2024-05-04T16:08:20.294713Z INFO rust_parallel::command: cmd="/bin/echo",args=["how"],line=command_line_args:3
2024-05-04T16:08:20.294721Z INFO rust_parallel::command: cmd="/bin/echo",args=["are"],line=command_line_args:4
2024-05-04T16:08:20.294728Z INFO rust_parallel::command: cmd="/bin/echo",args=["you"],line=command_line_args:5
Set environment variable RUST_LOG=debug to see debug output.
This logs structured information about command line arguments and commands being run.
Recommend enabling debug logging for all examples to understand what is happening in more detail.
$ RUST_LOG=debug rust-parallel echo ::: hi there how are you | grep command_line_args | head -1
2024-05-04T16:08:20.297176Z DEBUG try_main: rust_parallel::command_line_args: command_line_args = CommandLineArgs { discard_output: None, input_file: [], jobs: 8, null_separator: false, progress_bar: false, regex: None, shell: false, timeout_seconds: None, channel_capacity: 16, disable_path_cache: false, dry_run: false, exit_on_error: false, no_run_if_empty: false, shell_path: "/bin/bash", shell_argument: "-c", command_and_initial_arguments: ["echo", ":::", "hi", "there", "how", "are", "you"] }
$ RUST_LOG=debug rust-parallel echo ::: hi there how are you | grep command_line_args:1
2024-05-04T16:08:20.302757Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1}: rust_parallel::command: begin run
2024-05-04T16:08:20.302961Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 child_pid=40443}: rust_parallel::command: spawned child process, awaiting completion
2024-05-04T16:08:20.303670Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 child_pid=40443}: rust_parallel::command: command exit status = exit status: 0
2024-05-04T16:08:20.303681Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 child_pid=40443}: rust_parallel::command: end run
The following are considered command failures and error will be logged:
- Spawn error
- Timeout
- I/O error
- Command exits with non-0 status
By default rust-parallel runs all commands even if failures occur.
When rust-parallel terminates, if any command failed it logs failure metrics and exits with status 1.
Here we try to use cat to show non-existing files A, B, and C, so each command exits with status 1:
$ rust-parallel cat ::: A B C
cat: A: No such file or directory
2024-05-04T16:08:20.308001Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["A"],line=command_line_args:1 exit_status=1
cat: C: No such file or directory
2024-05-04T16:08:20.308071Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["C"],line=command_line_args:3 exit_status=1
cat: B: No such file or directory
2024-05-04T16:08:20.308132Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["B"],line=command_line_args:2 exit_status=1
2024-05-04T16:08:20.308165Z ERROR rust_parallel: fatal error in main: command failures: commands_run=3 total_failures=3 spawn_errors=0 timeouts=0 io_errors=0 exit_status_errors=3
$ echo $?
1
The --exit-on-error option can be used to exit after one command fails.
rust-parallel waits for in-progress commands to finish before exiting and then exits with status 1.
$ head -100 /usr/share/dict/words | rust-parallel --exit-on-error cat
cat: aa: No such file or directory
2024-05-04T16:08:20.311987Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aa"],line=stdin:3 exit_status=1
cat: A: No such file or directory
2024-05-04T16:08:20.312064Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["A"],line=stdin:1 exit_status=1
cat: a: No such file or directory
2024-05-04T16:08:20.312329Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["a"],line=stdin:2 exit_status=1
cat: Aani: No such file or directory
2024-05-04T16:08:20.312412Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["Aani"],line=stdin:7 exit_status=1
cat: aam: No such file or directory
2024-05-04T16:08:20.312521Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aam"],line=stdin:6 exit_status=1
cat: aardvark: No such file or directory
2024-05-04T16:08:20.312615Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aardvark"],line=stdin:8 exit_status=1
cat: aal: No such file or directory
2024-05-04T16:08:20.312681Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aal"],line=stdin:4 exit_status=1
cat: aalii: No such file or directory
2024-05-04T16:08:20.312736Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aalii"],line=stdin:5 exit_status=1
cat: aardwolf: No such file or directory
2024-05-04T16:08:20.312777Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aardwolf"],line=stdin:9 exit_status=1
2024-05-04T16:08:20.313071Z ERROR rust_parallel: fatal error in main: command failures: commands_run=9 total_failures=9 spawn_errors=0 timeouts=0 io_errors=0 exit_status_errors=9
$ echo $?
1
The -t/--timeout-seconds option can be used to specify a command timeout in seconds. If any command times out this is considered a command failure (see error handling).
$ rust-parallel -t 0.5 sleep ::: 0 3 5
2024-05-04T16:08:20.818579Z ERROR rust_parallel::command: child process error command: cmd="/bin/sleep",args=["5"],line=command_line_args:3 error: timeout: deadline has elapsed
2024-05-04T16:08:20.818583Z ERROR rust_parallel::command: child process error command: cmd="/bin/sleep",args=["3"],line=command_line_args:2 error: timeout: deadline has elapsed
2024-05-04T16:08:20.819071Z ERROR rust_parallel: fatal error in main: command failures: commands_run=3 total_failures=2 spawn_errors=0 timeouts=2 io_errors=0 exit_status_errors=0
$ echo $?
1
By default as commands are run the full paths are resolved using which. Resolved paths are stored in a cache to prevent duplicate resolutions. This is generally good for performance.
The path cache can be disabled using the --disable-path-cache option.
The -p/--progress-bar option can be used to enable a graphical progress bar.
This is best used for commands which are running for at least a few seconds, and which do not produce output to stdout or stderr. In the below commands -d all is used to discard all output from commands run.
Progress styles can be chosen with the PROGRESS_STYLE environment variable. If PROGRESS_STYLE is not set it defaults to light_bg.
The following progress styles are available:
-
PROGRESS_STYLE=light_bggood for light terminal background with colors, spinner, and steady tick enabled:
-
PROGRESS_STYLE=dark_bggood for dark terminal background with colors, spinner, and steady tick enabled:
-
PROGRESS_STYLE=simplegood for simple or non-ansi terminals/jobs with colors, spinner, and steady tick disabled:
Regular expressions can be specified by the -r or --regex command line argument.
Named or numbered capture groups are expanded with data values from the current input before the command is executed.
In these examples using command line arguments {url} and {filename} are named capture groups. {0} is a numbered capture group.
$ rust-parallel -r '(?P<url>.*),(?P<filename>.*)' echo got url={url} filename={filename} ::: URL1,filename1 URL2,filename2
got url=URL1 filename=filename1
got url=URL2 filename=filename2
$ rust-parallel -r '(?P<url>.*) (?P<filename>.*)' echo got url={url} filename={filename} full input={0} ::: URL1 URL2 ::: filename1 filename2
got url=URL2 filename=filename1 full input=URL2 filename1
got url=URL1 filename=filename2 full input=URL1 filename2
got url=URL1 filename=filename1 full input=URL1 filename1
got url=URL2 filename=filename2 full input=URL2 filename2
In the next example input file arguments {0} {1} {2} {3} are numbered capture groups, and the input is a csv file:
$ cat >./test <<EOL
foo,bar,baz
foo2,bar2,baz2
foo3,bar3,baz3
EOL
$ cat test | rust-parallel -r '(.*),(.*),(.*)' echo got arg1={1} arg2={2} arg3={3} full input={0}
got arg1=foo arg2=bar arg3=baz full input=foo,bar,baz
got arg1=foo2 arg2=bar2 arg3=baz2 full input=foo2,bar2,baz2
got arg1=foo3 arg2=bar3 arg3=baz3 full input=foo3,bar3,baz3
All occurrences of capture groups are replaced as exact strings. Surrounding characters have no effect on this.
This means capture groups can be nested with other { or } characters such as when building json:
$ cat >./test <<EOL
1,2,3
4,5,6
7,8,9
EOL
$ cat test | rust-parallel -r '(.*),(.*),(.*)' echo '{"one":{1},"two":{2},"nested_object":{"three":{3}}}'
{"one":7,"two":8,"nested_object":{"three":9}}
{"one":1,"two":2,"nested_object":{"three":3}}
{"one":4,"two":5,"nested_object":{"three":6}}
Shell commands can be written using -s shell mode.
Multiline commands can be written using ;.
Environment variables, $ characters, nested commands and much more are possible:
$ rust-parallel -s -r '(?P<arg1>.*) (?P<arg2>.*)' 'FOO={arg1}; BAR={arg2}; echo "FOO = ${FOO}, BAR = ${BAR}, shell pid = $$, date = $(date)"' ::: A B ::: CAT DOG
FOO = B, BAR = CAT, shell pid = 40492, date = Sat May 4 11:08:20 CDT 2024
FOO = B, BAR = DOG, shell pid = 40493, date = Sat May 4 11:08:20 CDT 2024
FOO = A, BAR = DOG, shell pid = 40491, date = Sat May 4 11:08:20 CDT 2024
FOO = A, BAR = CAT, shell pid = 40490, date = Sat May 4 11:08:20 CDT 2024
-s shell mode can be used to invoke an arbitrary bash function.
Similar to normal commands bash functions can be called using stdin, input files, or from command line arguments.
Define a bash fuction logargs that logs all arguments and make visible with export -f:
$ logargs() {
echo "logargs got $@"
}
$ export -f logargs
$ rust-parallel -s logargs ::: A B C ::: D E F
logargs got A D
logargs got A E
logargs got B F
logargs got B D
logargs got B E
logargs got A F
logargs got C E
logargs got C D
logargs got C F
$ cat >./test <<EOL
logargs hello alice
logargs hello bob
logargs hello charlie
EOL
$ cat test | rust-parallel -s
logargs got hello alice
logargs got hello charlie
logargs got hello bob
$ cat >./test <<EOL
alice
bob
charlie
EOL
$ cat test | rust-parallel -s logargs hello
logargs got hello alice
logargs got hello bob
logargs got hello charlie