Merge branch 'master' into 1475_rex-built-in-template-modules

.github/workflows/build_and_test.yml

@@ -42,14 +42,14 @@ jobs:
         uses: actions/cache@v2
         with:
           path: local
-          key: cache-build-modules-${{ env.OS_VERSION }}-${{ env.PERL_VERSION }}-${{ env.BUILD_TIMESTAMP }}
-          restore-keys: cache-build-modules-${{ env.OS_VERSION }}-${{ env.PERL_VERSION }}-
+          key: cache-build-modules-${{ secrets.CACHE_VERSION }}-${{ env.OS_VERSION }}-${{ env.PERL_VERSION }}-${{ env.BUILD_TIMESTAMP }}
+          restore-keys: cache-build-modules-${{ secrets.CACHE_VERSION }}-${{ env.OS_VERSION }}-${{ env.PERL_VERSION }}-
       - name: Cache perlcritic history
         uses: actions/cache@v2
         with:
           path: /tmp/cache/.perlcritic-history
-          key: cache-perlcritic-history-${{ env.BUILD_TIMESTAMP }}
-          restore-keys: cache-perlcritic-history-
+          key: cache-perlcritic-history-${{ secrets.CACHE_VERSION }}-${{ env.BUILD_TIMESTAMP }}
+          restore-keys: cache-perlcritic-history-${{ secrets.CACHE_VERSION }}-
       - name: Log perl information
         run: perl -V
       - name: Install packages

CONTRIBUTING.md

@@ -32,7 +32,20 @@ The first step of any change proposal is to open an issue about it. This gives a
 
 To cover the vast majority of typical discussion points in advance, there are predefined templates for issues and pull requests. Please make sure to use them in order to streamline the workflow.
 
-If something comes up that is not a good fit for the templates, that's probably already an early indicator that it should be discussed more closely. In this case please contact us first, or at least provide a reasoning about why the template had to be ignored in that specific case.
+If something comes up that is not a good fit for the templates, that's probably already an early indicator that it should be discussed more closely. In this case please [contact us](https://www.rexify.org/support/index.html) first, or at least provide a reasoning about why the template had to be ignored in that specific case.
+
+### Rex core vs extending Rex
+
+Strictly speaking the core competency of Rex is to execute commands, manage files, define tasks, and orchestrate their execution.
+
+Rex gained lots of other capabilities over time, and historically many of them landed in core as well. But it is often possible to extend Rex by only minimally changing the core, if at all. For example this includes adding support to:
+
+- manage new operating systems
+- new shell types
+- new virtualization methods
+- new cloud providers
+
+It is highly encouraged to add such new capabilities via their own extension modules outside the core. If in doubt, please check some of the common scenarios below, or [contact us](https://www.rexify.org/support/index.html).
 
 ### Cross platform support
 
@@ -183,6 +196,34 @@ It is generally recommended to:
 - [rebase](https://docs.github.com/en/github/using-git/about-git-rebase) your feature branch on top of the default branch if there are new commits since the feature branch has been created
 - use follow up/clean up commits on the same PR, but then please also [squash related commits](https://docs.github.com/en/github/using-git/about-git-rebase) together in the feature branch _before_ merging in order to keep a tidy history (in other words, no "tidy only" or "fix typo" commits are necessary)
 
+## Common scenarios
+
+### Add support to manage new operating systems
+
+Allowing Rex to manage a new OS requires the following steps:
+
+1. Teach rex about how to detect the given OS
+
+    - add a way to `Rex::Hardware::Host::get_operating_system()` to detect the given OS
+    - add a new `is_myos()` function to `Rex::Commands::Gather`
+
+1. Let Rex choose the proper package and service management modules for the given OS
+
+    - modify `Rex::Service` and `Rex::Pkg`
+
+1. Add new service and package management modules specific to the given OS
+
+    - add `Rex::Service::MyOS`
+    - add `Rex::Pkg::MyOS`
+
+### Add support for new virtualization methods
+
+Assuming the new virtualization method is called `MyVirt`, the following steps are required:
+
+- create the top-level `Rex::Virtualization::MyVirt` module which includes the constructor, and the documentation
+- create submodules for each virtualization command, e.g. `Rex::Virtualization::MyVirt::info`
+- implement the logic of the given command as the `execute` method
+
 ## Contribute to this guide
 
 If you think some of the information here is outdated, not clear enough, or have bugs, feel free to contribute to it too!

ChangeLog

@@ -4,10 +4,9 @@ Revision history for Rex
  [API CHANGES]
 
  [BUG FIXES]
- - Remove unused tasks array
+ - Detect invalid hostgroup expressions
 
  [DOCUMENTATION]
- - Update support channels
 
  [ENHANCEMENTS]
 
@@ -19,6 +18,27 @@ Revision history for Rex
 
  [REVISION]
 
+1.13.4 2021-07-05 Ferenc Erki <[email protected]>
+ [DOCUMENTATION]
+ - Add section about Rex core vs extending Rex
+ - Add common scenarios to contributing guide
+ - Clarify documentation of run command
+ - Clarify documentation of can_run command
+ - Fix Rex::Task synopsis
+ - Clarify documentation of rex script
+
+1.13.3.2-TRIAL 2021-07-03 Ferenc Erki <[email protected]>
+ [BUG FIXES]
+ - Fix local file copy on Windows
+
+1.13.3.1-TRIAL 2021-07-02 Ferenc Erki <[email protected]>
+ [BUG FIXES]
+ - Remove unused tasks array
+ - Fix file hooks when source option is used
+
+ [DOCUMENTATION]
+ - Update support channels
+
 1.13.3 2021-03-05 Ferenc Erki <[email protected]>
  [BUG FIXES]
  - Fix parsing debconf values containing colons

bin/rex

@@ -37,114 +37,149 @@ The C<rex> script can be used to execute tasks defined in a Rexfile from the com
 
 =head1 SYNOPSIS
 
-    bash# rex -h                      # Show usage
-    bash# rex -T                      # List tasks
-    bash# rex uname                   # Run the 'uname' task
-    bash# rex -H server[01..10] uname # Run the 'uname' task on all the specified hosts
-    bash# rex -G production uname     # Run 'uname' on hosts on the 'production' hostgroup
-    bash# rex deploy --gracefully     # Pass '--gracefully' to the 'deploy' task
+ rex -h                      # Show usage
+ rex -T                      # List tasks
+ rex uname                   # Run the 'uname' task
+ rex -H server[01..10] uname # Run the 'uname' task on all the specified hosts
+ rex -G production uname     # Run 'uname' on hosts on the 'production' hostgroup
+ rex deploy --gracefully     # Pass '--gracefully' to the 'deploy' task
 
 =head1 USAGE
 
-    rex [<options>] [-H <host>] [-G <group>] <task> [<task-options>]
-    rex -T[m|y|v] [<string>]
-
-    -b     Run batch
-    -e     Run the given code fragment
-    -E     Execute a task on the given environment
-    -G|-g  Execute a task on the given server groups
-    -H     Execute a task on the given hosts (space delimited)
-    -z     Execute a task on hosts from this command's output
-  
-    -K     Public key file for the ssh connection
-    -P     Private key file for the ssh connection
-    -p     Password for the ssh connection
-    -u     Username for the ssh connection
-  
-    -d     Show debug output
-    -ddd   Show more debug output (includes profiling output)
-    -m     Monochrome output: no colors
-    -o     Output format
-    -q     Quiet mode: no log output
-    -qw    Quiet mode: only output warnings and errors
-    -Q     Really quiet: output nothing
-  
-    -T     List tasks
-    -Ta    List all tasks, including hidden
-    -Tm    List tasks in machine-readable format
-    -Tv    List tasks verbosely
-    -Ty    List tasks in YAML format
-  
-    -c     Turn cache ON
-    -C     Turn cache OFF
-    -f     Use this file instead of Rexfile
-    -F     Force: disregard lock file
-    -h     Display this help message
-    -M     Load this module instead of Rexfile
-    -O     Pass additional options, like CMDB path
-    -s     Use sudo for every command
-    -S     Password for sudo
-    -t     Number of threads to use (aka 'parallelism' param)
-    -v     Display (R)?ex version
+ rex [<options>] [-H <host>] [-G <group>] <task> [<task-options>]
+ rex -T[m|y|v] [<string>]
+
+ -b     Run batch
+ -e     Run the given code fragment
+ -E     Execute a task on the given environment
+ -G|-g  Execute a task on the given server groups
+ -H     Execute a task on the given hosts (space delimited)
+ -z     Execute a task on hosts from this command's output
+
+ -K     Public key file for the ssh connection
+ -P     Private key file for the ssh connection
+ -p     Password for the ssh connection
+ -u     Username for the ssh connection
+
+ -d     Show debug output
+ -ddd   Show more debug output (includes profiling output)
+ -m     Monochrome output: no colors
+ -o     Output format
+ -q     Quiet mode: no log output
+ -qw    Quiet mode: only output warnings and errors
+ -Q     Really quiet: output nothing
+
+ -T     List tasks
+ -Ta    List all tasks, including hidden
+ -Tm    List tasks in machine-readable format
+ -Tv    List tasks verbosely
+ -Ty    List tasks in YAML format
+
+ -c     Turn cache ON
+ -C     Turn cache OFF
+ -f     Use this file instead of Rexfile
+ -F     Force: disregard lock file
+ -h     Display this help message
+ -M     Load this module instead of Rexfile
+ -O     Pass additional options, like CMDB path
+ -s     Use sudo for every command
+ -S     Password for sudo
+ -t     Number of threads to use (aka 'parallelism' param)
+ -v     Display (R)?ex version
 
 =head1 Rexfile
 
 When you run C<rex> it reads the file C<Rexfile> in the current working
-directory. A Rexfile consists of 2 major parts: Configuration and Task
-Definitions.
+directory. A Rexfile consists of 2 major parts: configuration and task
+definitions.
 
 =head2 Configuration
 
-=head3 Simple Authentication
+See all the available commands in L<Rex::Commands>.
 
-    user "bruce";
-    password "batman";
-    pass_auth;
+=head3 Simple authentication
 
-=head3 Key Authentication
+ user 'bruce';
+ password 'batman';
+ pass_auth;
 
-    private_key "/path/to/your/private/key.file";
-    public_key "/path/to/your/public/key.file";
+=head3 Key authentication
 
-=head3 Define Logging
+ private_key '/path/to/your/private/key.file';
+ public_key '/path/to/your/public/key.file';
+ key_auth;
 
-    logging to_file => "rex.log";
-    logging to_syslog => "local0";
+=head3 Define logging
+
+ logging to_file   => 'rex.log';
+ logging to_syslog => 'local0';
 
 =head3 Group your servers
 
 Rex gives you the ability to define groups of servers. Groups can be defined the Rexfile:
 
-    group "frontends" => "frontend01", "frontend02", "frontend03", "frontend04", "frontend[05..09]";
+ group 'frontends' => 'frontend01', 'frontend02', 'frontend[03..09]';
+
+Groups can also be defined in separate files, like C<server.ini>:
+
+ # server.ini
+ [frontends]
+ frontend[01..04]
 
-Groups can also be defined in a B<server.ini> file:
+ # Rexfile
+ use Rex::Group::Lookup::INI;
+ groups_file 'file.ini'
 
-    [frontends]
-    frontend[01..04]
+See L<Rex::Group::Lookup::INI> for more details, and check the C<Rex::Group::Lookup> namespace for other formats.
 
-=head2 Other Configuration
+=head2 Other configuration
 
-    timeout 10;    # ssh timeout
-    parallelism 2; # execute tasks in parallel
+ timeout 10;    # ssh timeout
+ parallelism 2; # execute tasks in parallel
 
 =head2 Defining tasks
 
 A basic task looks like this:
 
-    # task description
-    desc "This task tells you how long since the server was rebooted";
-
-    # task definition
-    task "shortname", sub {
-        say run "uptime";
-    };
+ # task description
+ desc 'This task tells you how long since the server was rebooted';
+
+ # task definition
+ task 'shortname', sub {
+   say run 'uptime';
+ };
+
+By default it will be targeted at the same host where `rex` is being executed.
+
+You can also set a default server as the task's target:
+
+ desc 'This is a long description of a task';
+ task 'shortname',
+   'frontend01',
+   sub {
+     say run 'uptime';
+   };
+
+or even a default server group:
+
+ desc 'This is a long description of a task';
+ task 'shortname',
+   group => 'frontends',
+   sub {
+     say run 'uptime';
+   };
+
+The task options from the command line will be passed to the task as well:
 
-You can also set a default server group:
+ # Rexfile
+ desc 'Get task options';
+ task 'get_task_options', sub {
+   my $task_options = shift;
+   my $option1 = $task_options->{option1};
+ };
 
-    desc "This is a long description of a task";
-    task "shortname", group => "frontends", sub {
-        say run "uptime";
-    };
+ # command line
+ rex get_task_options --option1=yes
 
 =head1 TAB COMPLETION
 

lib/Rex/Commands.pm

@@ -1786,9 +1786,12 @@ sub evaluate_hostname {
   if ( $rule =~ m/,/ ) {
     @ret = _evaluate_hostname_list( $start, $rule, $end );
   }
-  else {
+  elsif ( $rule =~ m/[.]{2}/msx ) {
     @ret = _evaluate_hostname_range( $start, $rule, $end );
   }
+  else {
+    croak('Invalid hostgroup expression');
+  }
 
   return @ret;
 }