Initial version of integration tests (#330)

* Initial commit for integration tests. Experimental. Playing with potential syntax

* Some experimental code to setup tests

* Piecewise building of CMakeLists

* First check

* Alternative approach. Using ruby's test/unit

* Parse CMakeCache. Separate lib

* First integration test

* Latest Format.cmake. Passing style

* Allow user-provided integration test dir. Allow reuse

* Separate class with utils for cache (no longer pure Hash)

* Allow running of tests from any dir

* Add integration tests to CI

* Use an in-source integration test directory

* Allow relative integration test dir from env

* Custom assertion for a success of CommandResult

* Windows-latest-latest

* Enrich CMakeCache class with more CPM data

* Added test for CPM-specific CMakeCache values

* Style

* Style

* test_update_single_package

* WiP for source cache test

* Small source_cache test

* Style

* Moved env clean to cleanup to make setup methods simpler (not require super)

* WiP for integration test documentation

* WiP for integration test documentation

* Project file creation tweaks

* Split docs into multiple files. Complete tutorial. Reference.

* Tips

* Typo

* Setup Ruby inistead of requiring windows-2022

* Revert "Setup Ruby inistead of requiring windows-2022"

This reverts commit 8aa2732.

.github/workflows/test.yml

@@ -13,7 +13,10 @@ jobs:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        os: [ubuntu-latest, windows-latest, macos-latest]
+        # windows-latest is windows-2019 which carries a pretty old version of ruby (2.5)
+        # we need at least ruby 2.7 for the tests
+        # instead of dealing with installing a modern version of ruby on 2019, we'll just use windows-2022 here
+        os: [ubuntu-latest, windows-2022, macos-latest]
 
     steps:
       - name: clone
@@ -23,3 +26,8 @@ jobs:
         run: |
           cmake -Htest -Bbuild/test
           cmake --build build/test --target test-verbose
+
+      - name: integration tests
+        run: ruby test/integration/runner.rb
+        env:
+          CPM_INTEGRATION_TEST_DIR: ./build/integration

test/integration/.gitignore

@@ -0,0 +1,2 @@
+# Use this to have a local integration test which is for personal experiments
+test_local.rb

test/integration/README.md

@@ -0,0 +1,44 @@
+# CPM.cmake Integration Tests
+
+The integration tests of CPM.cmake are written in Ruby. They use a custom integration test framework which extends the [Test::Unit](https://www.rubydoc.info/github/test-unit/test-unit/Test/Unit) library.
+
+They require Ruby 2.7.0 or later.
+
+## Running tests
+
+To run all tests from the repo root execute:
+
+```
+$ ruby test/integration/runner.rb
+```
+
+The runner will run all tests and generate a report of the exeuction.
+
+The current working directory doesn't matter. If you are in `<repo-root>/test/integration`, you can run simply `$ ruby runner.rb`.
+
+You can execute with `--help` (`$ ruby runner.rb --help`) to see various configuration options of the runner like running individual tests or test cases, or ones that match a regex.
+
+The tests themselves are situated in the Ruby scripts prefixed with `test_`. `<repo-root>/test/integration/test_*`. You can also run an individual test script. For example to only run the **basics** test case, you can execute `$ ruby test_basics.rb`
+
+The tests generate CMake scripts and execute CMake and build toolchains. By default they do this in a directory they generate in your temp path (`/tmp/cpm-test/` on Linux). You can configure the working directory of the tests with an environment variable `CPM_INTEGRATION_TEST_DIR`. For example `$ CPM_INTEGRATION_TEST_DIR=~/mycpmtest; ruby runner.rb`
+
+## Writing tests
+
+Writing tests makes use of the custom integration test framework in `lib.rb`. It is a relatively small extension of Ruby's Test::Unit library.
+
+### The Gist
+
+* Tests cases are Ruby scripts in this directory. The file names must be prefixed with `test_`
+* The script should `require_relative './lib'` to allow for individual execution (or else if will only be executable from the runner)
+* A test case file should contain a single class which inherits from `IntegrationTest`. It *can* contain multiple classes, but that's bad practice as it makes individual execution harder and implies a dependency between the classes.
+* There should be no dependency between the test scripts. Each should be executable individually and the order in which multiple ones are executed mustn't matter.
+* The class should contain methods, also prefixed with `test_` which will be executed by the framework. In most cases there would be a single test method per class.
+* In case there are multiple test methods, they will be executed in the order in which they are defined.
+* The test methods should contain assertions which check for the expected state of things at varous points of the test's execution.
+
+### More
+
+* [A basic tutorial on writing integration tests.](tutorial.md)
+* [A brief reference of the integration test framework](reference.md)
+* Make sure you're familiar with the [idiosyncrasies](idiosyncrasies.md) of writing integration tests
+* [Some tips and tricks](tips.md)

test/integration/idiosyncrasies.md

@@ -0,0 +1,98 @@
+# Notable Idiosyncrasies When Writing Integration Tests
+
+As an integration test framework based on a unit test framework the one created for CPM.cmake suffers from several idiosyncrasies. Make sure you familiarize yourself with them before writing integration tests.
+
+## No shared instance variables between methods
+
+The runner will create an instance of the test class for each test method. This means that instance variables defined in a test method, *will not* be visible in another. For example:
+
+```ruby
+class MyTest < IntegrationTest
+  def test_something
+    @x = 123
+    assert_equal 123, @x # Pass. @x is 123
+  end
+  def test_something_else
+    assert_equal 123, @x # Fail! @x would be nil here
+  end
+end
+```
+
+There are hacks around sharing Ruby state between methods, but we choose not to use them. If you want to initialize something for all test methods, use `setup`.
+
+```ruby
+class MyTest < IntegrationTest
+  def setup
+    @x = 123
+  end
+  def test_something
+    assert_equal 123, @x # Pass. @x is 123 thanks to setup
+  end
+  def test_something_else
+    assert_equal 123, @x # Pass. @x is 123 thanks to setup
+  end
+end
+```
+
+## `IntegrationTest` makes use of `Test::Unit::TestCase#cleanup`
+
+After each test method the `cleanup` method is called thanks to Test::Unit. If you require the use of `cleanup` in your own tests, make sure you call `super` to also run `IntegrationTest#cleanup`.
+
+```ruby
+class MyTest < IntegrationTest
+  def cleanup
+    super
+    my_cleanup
+  end
+  # ...
+end
+```
+
+## It's better to have assertions in test methods as opposed to helper methods
+
+Test::Unit will display a helpful message if an assertion has failed. It will also include the line of code in the test method which caused the failure. However if an assertion is not in the test method, it will display the line which calls the method in which it is. So, please try, to have most assertions in test methods (though we acknowledge that in certain cases this is not practical). For example, if you only require scopes, try using lambdas.
+
+Instead of this:
+
+```ruby
+class MyTest < IntegrationTest
+  def test_something
+    do_a
+    do_b
+    do_c
+  end
+  def do_a
+    # ...
+  end
+  def do_b
+    # ...
+    assert false # will display failed line as "do_b"
+  end
+  def do_c
+    # ...
+  end
+end
+```
+
+...write this:
+
+```ruby
+class MyTest < IntegrationTest
+  def test_something
+    do_a = -> {
+      # ...
+    }
+    do_b = -> {
+      # ...
+      assert false # will display failed line as "assert false"
+    }
+    do_c = -> {
+      # ...
+    }
+
+    do_a.()
+    do_b.()
+    do_c.()
+  end
+end
+```

test/integration/lib.rb

@@ -0,0 +1,200 @@
+require 'fileutils'
+require 'open3'
+require 'tmpdir'
+require 'test/unit'
+
+module TestLib
+  TMP_DIR = File.expand_path(ENV['CPM_INTEGRATION_TEST_DIR'] || File.join(Dir.tmpdir, 'cpm-test', Time.now.strftime('%Y_%m_%d-%H_%M_%S')))
+  CPM_PATH = File.expand_path('../../cmake/CPM.cmake', __dir__)
+
+  TEMPLATES_DIR = File.expand_path('templates', __dir__)
+
+  # Environment variables which are read by cpm
+  CPM_ENV = %w(
+    CPM_USE_LOCAL_PACKAGES
+    CPM_LOCAL_PACKAGES_ONLY
+    CPM_DOWNLOAD_ALL
+    CPM_DONT_UPDATE_MODULE_PATH
+    CPM_DONT_CREATE_PACKAGE_LOCK
+    CPM_INCLUDE_ALL_IN_PACKAGE_LOCK
+    CPM_USE_NAMED_CACHE_DIRECTORIES
+    CPM_SOURCE_CACHE
+  )
+  def self.clear_env
+    CPM_ENV.each { ENV[_1] = nil }
+  end
+end
+
+puts "Warning: test directory '#{TestLib::TMP_DIR}' already exists" if File.exist?(TestLib::TMP_DIR)
+raise "Cannot find 'CPM.cmake' at '#{TestLib::CPM_PATH}'" if !File.file?(TestLib::CPM_PATH)
+
+puts "Running CPM.cmake integration tests"
+puts "Temp directory: '#{TestLib::TMP_DIR}'"
+
+# Clean all CPM-related env vars
+TestLib.clear_env
+
+class Project
+  def initialize(src_dir, bin_dir)
+    @src_dir = src_dir
+    @bin_dir = bin_dir
+  end
+
+  attr :src_dir, :bin_dir
+
+  def create_file(target_path, text, args = {})
+    target_path = File.join(@src_dir, target_path)
+
+    # tweak args
+    args[:cpm_path] = TestLib::CPM_PATH if !args[:cpm_path]
+    args[:packages] = [args[:package]] if args[:package] # if args contain package, create the array
+    args[:packages] = args[:packages].join("\n") if args[:packages] # join all packages if any
+
+    File.write target_path, text % args
+  end
+
+  def create_file_from_template(target_path, source_path, args = {})
+    source_path = File.join(@src_dir, source_path)
+    raise "#{source_path} doesn't exist" if !File.file?(source_path)
+    src_text = File.read source_path
+    create_file target_path, src_text, args
+  end
+
+  # common function to create ./CMakeLists.txt from ./lists.in.cmake
+  def create_lists_from_default_template(args = {})
+    create_file_from_template 'CMakeLists.txt', 'lists.in.cmake', args
+  end
+
+  CommandResult = Struct.new :out, :err, :status
+  def configure(extra_args = '')
+    CommandResult.new *Open3.capture3("cmake -S #{@src_dir} -B #{@bin_dir} #{extra_args}")
+  end
+  def build(extra_args = '')
+    CommandResult.new *Open3.capture3("cmake --build #{@bin_dir} #{extra_args}")
+  end
+
+  class CMakeCache
+    class Entry
+      def initialize(val, type, advanced, desc)
+        @val = val
+        @type = type
+        @advanced = advanced
+        @desc = desc
+      end
+      attr :val, :type, :advanced, :desc
+      alias_method :advanced?, :advanced
+      def inspect
+        "(#{val.inspect} #{type}" + (advanced? ? ' ADVANCED)' : ')')
+      end
+    end
+
+    Package = Struct.new(:ver, :src_dir, :bin_dir)
+
+    def self.from_dir(dir)
+      entries = {}
+      cur_desc = ''
+      file = File.join(dir, 'CMakeCache.txt')
+      return nil if !File.file?(file)
+      File.readlines(file).each { |line|
+        line.strip!
+        next if line.empty?
+        next if line.start_with? '#' # comment
+        if line.start_with? '//'
+          cur_desc += line[2..]
+        else
+          m = /(.+?)(-ADVANCED)?:([A-Z]+)=(.*)/.match(line)
+          raise "Error parsing '#{line}' in #{file}" if !m
+          entries[m[1]] = Entry.new(m[4], m[3], !!m[2], cur_desc)
+          cur_desc = ''
+        end
+      }
+      CMakeCache.new entries
+    end
+
+    def initialize(entries)
+      @entries = entries
+
+      package_list = self['CPM_PACKAGES']
+      @packages = if package_list
+        # collect package data
+        @packages = package_list.split(';').map { |name|
+          [name, Package.new(
+            self["CPM_PACKAGE_#{name}_VERSION"],
+            self["CPM_PACKAGE_#{name}_SOURCE_DIR"],
+            self["CPM_PACKAGE_#{name}_BINARY_DIR"]
+          )]
+        }.to_h
+      else
+        {}
+      end
+    end
+
+    attr :entries, :packages
+
+    def [](key)
+      e = @entries[key]
+      return nil if !e
+      e.val
+    end
+  end
+  def read_cache
+    CMakeCache.from_dir @bin_dir
+  end
+end
+
+class IntegrationTest < Test::Unit::TestCase
+  self.test_order = :defined # run tests in order of defintion (as opposed to alphabetical)
+
+  def cleanup
+    # Clear cpm-related env vars which may have been set by the test
+    TestLib.clear_env
+  end
+
+  # extra assertions
+
+  def assert_success(res)
+    msg = build_message(nil, "command status was expected to be a success, but failed with code <?> and STDERR:\n\n#{res.err}", res.status.to_i)
+    assert_block(msg) { res.status.success? }
+  end
+
+  def assert_same_path(a, b)
+    msg = build_message(nil, "<?> expected but was\n<?>", a, b)
+    assert_block(msg) { File.identical? a, b }
+  end
+
+  # utils
+  class << self
+    def startup
+      @@test_dir = File.join(TestLib::TMP_DIR, self.name.
+        # to-underscore conversion from Rails
+        gsub(/::/, '/').
+        gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2').
+        gsub(/([a-z\d])([A-Z])/,'\1_\2').
+        tr("-", "_").
+        downcase
+      )
+    end
+  end
+
+  def cur_test_dir
+    @@test_dir
+  end
+
+  def make_project(template_dir = nil)
+    test_name = local_name
+    test_name = test_name[5..] if test_name.start_with?('test_')
+
+    base = File.join(cur_test_dir, test_name)
+    src_dir = base + '-src'
+
+    FileUtils.mkdir_p src_dir
+
+    if template_dir
+      template_dir = File.join(TestLib::TEMPLATES_DIR, template_dir)
+      raise "#{template_dir} is not a directory" if !File.directory?(template_dir)
+      FileUtils.copy_entry template_dir, src_dir
+    end
+
+    Project.new src_dir, base + '-bin'
+  end
+end