Add initial E2E test for latest-release.php

.github/PULL_REQUEST_TEMPLATE

@@ -12,8 +12,8 @@ Draft for a patch:
 
 TODO:
 - [ ] Added patch for [describe issue]
-- [ ] Add/update tests -- unit and/or integrated (if needed)
-  - [ ] Ensure only one function is tested per test file.
+- [ ] Add/update tests -- unit/integrated/E2E (if needed)
+  - [ ] Ensure only one function/functionality is tested per test file.
 - [ ] Add to, or update, `Scan run detail` report as applicable
 - [ ] Check status of automated tests
 - [ ] Ensure `PHPDoc` comments are up to date for functions added or altered
@@ -29,8 +29,8 @@ TODO:
 - [ ] Update `--help` message
 - [ ] Implement [new feature / logic]
 - [ ] Add to, or update, `Scan run detail` report as applicable
-- [ ] Add/update tests -- unit and/or integrated
-  - [ ] Ensure only one function is tested per test file.
+- [ ] Add/update tests -- unit/integrated/E2E
+  - [ ] Ensure only one function/functionality is tested per test file.
 - [ ] Ensure `PHPDoc` comments are up to date for functions added or altered
 - [ ] Update repository documentation (README.md, RELEASING.md, TESTING.md, TOOLS-UPDATE.md)
 - [ ] Assign appropriate [priority](https://github.com/Automattic/vip-go-ci/blob/trunk/CONTRIBUTING.md#priorities) and [type of change labels](https://github.com/Automattic/vip-go-ci/blob/trunk/CONTRIBUTING.md#type-of-change-labels).
@@ -47,8 +47,10 @@ TODO:
 - [ ] Add same version number in defines.php
 - [ ] Assign a milestone corresponding to the version number to this PR and PRs that will form the release
 - [ ] Assign label ([Changelog and version](https://github.com/Automattic/vip-go-ci/blob/trunk/CONTRIBUTING.md#type-of-change-labels))
-- [ ] Run unit-test suite
+- [ ] Unit-test suite run successful 
+- [ ] Integration-test suite successful (without secrets)
 - [ ] Run integration-test suite with secrets
+- [ ] E2E-test suite run successful
 - [ ] Manual testing
   - [ ] Pull request with PHP linting issues
   - [ ] Pull request with PHPCS issues

.github/workflows/ci.yml

@@ -118,6 +118,41 @@ jobs:
         run: |
           rm -f tests/config-secrets.ini
 
+  e2e-testing:
+    name: Run E2E tests (PHP ${{ matrix.php }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        php:
+          - '8.0'
+          - '8.1'
+          - '8.2'
+    steps:
+      - name: Check out the source code
+        uses: actions/checkout@v3
+
+      - name: Ask git to fetch latest branch and other branches
+        run: git fetch origin latest && git pull
+
+      - name: Set up PHP
+        uses: shivammathur/[email protected]
+        with:
+          coverage: none
+          php-version: "${{ matrix.php }}"
+          tools: phpunit:9
+        env:
+          fail-fast: 'true'
+
+      - name: Configure PHPUnit
+        run: sed "s:PROJECT_DIR:$(pwd):g" phpunit.xml.dist > phpunit.xml
+
+      - name: Run E2E tests
+        run: phpunit --testsuite=e2e-tests
+        env:
+          VIPGOCI_TESTING_DEBUG_MODE: 'true'
+
+
   php-code-compatibility:
     name: PHP code compatibility (8.2)
     runs-on: ubuntu-latest

TESTING.md

@@ -2,7 +2,7 @@
 
 ## Introduction 
 
-`vip-go-ci` relies on both manual and automated testing. Much of the functionality `vip-go-ci` provides is automatically tested using it's extensive unit and integration test suites. _Most_ of the tests in the test suites are run automatically when code is committed and pushed to the repository, though _some_ integration tests need to be run manually (due to secrets, see below). The manual testing that should be performed is functional, testing the final behaviour of the software. 
+`vip-go-ci` relies on both manual and automated testing. Much of the functionality `vip-go-ci` provides is automatically tested using it's extensive unit, integration and E2E (End-to-End) test suites. _Most_ of the tests in the test suites are run automatically when code is committed and pushed to the repository, though _some_ integration tests need to be run manually (due to secrets, see below). The manual testing that should be performed is functional, testing the final behaviour of the software. 
 
 ## Automated testing
 
@@ -80,6 +80,12 @@ Integration tests will execute the scanning utilities — PHPCS, SVG scanner and
 
 By using this command, you will run the tests of the test-suite which can be run (depending on tokens and other detail), and get feedback on any errors or warnings. Note that when run, requests will be made to the GitHub API using anonymous calls (unless configured to use an access-token as shown above). It can happen that the GitHub API returns with an error indicating that the maximum limit of API requests has been reached; the solution is to wait and re-run or switch to authenticted calls. 
 
+### E2E test suite
+
+The E2E (End-to-End) tests can be run using the following command:
+
+> VIPGOCI_TESTING_DEBUG_MODE=true phpunit --testsuite=e2e-tests
+
 ### Test isolation
 
 Note that the test suite uses the `@runTestsInSeparateProcesses` and `@preserveGlobalState` PHPUnit flags to avoid any influence of one test on another. Further, tests should include all required files in `setUp()` function to avoid the same function being defined multiple times across multiple tests during the same run. Combining the usage of `@runTestsInSeparateProcesses` and the inclusion of required files in `setUp()` means each test is independent of other tests, which enables functions to be defined for each test easily and avoids leakage between tests.

phpunit.xml.dist

@@ -15,5 +15,8 @@
         <testsuite name="integration-tests">
             <directory>PROJECT_DIR/tests/integration</directory>
         </testsuite>
+        <testsuite name="e2e-tests">
+            <directory>PROJECT_DIR/tests/e2e</directory>
+        </testsuite>
     </testsuites>
 </phpunit>

tests/e2e/LatestReleaseTest.php

@@ -0,0 +1,115 @@
+<?php
+/**
+ * Verify that latest-release.php behaves as it should.
+ *
+ * @package Automattic/vip-go-ci
+ */
+
+declare(strict_types=1);
+
+namespace Vipgoci\Tests\E2E;
+
+use PHPUnit\Framework\TestCase;
+
+/**
+ * Class that implements the testing.
+ *
+ * @runTestsInSeparateProcesses
+ * @preserveGlobalState disabled
+ */
+final class LatestReleaseTest extends TestCase {
+	/**
+	 * Temporary file for contents of defines.php.
+	 *
+	 * @var $temp_file_name
+	 */
+	private mixed $temp_file_name = '';
+
+	/**
+	 * Setup function. Require files, etc.
+	 *
+	 * @return void
+	 */
+	protected function setUp() :void {
+		$this->temp_file_name = tempnam(
+			sys_get_temp_dir(),
+			'vipgoci-defines-php-file'
+		);
+	}
+
+	/**
+	 * Clean up.
+	 *
+	 * @return void
+	 */
+	protected function tearDown() :void {
+		if ( false !== $this->temp_file_name ) {
+			unlink( $this->temp_file_name );
+		}
+	}
+
+	/**
+	 * Verify that return value from the script matches
+	 * real version number. Also verify that the format
+	 * is correct.
+	 *
+	 * @return void
+	 */
+	public function testResults(): void {
+		// phpcs:disable WordPress.PHP.DiscouragedPHPFunctions.system_calls_exec
+		if ( false === $this->temp_file_name ) {
+			$this->markTestSkipped(
+				'Unable to create temporary file'
+			);
+
+			return;
+		}
+
+		/*
+		 * Get 'defines.php' from latest branch,
+		 * put contents of the file into temporary file
+		 * and then retrieve the version number
+		 * by including the file.
+		 */
+		exec( 'git -C . show latest:defines.php > ' . $this->temp_file_name );
+
+		require_once $this->temp_file_name;
+
+		$correct_version_number = VIPGOCI_VERSION;
+
+		/*
+		 * Run latest-release.php to get latest version number.
+		 */
+		$returned_version_number = exec( 'php latest-release.php' );
+
+		/*
+		 * Verify format of version number is correct.
+		 */
+		$version_number_preg = '/^(\d+\.)?(\d+\.)?(\*|\d+)$/';
+
+		$this->assertSame(
+			1,
+			preg_match(
+				$version_number_preg,
+				$correct_version_number
+			)
+		);
+
+		$this->assertSame(
+			1,
+			preg_match(
+				$version_number_preg,
+				$returned_version_number
+			)
+		);
+
+		/*
+		 * Verify both version numbers match.
+		 */
+		$this->assertSame(
+			$correct_version_number,
+			$returned_version_number
+		);
+		// phpcs:enable WordPress.PHP.DiscouragedPHPFunctions.system_calls_exec
+	}
+}