[4.0] Improve the generator tool for lists of files and folders to be removed or renamed on update

[richard67]

Pull Request for Issue # .

Replaces #20 .

Summary of Changes

This pull request (PR) is just to document the changed script "build/deleted_file_check.php" which I am currently using to generate the lists of deleted files and folders and renamed files for updating script.php.

The changes in the script are:

• Not create the result files but append the results of the current comparison to the particular files if they aren't already included in that file from previous comparisons.
  This allows to execute a sequence of comparisons, see description below on how I use the changed script with a shell script to completely rebuild the lists.
• When a file or a folder once had been removed so it is included in results of previous comparisons and now with the current "to" version is added back, it is removed from the previous results.
  For an example see [4.0] Remove _com_joomlaupdate.scss from deleted files in script.php joomla/joomla-cms#36161 .
• Prepend each comparison with a comment which contains the from and to version.
  The version numbers are read from the manifest XML files "administrator/manifests/files/joomla.xml" of the "from" and "to" directories.
• Fix help doc and comments to reflect past changes in the 4.0-dev branch.

Testing Instructions

Currently I use following Bash script to generate the lists for the 4.0-dev branch:

#!/usr/bin/env bash

# Function for download and unpack of a full package if not done before
function fetchZipDownload {
	if [ ! -d ./tmp/deleted_files_check/$1 ]
	then
		if [ ! -f ./tmp/deleted_files_check/${1}.zip ]
		then
			curl -L -o ./tmp/deleted_files_check/${1}.zip $2
		fi
		if [ -f ./tmp/deleted_files_check/${1}.zip ]
		then
			mkdir -p ./tmp/deleted_files_check/$1
			unzip ./tmp/deleted_files_check/${1}.zip -d ./tmp/deleted_files_check/$1
		fi
	fi
}

BASEDIR=$(dirname "$(realpath $0)")
cd $BASEDIR

# Build current version if there is no result present from a previous build
if [ ! -d ./tmp/deleted_files_check/current ]
then
	if [ ! -f ./tmp/packages/*Full_Package.zip ]
	then
		php ./build.php --remote=HEAD --exclude-gzip --exclude-bzip2
	fi
	if [ -f ./tmp/packages/*Full_Package.zip ]
	then
		mkdir -p ./tmp/deleted_files_check/current
		unzip `ls -1 ./tmp/packages/*Full_Package.zip` -d ./tmp/deleted_files_check/current
	fi
fi

# Clone and build previous major version if there is no result present from a previous build
if [ ! -d ./tmp/deleted_files_check/previous ]
then
	if [ ! -f ./tmp/deleted_files_check/previous-package/*Full_Package.zip ]
	then
		mkdir -p ./tmp/deleted_files_check/previous-package
		mkdir -p ./tmp/deleted_files_check/previous-build
		cd ./tmp/deleted_files_check/previous-build
		git clone -b 3.10-dev https://github.com/joomla/joomla-cms.git .
		php ./build/build.php --remote=HEAD --exclude-gzip --exclude-bzip2
		cd $BASEDIR
		if [ -f ./tmp/deleted_files_check/previous-build/build/tmp/packages/*Full_Package.zip ]
		then
			cp ./tmp/deleted_files_check/previous-build/build/tmp/packages/*Full_Package.zip ./tmp/deleted_files_check/previous-package/
		fi
	fi
	if [ -f ./tmp/deleted_files_check/previous-package/*Full_Package.zip ]
	then
		mkdir -p ./tmp/deleted_files_check/previous
		unzip `ls -1 ./tmp/deleted_files_check/previous-package/*Full_Package.zip` -d ./tmp/deleted_files_check/previous
	fi
fi

rm -rf ./tmp/deleted_files_check/previous-build

# Fetch and unpack zip downloads of the current major version's past releases if not present from a previous run
fetchZipDownload 4.0.0-beta1 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta/Joomla_4.0.0-beta1-Beta-Full_Package.zip
fetchZipDownload 4.0.0-beta2 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta2/Joomla_4.0.0-beta2-Beta-Full_Package.zip
fetchZipDownload 4.0.0-beta3 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta3/Joomla_4.0.0-beta3-Beta-Full_Package.zip
fetchZipDownload 4.0.0-beta4 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta4/Joomla_4.0.0-beta4-Beta-Full_Package.zip
fetchZipDownload 4.0.0-beta5 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta5/Joomla_4.0.0-beta5-Beta-Full_Package.zip
fetchZipDownload 4.0.0-beta6 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta6/Joomla_4.0.0-beta6-Beta-Full_Package.zip
fetchZipDownload 4.0.0-beta7 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta7/Joomla_4.0.0-beta7-Beta-Full_Package.zip
fetchZipDownload 4.0.0-rc1   https://github.com/joomla/joomla-cms/releases/download/4.0.0-rc1/Joomla_4.0.0-rc1-Release_Candidate-Full_Package.zip
fetchZipDownload 4.0.0-rc2   https://github.com/joomla/joomla-cms/releases/download/4.0.0-rc2/Joomla_4.0.0-rc2-Release_Candidate-Full_Package.zip
fetchZipDownload 4.0.0-rc3   https://github.com/joomla/joomla-cms/releases/download/4.0.0-rc3/Joomla_4.0.0-rc3-Release_Candidate-Full_Package.zip
fetchZipDownload 4.0.0-rc4   https://github.com/joomla/joomla-cms/releases/download/4.0.0-rc4/Joomla_4.0.0-rc4-Release_Candidate-Full_Package.zip
fetchZipDownload 4.0.0-rc5   https://github.com/joomla/joomla-cms/releases/download/4.0.0-rc5/Joomla_4.0.0-rc5-Release_Candidate-Full_Package.zip
fetchZipDownload 4.0.0-rc6   https://github.com/joomla/joomla-cms/releases/download/4.0.0-rc6/Joomla_4.0.0-rc6-Release_Candidate-Full_Package.zip
fetchZipDownload 4.0.0       https://github.com/joomla/joomla-cms/releases/download/4.0.0/Joomla_4.0.0-Stable-Full_Package.zip
fetchZipDownload 4.0.1       https://github.com/joomla/joomla-cms/releases/download/4.0.1/Joomla_4.0.1-Stable-Full_Package.zip
fetchZipDownload 4.0.2       https://github.com/joomla/joomla-cms/releases/download/4.0.2/Joomla_4.0.2-Stable-Full_Package.zip
fetchZipDownload 4.0.3       https://github.com/joomla/joomla-cms/releases/download/4.0.3/Joomla_4.0.3-Stable-Full_Package.zip
fetchZipDownload 4.0.4       https://github.com/joomla/joomla-cms/releases/download/4.0.4/Joomla_4.0.4-Stable-Full_Package.zip

# Start with comparing the previous major release build with the current version's build.
rm -f ./deleted_files.txt
rm -f ./deleted_folders.txt
rm -f ./renamed_files.txt
php ./deleted_file_check.php --from=./tmp/deleted_files_check/previous --to=./tmp/deleted_files_check/current

# Compare past beta or RC or stable releases of the current version from old to new
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta1 --to=./tmp/deleted_files_check/4.0.0-beta2
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta2 --to=./tmp/deleted_files_check/4.0.0-beta3
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta3 --to=./tmp/deleted_files_check/4.0.0-beta4
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta4 --to=./tmp/deleted_files_check/4.0.0-beta5
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta5 --to=./tmp/deleted_files_check/4.0.0-beta6
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta6 --to=./tmp/deleted_files_check/4.0.0-beta7
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta7 --to=./tmp/deleted_files_check/4.0.0-rc1
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-rc1   --to=./tmp/deleted_files_check/4.0.0-rc2
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-rc2   --to=./tmp/deleted_files_check/4.0.0-rc3
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-rc3   --to=./tmp/deleted_files_check/4.0.0-rc4
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-rc4   --to=./tmp/deleted_files_check/4.0.0-rc5
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-rc5   --to=./tmp/deleted_files_check/4.0.0-rc6
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-rc6   --to=./tmp/deleted_files_check/4.0.0
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0       --to=./tmp/deleted_files_check/4.0.1
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.1       --to=./tmp/deleted_files_check/4.0.2
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.2       --to=./tmp/deleted_files_check/4.0.3
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.3       --to=./tmp/deleted_files_check/4.0.4

# Finish with comarison between the latest release and the current build
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.4       --to=./tmp/deleted_files_check/current

When keeping the intermediate results from comparing older 4.0 versions since Beta 1, it only needs to do the first comparison from current 3.10-dev to current 4.0-dev and then the last 2 comparisons. The 2nd last because something might have been forgotten in script.php when the last release was made, and the last one for the last release and the current branch.

At the end I can take the complete content of each file, intend it by 3 tabs and copy it into script PHP and can see the changes in Git.

Then it needs only 3 manual steps:

• Add back the intermediate 4.0.0 update SQL scripts which I had removed between 2 beta versions when combining the update SQL script, so the comparisons don't know about them.
• Move deletion of folder "/libraries/vendor/paragonie/random_compat" from section "// From 3.10 to 4.0" to section "// 4.0 from Beta 7 to RC 1". The reason is that we have different child folders of that being deleted when updating from 3.10 or when updating from 4.0 Beta 7, so we get the folder 2 times, and the duplicate check added in this PR here will add it do the 1st and not to the 2nd section, but in this case it should be in the 2nd.
• Remove section comments for empty sections

This can be automated later with further improvements, but the question now is if what I am doing here is the right way to go.

For 4.1-dev I run the following shell script after having run the previously mentioned one for 4.0-dev:

#!/usr/bin/env bash

# Copy results from 4.0-dev
cp -f /home/richard/lamp/public_html/joomla-cms-4.0-dev/build/deleted_files.txt ./
cp -f /home/richard/lamp/public_html/joomla-cms-4.0-dev/build/deleted_folders.txt ./
cp -f /home/richard/lamp/public_html/joomla-cms-4.0-dev/build/renamed_files.txt ./

# From 4.0-dev to 4.1-dev - ignore results from previous comparisons
php ./deleted_file_check.php \
    --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.5-rc5-dev-Development-Full_Package \
    --to=/home/richard/lamp/public_html/4.1-compare-releases/Joomla_4.1.0-dev-Development-Full_Package

Actual result BEFORE applying this Pull Request

I have to identify and remove duplicates myself from the right section of the deleted files and folder lists in script.php.

I have to remove files or folders from the lists which once had been removed and later been added back between releases.

Expected result AFTER applying this Pull Request

Files and folders which have been identified to be removed when updating from the previous major (or minor) release on another branch, e.g. in case of J4 from 3.10, are not to be removed again when updating from a previous version of the current release to be checked.

When some file or folder is removed between 2 versions and later added back with another, later version, the tool is able to handle that yet and remove them from the lists.

Documentation Changes Required

There is no documentation yet on how to use deleted_file_check.php.

[richard67]

Closing in favour of #22 .