[4.0] Improve the generator tool for lists of files and folders to be removed or renamed on update #22

richard67 · 2021-12-18T13:44:33Z

Pull Request for Issue # .

Replaces #21 .

Summary of Changes

This pull request (PR) is just to document the changed script "build/deleted_file_check.php" which I am currently using to generate the lists of deleted files and folders and renamed files for updating script.php.

The changes in the script are:

Support zip files as parameters - Zip files will not be unpacked for the check. Instead of this, the files and folders lists and the version number from the manifest XML are directly read from the zip file.
Let the "to" parameter for the ending version be optional like it was intended to be but up to now not had been implemented.
If the "to" parameter is not given, a full package zip from a previous run of the build script will be used if such a file exists.
Not create the result files but append the results of the current comparison to the particular files if they aren't already included in that file from previous comparisons.
This allows to execute a sequence of comparisons, see description below on how I use the changed script with a shell script to completely rebuild the lists.
When a file or a folder once had been removed so it is included in results of previous comparisons and now with the current "to" version is added back, it is removed from the previous results.
For an example see [4.0] Remove _com_joomlaupdate.scss from deleted files in script.php joomla/joomla-cms#36161 .
Prepend each comparison with a comment which contains the from and to version.
The version numbers are read from the manifest XML files "administrator/manifests/files/joomla.xml" of the "from" and "to" directories or zip packages.
Update docs and comments to reflect all previous changes as well as up to now undocumented changes in the 4.0-dev branch.

Testing Instructions

Currently I use following Bash script to generate the lists for the 4.0-dev branch:

#!/usr/bin/env bash

BASEDIR=$(dirname "$(realpath $0)")
cd $BASEDIR

mkdir -p ./tmp/deleted_files_check/

# Build current version if there is no result present from a previous build
if [ ! -f ./tmp/packages/*Full_Package.zip ]
then
	php ./build.php --remote=HEAD --exclude-gzip --exclude-bzip2
fi

CURRENTZIP=`ls -1 ./tmp/packages/*Full_Package.zip`

# Clone and build previous major version if there is no result present from a previous build
if [ ! -f ./tmp/deleted_files_check/previous-package/*Full_Package.zip ]
then
	mkdir -p ./tmp/deleted_files_check/previous-package
	mkdir -p ./tmp/deleted_files_check/previous-build
	cd ./tmp/deleted_files_check/previous-build
	git clone -b 3.10-dev https://github.com/joomla/joomla-cms.git .
	php ./build/build.php --remote=HEAD --exclude-gzip --exclude-bzip2
	cd $BASEDIR
	if [ -f ./tmp/deleted_files_check/previous-build/build/tmp/packages/*Full_Package.zip ]
	then
		cp ./tmp/deleted_files_check/previous-build/build/tmp/packages/*Full_Package.zip ./tmp/deleted_files_check/previous-package/
	fi
fi

rm -rf ./tmp/deleted_files_check/previous-build

PREVIOUSZIP=`ls -1 ./tmp/deleted_files_check/previous-package/*Full_Package.zip`

# Start with comparing the previous major release build with the current version's build.
rm -f ./deleted_files.txt
rm -f ./deleted_folders.txt
rm -f ./renamed_files.txt
php ./deleted_file_check.php --from=$PREVIOUSZIP --to=$CURRENTZIP

# Compare past beta or RC or stable releases of the current version from old to new
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta1-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta2-Beta-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta2-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta3-Beta-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta3-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta4-Beta-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta4-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta5-Beta-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta5-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta6-Beta-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta6-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta7-Beta-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta7-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc1-Release_Candidate-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc1-Release_Candidate-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc2-Release_Candidate-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc2-Release_Candidate-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc3-Release_Candidate-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc3-Release_Candidate-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc4-Release_Candidate-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc4-Release_Candidate-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc5-Release_Candidate-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc5-Release_Candidate-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc6-Release_Candidate-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc6-Release_Candidate-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-Stable-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-Stable-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.1-Stable-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.1-Stable-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.2-Stable-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.2-Stable-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.3-Stable-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.3-Stable-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.4-Stable-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.4-Stable-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.5-Stable-Full_Package.zip

# Finish with comarison between the latest release and the current build
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.5-Stable-Full_Package.zip \
  --to=$CURRENTZIP

When keeping the intermediate results from comparing older 4.0 versions since Beta 1, it only needs to do the first comparison from current 3.10-dev to current 4.0-dev and then the last 2 comparisons. The 2nd last because something might have been forgotten in script.php when the last release was made, and the last one for the last release and the current branch.

At the end I can take the complete content of each file, intend it by 3 tabs and copy it into script PHP and can see the changes in Git.

Then it needs only 3 manual steps:

Add back the intermediate 4.0.0 update SQL scripts which I had removed between 2 beta versions when combining the update SQL script, so the comparisons don't know about them.
Move deletion of folder "/libraries/vendor/paragonie/random_compat" from section "// From 3.10 to 4.0" to section "// 4.0 from Beta 7 to RC 1". The reason is that we have different child folders of that being deleted when updating from 3.10 or when updating from 4.0 Beta 7, so we get the folder 2 times, and the duplicate check added in this PR here will add it do the 1st and not to the 2nd section, but in this case it should be in the 2nd.
Remove section comments for empty sections

This can be automated later with further improvements, but the question now is if what I am doing here is the right way to go.

For 4.1-dev I run the following shell script after having run the previously mentioned one for 4.0-dev:

#!/usr/bin/env bash

# Copy results from 4.0-dev
cp -f /home/richard/lamp/public_html/joomla-cms-4.0-dev/build/deleted_files.txt ./
cp -f /home/richard/lamp/public_html/joomla-cms-4.0-dev/build/deleted_folders.txt ./
cp -f /home/richard/lamp/public_html/joomla-cms-4.0-dev/build/renamed_files.txt ./

# From 4.0-dev to 4.1-dev - ignore results from previous comparisons
php ./deleted_file_check.php \
    --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.5-rc5-dev-Development-Full_Package \
    --to=/home/richard/lamp/public_html/4.1-compare-releases/Joomla_4.1.0-dev-Development-Full_Package

Actual result BEFORE applying this Pull Request

I have to download or build full package zip files and then unpack them for all comparisons.

I have to identify and remove duplicates myself from the right section of the deleted files and folder lists in script.php.

I have to remove files or folders from the lists which once had been removed and later been added back between releases.

The help and the documentation in the generator tool are wrong.

Expected result AFTER applying this Pull Request

I don't have to unpack full package zip files.

If the "to" parameter is not given, the full package from a previous run of the build script is used, if present.

Files and folders which have been identified to be removed when updating from the previous major (or minor) release on another branch, e.g. in case of J4 from 3.10, are not to be removed again when updating from a previous version of the current release to be checked.

When some file or folder is removed between 2 versions and later added back with another, later version, the tool is able to handle that yet and remove them from the lists.

The documentation and help in the generator tool describe what is implemented.

Documentation Changes Required

There is no documentation yet on how to use deleted_file_check.php.

Remove files and folders from the lists if they once were removed and later were added back.

SQL fixes for PR #14

* Removed hardcoded username from Actionlogs.cy.js On line #22 "ci-admin" was hardcoded, causing the test to fail, if another user is configured in cypress.config.mjs * Changed Syntax to use Template Literals Using Template Literals to pass JS linting * Update tests/System/integration/administrator/components/com_actionlogs/Actionlogs.cy.js Co-authored-by: Richard Fath <[email protected]> --------- Co-authored-by: Richard Fath <[email protected]>

richard67 added 16 commits December 18, 2021 12:46

Let deleted files check append and not replace results

95c1312

Append only if some results

3bc06f0

Report both versions at the end

e9444b3

Fix "to" parameter and fix wrong help and comments

7d3dc49

One more code comment fix

407f518

Prepend comment with versions from manifest XML files

94a13e3

Handle files and folders which have been added back

91fc6ea

Remove files and folders from the lists if they once were removed and later were added back.

Support reading from zip files

39afbe6

Fix help and change order of processing

cd357b8

Change back "to" parameter to optional

1a71b78

Fix comment

e25ca84

Update documentation

fd55adf

Fix my typo in documentation

02aa72a

Get version from manifest XML

d8af83d

Add some function documentation

e16d20b

Extend the help text

f85cad4

richard67 mentioned this pull request Dec 18, 2021

[4.0] Improve the generator tool for lists of files and folders to be removed or renamed on update #21

Closed

richard67 closed this Jun 12, 2022

richard67 deleted the 4.0-dev-deleted-files-and-folders-improvements-4 branch June 12, 2022 17:39

richard67 pushed a commit that referenced this pull request Jul 14, 2023

Merge pull request #22 from richard67/base-fix-sql-scripts-by-richard67

fa1e081

SQL fixes for PR #14

richard67 pushed a commit that referenced this pull request Feb 25, 2024

Update root key postgresql (#22)

4bf7b2b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[4.0] Improve the generator tool for lists of files and folders to be removed or renamed on update #22

[4.0] Improve the generator tool for lists of files and folders to be removed or renamed on update #22

richard67 commented Dec 18, 2021

[4.0] Improve the generator tool for lists of files and folders to be removed or renamed on update #22

[4.0] Improve the generator tool for lists of files and folders to be removed or renamed on update #22

Conversation

richard67 commented Dec 18, 2021

Summary of Changes

Testing Instructions

Actual result BEFORE applying this Pull Request

Expected result AFTER applying this Pull Request

Documentation Changes Required