[4.0] Combine update SQL scripts to reduce their number

[richard67]

Pull Request for Issue # .

Summary of Changes

This pull request (PR) combines update SQL scripts to a lower number of scripts according to the following scheme, not combining scripts between different Beta versions, with content of scripts which are removed (marked with (-)) being moved forward to existing scripts marked with an (+).

The order of SQL statements hasn't been changed.

In the remaining scripts where content of removed scripts has been moved to, a comment before the moved content shows from which script it was moved.

The obsolete scripts have been added to the list of files to be deleted in script.php. There is an additional change in that file for correcting alphabetical order.

Here comes the scheme for combining the SQL scripts:

### Beta 1
--------------------
4.0.0-2016-07-03.sql (-)
4.0.0-2016-09-22.sql (-)
4.0.0-2016-09-28.sql (-)
4.0.0-2016-10-02.sql (-)
4.0.0-2016-10-03.sql (-)
4.0.0-2017-03-18.sql (-)
4.0.0-2017-04-25.sql (-)
4.0.0-2017-05-31.sql (-)
4.0.0-2017-06-03.sql (-)
4.0.0-2017-10-10.sql (-)
4.0.0-2018-02-24.sql (-)
4.0.0-2018-03-05.sql (+)
--------------------
4.0.0-2018-05-15.sql (no change)
--------------------
4.0.0-2018-06-03.sql (-)
4.0.0-2018-06-26.sql (-)
4.0.0-2018-07-02.sql (-)
4.0.0-2018-07-19.sql (+)
--------------------
4.0.0-2018-07-29.sql (no change)
--------------------
4.0.0-2018-08-01.sql (-)
4.0.0-2018-08-29.sql (+)
--------------------
4.0.0-2018-09-12.sql (-)
4.0.0-2018-10-18.sql (-)
4.0.0-2019-01-05.sql (-)
4.0.0-2019-01-16.sql (-)
4.0.0-2019-02-03.sql (-)
4.0.0-2019-03-09.sql (+)
--------------------
4.0.0-2019-03-30.sql (no change)
--------------------
4.0.0-2019-03-31.sql (-)
4.0.0-2019-04-15.sql (+)
--------------------
4.0.0-2019-04-22.sql (no change)
--------------------
4.0.0-2019-05-05.sql (-)
4.0.0-2019-05-20.sql (+)
--------------------
4.0.0-2019-06-28.sql (-)
4.0.0-2019-06-29.sql (+)
--------------------
4.0.0-2019-07-02.sql (-)
4.0.0-2019-07-13.sql (+)
--------------------
4.0.0-2019-07-14.sql (-)
4.0.0-2019-07-16.sql (-)
4.0.0-2019-08-03.sql (-)
4.0.0-2019-08-20.sql (-)
4.0.0-2019-08-21.sql (-)
4.0.0-2019-09-13.sql (+)
--------------------
4.0.0-2019-09-14.sql (-)
4.0.0-2019-09-22.sql (+)
--------------------
4.0.0-2019-09-23.sql (-)
4.0.0-2019-09-24.sql (-)
4.0.0-2019-09-25.sql (-)
4.0.0-2019-09-26.sql (-)
4.0.0-2019-09-27.sql (-)
4.0.0-2019-09-28.sql (-)
4.0.0-2019-09-29.sql (-)
4.0.0-2019-10-06.sql (+)
--------------------
4.0.0-2019-10-13.sql (-)
4.0.0-2019-10-17.sql (+)
--------------------
4.0.0-2019-10-29.sql (-)
4.0.0-2019-11-07.sql (-)
4.0.0-2019-11-19.sql (-)
4.0.0-2020-02-02.sql (+)
--------------------
4.0.0-2020-02-08.sql (-)
4.0.0-2020-02-20.sql (-)
4.0.0-2020-02-22.sql (-)
4.0.0-2020-02-29.sql (-)
4.0.0-2020-03-10.sql (+)
--------------------
4.0.0-2020-03-25.sql (no change)
--------------------
4.0.0-2020-04-11.sql (-)
4.0.0-2020-04-16.sql (-)
4.0.0-2020-05-21.sql (-)
4.0.0-2020-05-29.sql (+)
--------------------

### Beta 2 and 3 nothing

### Beta 4
--------------------
4.0.0-2020-08-01.sql (only postgresql, no change)
--------------------

### Beta 5
--------------------
4.0.0-2020-09-19.sql (-)
4.0.0-2020-09-22.sql (-)
4.0.0-2020-09-27.sql (+)
--------------------

### Beta 6
--------------------
4.0.0-2020-12-08.sql (-)
4.0.0-2020-12-19.sql (-)
4.0.0-2020-12-20.sql (+)
--------------------

### Beta 7 nothing

### Beta 8
--------------------
4.0.0-2021-02-28.sql (-)
4.0.0-2021-04-11.sql (-)
4.0.0-2021-04-20.sql (-)
4.0.0-2021-04-22.sql (+)
--------------------
4.0.0-2021-04-27.sql (no change)
--------------------
4.0.0-2021-05-01.sql (-)
4.0.0-2021-05-04.sql (-)
4.0.0-2021-05-07.sql (-)
4.0.0-2021-05-10.sql (-)
4.0.0-2021-05-21.sql (+)
--------------------

Testing Instructions

Test 1: Code review

Review the changes made by this PR regarding following aspects:

• Files are removed and contents of removed files is moved to remaining files exactly as shown in the scheme in the description of changes above.
• No files are combined or file content is moved across different Beta versions.
• The order of SQL statements is not changed.
• The -- From ... comments in the files where code is moved to are correct.
• The obsolete SQL scripts are added to the file removal in script.php, none is forgotten.

Test 2: Update from 3.10

1. Update a current 3.10-dev branch or latest 3.10 nightly to the last 4.0 nightly build, using the custom update URL or the update package provided here: https://developer.joomla.org/nightly-builds.html .
2. Export the complete database into an SQL file e.g. with phpMyAdmin (or phpPgAdmin if using PostgreSQL).
3. Save the update log administrator/logs/joomla_update.php somewhere outside the Joomla installation.
4. Repeat steps 1 with the same starting conditions, but this time update to the custom update URL or update package created by drone for this PR.
   You can find the links to the update URL or update package when following the link for the downloads atb the bottom of this PR:
   
5. Export the complete database into another SQL file than the one created in step 2.
6. Save the update log administrator/logs/joomla_update.php somewhere outside the Joomla installation at a different place or with a different name compared to step 3.
7. Compare the SQL files from steps 2 and 5.
   Result: There are no differences beside the expected ones like e.g. different super user ID.
8. Compare the saved update logs from steps 3 and 6.
   Result: There are no differences beside the time stamps and the names of the SQL scripts reported with the SQL statements. The order of SQL statements is the same.

Test 3: Update from a previous 4.0 Beta version

Same procedure as test 2, except that here any 4.0 Beta version is used as starting point.

Attention: When updating from a 4.0 Beta1, 2 or 3, it requires an additional SQL step to be executed e.g. in phpMyAdmin, see https://docs.joomla.org/J4.x:Upgrade_to_4.0_Beta_4 .

Actual result BEFORE applying this Pull Request

84 scripts for mysql and 85 for postgresql.

Expected result AFTER applying this Pull Request

25 scripts for mysql and 26 for postgresql.

=> 59 scripts less than before for each database type.

Update from 3.10 or from previous 4.0 Beta versions works same as before (including the necessary additional SQL step for updating from 4.0 Beta 2, 2 or 3 described here https://docs.joomla.org/J4.x:Upgrade_to_4.0_Beta_4 ).

The result in database of an update with this PR is the same as without this PR.

The obsolete update scripts are deleted after an update.

Documentation Changes Required

None.

[brianteeman]

Great stuff - did a quick look at some obvious things like the date of files and they all look good.

[richard67]

PR is ready except of testing instructions and real test by myself. I will continue after dinner and mark it as ready for review later today.

[richard67]

Ready for review and tests.

[richard67]

Wait, I will maybe update this to recent commits which have added new small update SQL scripts.

[richard67]

Updated. Ready for test again.

[alikon]

a review

[wilsonge]

Ok let’s get this in so it can be tested. Thankyou!

[richard67]

Thanks.