[HTTPS] Finalize update server to https process

[andrepereiradasilva]

Summary of Changes

This PR is a revival of #8196 that was closed because by then the Joomla update server did not fully supported HTTPS.

Now that the update server is working in HTTPS i think we can now safely update all update.joomla.org http:// references to https://.

Testing Instructions

Experience joomla developers can check the code review.
For more extensive tests:

Update

1. Apply this patch to latest staging or 3.5.0
2. Run the queries from https://raw.githubusercontent.com/andrepereiradasilva/joomla-cms/update-https/administrator/components/com_admin/sql/updates/mysql/3.5.2-2016-04-01.sql (replace #__ for your table prefix)
3. Check if the update server URI are in https in #__update_sites table.
4. Test if fetching for a joomla update does not give any error.

New installation

1. Download patched Joomla 3.5.0 (https://github.com/andrepereiradasilva/joomla-cms/archive/update-https.zip)
2. Install joomla
3. Check if the update server URI are in https in #__update_sites table.
4. Test if fetching for a joomla update does not give any error.

[dgrammatiko]

@andrepereiradasilva instead of asking ppl to patch themselves the 3.5 you can ask them to use your repo: https://github.com/andrepereiradasilva/joomla-cms/archive/update-https.zip

[andrepereiradasilva]

updated thanks @DGT41

[waader]

I have tested this item ✅ successfully on 40d4ef4

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/9579.}

[wojsmol]

@andrepereiradasilva Please rebase this PR

[andrepereiradasilva]

yes, will do when i have time.

[wojsmol]

@andrepereiradasilva see andrepereiradasilva#25

[joomla-cms-bot]

This PR has received new commits.

CC: @waader

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/9579.}

[andrepereiradasilva]

just fixed conflicts no other change

[brianteeman]

I have tested this item ✅ successfully on 8f9dcd6

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/9579.}

[andrepereiradasilva]

@waader can you retest after the latest (solve conflicts) changes?

[conconnl]

I have tested this item ✅ successfully on 8f9dcd6

Tested Find Updates after clearing cache with UpdateSite too Joomla! updates on HTTPS.

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/9579.}

[brianteeman]

RTC

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/9579.}

[brianteeman]

@rdeutz your choice on the milestone ;)

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/9579.}

[richard67]

@andrepereiradasilva Just for info: The changes in your update scripts "3.5.2-2016-04-01.sql" will be applied when people update with the Joomla! Update component (including new upload and install feature), because in this case the update scripts are just executed.

But when people update with the "copy the files or unzip the container and then run the db fixer" method, the statements will not be applied, because silly db fix only handles DDL statements (data definition language, "CREATE TABLE" or "ALTER TABLE"), but not DML (data manipulation language, "INSERT", "UPDATE", "DELETE").

Just in case you were not aware of this and later find this during testing somehow.

[andrepereiradasilva]

huumm i tought the db fixer would run all sql statements.
so there are a lot of sql in the sql updates dir that are not applied!

[richard67]

Yes, all insert update delete is not applied by the db fixer. And in the way things work now, it would not be possible to implement that in a safe way, using check queries for verifying if a statement has been applied, because some of the statements are executed when doing a discovery install, e.g. all those where new core components were introduced and so the schema updates contain inserts into the extensions table.

One way do solve that (with a certain risk) would be to do in the db fixer what the updater does, too: Run all schema update scripts with a version in their filename greater than the schema version saved in db. But this works only 1 time, because at the end, the schema manager sets the db schema to the version of the last file in the array, regardless if the file has been executed with success or if there were some errors.

Maybe there were some attempts to do such a correction in past, but people noticed it did not work?

@wilsonge could maybe know that.

I currently work a bit on the next step for utf8mb4: make it work for extensions, too, and use also versioned files like for schema updates, to when some new stuff to be added it does not require to run the complete conversion again.

When this will be done in a few days or some week or 2, my idea was to work on the db fixer to make it work better. Am not sure if I will still have enought time then, but hope so.

And maybe make it also handle extensions, but this could collide with what discovery install does, so am not sure about that thing.

[wojsmol]

@richard67 The situation described above is associated with the query syntax -See #8788. Already working on PR.

[richard67]

@wojsmol Did you comment on the wrong discussion?

What I discuss here with @andrepereiradasilva has absolutely nothing to do with the issue you refer to.

The issue you refer to has nothing to do with the fact that when you use the "copy the files or unzip the zip and then use the db fixer" method to update Joomla!, the schema manager (db fixer) does not apply INSERT or UPDATE or DELETE statements.

[wojsmol]

@richard67 No, see #8788 (comment) UPDATE vs UPDATE COLUMN.

[richard67]

@wojsmol Where in this sql handled there can you see any "UPDATE"? You are wrong! What @wilsonge meant in his comment is that the quoting of the column name is required for the schema manager to correctly detect the column name in an alter table statement, and yes, that's right. But it has nothing to do with the fact that the schema manager does not apply insert or update or delete statements when doing the database fix! @wilsonge Please confirm so he believes me.

[richard67]

Ahh I just see: The code link of @wilsonge in his comment does not fit anymore because the file meanwhile changed!!! He linked to line 25 once but meanwhile new code has been inserted in this file.

But again, it has nothing to do with this issue here, it only corrected a syntax error.

What I discuss here with @andrepereiradasilva is the fact that the schema manager not runs the schema update script completely when doing a db fix, it only checks and if necessary repairs statements "alter table" and "create table". I added a special thing for resetting the utf8mb4 conversion status, and this is now what is shown in line 25, but when George commented that, my code was not there yet.

So again: Beside this special update statement for this special column of this special table, the schema manager does not apply DML statements. Full stop.

@wilsonge again if you reead please confirm so I do not get nothered with this silly discussion!

[wojsmol]

@richard67 you are right, this is update for column content not definition as in #8788. Sorry for the confusion.

[richard67]

@andrepereiradasilva The question remains what we can do to make your change from this PR here (and others with updates of database records e.g. for https) also be applied when people use the "copy files and then use db fixer" method.

That was the reason why I commented that here, because your changes are important, and those people will miss them.

It is a big mess what we have now in Joomla! with schema updating.

If you have any ideas or opinions let me know.

I wish @wilsonge or some other maintainer being familiar with the schema updating and db fixer would find the time for a discussion, because I have a few ideas which ways we could go to solve that, but I have no idea if some of these ideas once in past already have been tried by someone else and have been given up for some reason.

Unfortunately I do not plan to join JAB so we cannot do this discussion there, at least not with me.

But there has to be done something.

Or Joomla! stops to support and advertise this trouble causing update method, now as we have the "Upload & Install" option in the update component.

[infograf768]

Folks, I can't update or install languages anymore.
Is it related to this PR?

[infograf768]

Sorry, forget it: I had left the tmp folder readonly after some other tests. 😃

[brianteeman]

lol