Encoding problem with cli export/import

[X-dark]

Hi,

After exporting my user database to sqlite from MySQL with the cli, I am trying to import it back into PostgreSQL, still with the cli.

It fails with the following error in PostgreSQL log:

Nov 08 14:38:18 htpc postgres[815]: 2019-11-08 14:38:18.987 CET [314058] ERROR:  invalid byte sequence for encoding "UTF8": 0xe9 0x20 0x69

Is it related to utf8mb4 in MySQL? Is there any way to clean my data to allow import or just skip the invalid value?

[Alkarex]

Hum, I did not observe such a problem during my tests. Would you be able to find the more precise data, which makes it fail? We could add some Unicode sanitizing before insertion

[Alkarex]

Would you be able to produce a small version of your SQLite file, exhibiting the issue?

[X-dark]

Invalid entries where limited to a feed that is no longer updated: http://www.hardware.fr/backend/news.xml. I have been able to import my database, removing all entries from that feed.

Here is the stripped down db (just one article of this feed): https://owncloud.cedricgirard.com/s/dtmHp66jqW5bfSS/download

[Alkarex]

It is this comment, apparently in ISO-8859-x in the middle of some UTF-8, which seems to cause the problem, only with PostgreSQL:

<!-- Page actualit� individuelle -->

[Alkarex]

@X-dark Could you please check this candidate patch? #2649

[X-dark]

Sorry for not having been able to test this sooner. It does fix this bug. Thank you.