Full library updates

[brendan-pike]

I was puzzled why so many of my albums didn't contain full details, like track names for example even after a full update. I don't know the cause yet but wanted to start this ticket to get your guidance on debugging it. I've logged into my MPD player which is moode audio running on a raspberrypi and tailed the /var/log/mpd/mpd.log file while the re-indexing was happening. Everything goes well for awhile with each track been listed in the mpd log but then all of a sudden the log just stops and no further albums are indexed. Oddly the update on OMPD carries on happily until the end.
So I'm not fully aware of the relationship between the two, how can I debug what is happening here? Thanks

[ArturSierzant]

Mpd has it's own indexing engine, while O!MPD uses getID3 to read metadata from files. When you update DB in O!MPD, it sends update command to all of defined in configuration -> player profile mpd players. Then O!MPD starts its own update procedure. So updating of mpd and O!MPD are two different, independent processes.
As far as I understand, you have problems with updating mpd's database, right? If yes, you can try to set log_level to verbose in mpd.conf (restart of mpd is required after this). Then in mpd.log you might find some more info.
If you have problems with O!MPD DB, first go to include/config.inc.php (or config.local.inc.php) and change $cfg['debug'] to true. Then try to update DB - in /tmp/update_log.txt you'll find detailed info from update process.

[uriel1998]

Okay, so I'm having this same problem with the album "We are not your kind" - other albums before and since that I've added show up in O!MPD just fine (because it's a great bit of software!) - but this particular album just keeps not having it's data show up. It gets the album art just fine, but it seems as if it can't see the music. (See screenshot). I've also attached the update log, because to me it looks as if it's actually adding the tracks. As far as I can tell, the tracks are properly picked up by MPD (and will play just fine if I start it from MPD itself - you can see where I was doing that in the screenshot as well).

update_log.txt

Any guidance on where to poke next to try to debug this?

[ArturSierzant]

Thanks for reporting this. Do you use lighttpd? If yes, please look at #86 (comment) - point 1. I looked at attached update_log.txt and remembered that I saw similar behaviour (several updates started ) when using lighttpd with DietPi.

[uriel1998]

I'm using Apache, sorry. I've seen some other people in a cursory search showing some problems with server.stream-response-body, but the answers seem to be fixing the code of the program? Again, though, it plays just fine, and it seems to be just that one album. :/

[ArturSierzant]

Strange, when I switch to Apache (normally I use nginx, but for a long time I had been using Apache) everything works OK.
Please try this: remove update_log.txt, go to album view (the one you attached above) and click Update album. When update starts let it finish - do not click anything. Once update stops please show the screen with update progress/results (the one with Structure & image, File info, Cleanup, etc.) and attach new update_log.txt.
And have you tried to do full db update (config -> update all)?

[ArturSierzant]

And please attach Config -> PHP information - maybe it's something with PHP configuration.

[mogwa1]

First of all, thanks for OMPD! I've been using it for many years now.

I also have the exact same problem. These problems started appearing several months ago, probably around the same time the PHP package was upgraded. Now I can't add a single album without this issue.
I'm also using apache. See attached log-file and PHP information:
update_log.txt
phpinfo.txt

Let me know if there is anything else I can help with to debug this problem.

[ArturSierzant]

Thanks for that. In 2f80962 I changed update procedure a bit to prevent such situations. Hope it'll help - please try it and let me know.

[mogwa1]

Thanks for the quick response!
I've just tried the update, but sadly to no avail.
It's still not picking up the file details. I've also tried removing and adding the folders (each time doing an "update all").

[ArturSierzant]

Thanks for checking that. Have you tried to update only that single folder?

[mogwa1]

Yes, I've also tried that. Same result.
Is there anything else that might help with analyzing the problem?

[ArturSierzant]

It looks like track info are not written to track table although they are scanned. Try to look into your server log file - maybe you'll find some clue there.

[mogwa1]

After several hours of debugging, phpmyadmin analysis and understanding the code, I finally found the problem.
It turns out that the "audio_encoder" field contained characters which made the SQL track update fail.
This is the query which happens around line 1326 in update.php.
I had re-enabled the debugging cliLog statements in the code, and that produced the following update_log.txt file if I tried it on one of the failing albums:
update_log.txt
As you can see the encoder field contains strange characters.

If I remove the audio_encoder line from the SQL track update, then suddenly all the failing albums are importing correctly!

So, in short: it seems that the SQL track update is currently escaping all the strings (real_escape_string) but that's still not making sure that they don't contain dodgy characters. My php knowledge is not up to the level of suggesting a solution: might there be a built-in function that strips such characters?

In parallel to this, it might be worthwhile to add a check to see whether that SQL query succeeds or not. If it fails, you could print something to the update_log.txt file and also remove the placeholder track entry that was already created before (just containing the filename and the album reference). These almost-empty entries are causing the strange symptoms described above in the post of @uriel1998 .

I can try to see if I have time to brush up on my php, and make a pull request for this. But no guarantees.

NB: It seems that the dodgy characters are produced by LAME encoder version 3.100 (or perhaps by improper parsing of getId3, which I doubt). I haven't been able to find a way to modify the LAME encoder version field in the mp3 files themselves, so screening for dodgy characters in the update routine might be the only way to solve this without re-encoding the MP3s.

[ArturSierzant]

Great work - thanks for investigating. I tried to reproduce this error in my system - I used LAME 3.100 for Windows (64bit) to create mp3 file. Indeed, it placed some strange character(s?) in encoder TAG, but O!MPD imported that file without any problems. It would be great if you could share one of your file - maybe then I could find a solution.

[mogwa1]

Here is an example of a file that fails to be imported on my setup. (I had to zip the mp3 file):
silence.zip
It's just one minute of silence encoded by lame.

I've added some code to check that the mysqli_query indeed fails on this track. Also, in the database the track entry only contains the path and the album reference.
update_log.txt

NB: If I apply the trick where I don't add the encoder to the query, then this track imports ok with metadata such as "Unknown Artist" etc. in the database.

[mogwa1]

Bingo, I've found the problem: it's a bug in the getID3 package!

That package is checking the following (line 723 of module.audio.mp3.php:
if ($thisfile_mpeg_audio_lame['short_version'] >= 'LAME3.90')
If the lame version is 3.100, then that if-statement will fail! The version string is then extracted as if it were a very old lame version. I think it's then reading a few bytes past the end of the actual version string. That makes sense because I had already noticed that the garbage characters at the end of the version string were different every single time. Maybe the fact that these are random characters, is why you didn't have a problem on your side? If my example file fails on your system, then that might be additional proof.

Anyway, if I manually change the if-statement to be true with version 3.100, then all the mp3s that were failing are suddenly importing without a problem!

I will file a bug report in the getID3 github repo.

In the meantime, since you are using a local copy of getID3, it might be possible to patch it locally in OMPD?

[ArturSierzant]

I used file you attached above: in my case there were no problems with import - strange. But in 80e12fe I modified tagProcessor.inc.php to use ['mpeg']['audio']['LAME']['short_version'] tag if it is present in file. short_version contains only 9 first chars of entire encoder name, so it is LAME3.100 for trouble-making version - all causing problems chars are cut off. It should be good as workaround solution. If it helps in your case, I'll try to add some log info in case of update query fails, as you suggested.
And thanks again for your effort :)

[mogwa1]

I just checked: 80e12fe indeed solves the problem! I can import all failing albums now without any workarounds.
Thanks for the update.

[tomchiverton]

We should close this ?