Bad IDs #125

wills721 · 2019-09-09T22:04:16Z

I have a total of 76 'Bad Ids' issues. I read in comments this is a Google issue. Any workaround?

09-09 15:41:02 WARNING: skipped 76 files listed in /mnt/Photos/gphotos.bad_ids.yaml

gilesknap · 2019-09-10T08:26:31Z

I was under the impression that the cause of this issue had been fixed by Google. I don't get bad IDs anymore. Are these video files by any chance? I do have an outstanding bug in my code which is that intermittent network errors might be mistaken for 'Bad IDs'. To check if this is happening you can just delete the bad ids file and re-run gphotos-sync.

…

On Mon, 9 Sep 2019 at 23:04, wills721 ***@***.***> wrote: I have a total of 76 'Bad Ids' issues. I read in comments this is a Google issue. Any workaround? 09-09 15:41:02 WARNING: skipped 76 files listed in /mnt/Photos/gphotos.bad_ids.yaml — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#125?email_source=notifications&email_token=AAHLRW2YAQEC3AS7XXOBORLQI3B6DA5CNFSM4IVAUMJKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HKJMSGA>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHLRW7Z54TGJBWCKMRQIMDQI3B6DANCNFSM4IVAUMJA> .

wills721 · 2019-09-10T10:42:31Z

All Videos yes. Either .MOV or .VOB extension but mostly .MOV

gilesknap · 2019-09-10T12:17:30Z

Please can delete your bad Ids file from the roor folder, run again and send your logfile from your root folder to [email protected]. Thanks. giles.

…

On Tue, 10 Sep 2019 at 11:42, wills721 ***@***.***> wrote: All Videos yes. Either .MOV or .VOB extension but mostly .MOV — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#125?email_source=notifications&email_token=AAHLRW3GM5NSY36RF7S7PWTQI52ZPA5CNFSM4IVAUMJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6KU5HQ#issuecomment-529878686>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHLRW4WF2DXBJNWZVCRSPDQI52ZPANCNFSM4IVAUMJA> .

wills721 · 2019-09-10T12:41:56Z

I did that. It is strange. The first time I re-ran the badid yaml file re-appeared. Same exact number of files. However the next few times the bad id yaml file did not re-appear (after removing it) however the files in the bad id file (original and next one) did not download. Its as if the bad id MOV & VOB files are now seen as already downloaded?

09-10 09:03:53 gphotos.Main INFO version: 2.9.2, database schema version 5.6
09-10 09:03:53 gphotos.BadIds DEBUG no bad_ids file, bad ids list is empty
09-10 09:03:53 gphotos.GooglePhotosIndex WARNING Indexing Google Photos Files ...
09-10 09:03:53 gphotos.GooglePhotosIndex INFO searching for media start=2019-09-10 09:45:21, end=None, videos=True
09-10 09:03:53 gphotos.GooglePhotosIndex DEBUG mediaItems.search with body:
{'pageToken': None, 'pageSize': 100, 'filters': {'dateFilter': {'ranges': [{'startDate': {'year': 2019, 'month': 9, 'day': 10}, 'endDate': {'year': 3000, 'month': 1, 'day': 1}}]}, 'mediaTypeFilter': {'mediaTypes': ['ALL_MEDIA']}, 'featureFilter': {'includedFeatures': ['NONE']}}}
09-10 09:03:54 gphotos.GooglePhotosIndex DEBUG Skipped Index (already indexed) 1 photos/2019/09/IMG_2168.JPG
09-10 09:03:54 gphotos.LocalData INFO Saving Database ...
09-10 09:03:54 gphotos.LocalData INFO Database Saved.
09-10 09:03:54 gphotos.Main WARNING Done.
09-10 09:03:54 gphotos.Main INFO Elapsed time = 0:00:01.213283

gilesknap · 2019-09-10T14:38:03Z

I should have mentioned that you will need to use the --rescan option, otherwise, it will only look for files newer than the previous scan date. Please resend the log file when you try that. Thanks.

…

On Tue, 10 Sep 2019 at 13:41, wills721 ***@***.***> wrote: I did that. It is strange. The first time I re-ran the badid files re-appeared. Same exact number of files. However the next few times the bad id file did not re-appear (after removing it) however the files in the bad id file (original and next one) did not download. Its as if the bad id MOV & VOB files are now seen as already downloaded? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#125?email_source=notifications&email_token=AAHLRWZJHKWQ6F3B3RYL3YLQI6IZJA5CNFSM4IVAUMJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6K6C6Y#issuecomment-529916283>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHLRWYUKWBBUWGQFLZSR3LQI6IZJANCNFSM4IVAUMJA> .

wills721 · 2019-09-10T15:19:19Z

Email sent to your gmail

gilesknap · 2019-09-10T18:53:04Z

Thanks for the extra info. It does look like you are getting HTTP 500 errors downloading those video files. So this does look like the Google issue that I thought was fixed.

One more question. How long ago did you first run gphotos-sync? If it indexed the items before the fix (a few weeks ago) then they will need to be re-indexed. I think the bug used to deliver bad remote IDs and the DB saves these and tries to use them to get the download URL.

If you do a gphotos-sync with option --flush-index this will rescan everything and re-index the offending files. If this does not fix it then we need to compile a report to Google.

wills721 · 2019-09-10T19:11:25Z

I've only got this running in the last couple days. I will give the --flush-index a shot anyway and let you know.

wills721 · 2019-09-10T20:42:11Z

So, sadly, the --flush-index run ended the same. Same 76 files failed. Same yaml results I sent you :(

gilesknap · 2019-09-10T20:58:54Z

OK, we have confirmed that the Google API bug still exists even though it is fixed for all of my videos.

Please, could you place a few of the videos that you are willing to share into a publicly shared album and update the bug https://issuetracker.google.com/issues/116842164 with a link to the album?

Many thanks - sorry we did not resolve this but Google was responsive on this bug before so you may be lucky.

gilesknap · 2019-09-10T21:01:23Z

closing as a duplicate of #119 (updated with a link here)

wills721 · 2019-09-11T09:44:20Z

So I made a comment up on the google site but did not yet upload any sample. I'm not crazy about uploading my videos -- they are home movies of my kids mostly. However, if you read to the end of the google issue thread seems I am not alone.

gilesknap · 2019-09-11T10:07:38Z

OK, thanks for that. Let's hope they address it. If you don't mind a couple of Google engineers looking at the videos then there is an option to give them permission to look at photos in your account. They did this for my account. If you are OK with that then mention it on the ticket and they may contact you directly to arrange. Cheers, giles.

…

On Wed, 11 Sep 2019 at 10:44, wills721 ***@***.***> wrote: So I made a comment up on the google site but did not yet upload any sample. I'm not crazy about uploading my videos -- they are home movies of my kids mostly. However, if you read to the end of the google issue thread seems I am not alone. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#125?email_source=notifications&email_token=AAHLRWYHOXHCCLCJFB3IA5DQJC4XJA5CNFSM4IVAUMJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6N5J4I#issuecomment-530306289>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHLRW4ZMZZ7F462MLRXQYTQJC4XJANCNFSM4IVAUMJA> .

wills721 · 2019-09-11T10:33:16Z

I'm good with that. I just left that comment up there.

gilesknap · 2019-09-12T07:51:24Z

This got a response from google on buganizer - result.

wills721 · 2019-09-12T12:00:53Z

Thanks -- just saw. Hopefully they fix the fix and its then is fully fixed and there is no more fixes.

gilesknap · 2019-11-03T10:17:22Z

It looks like https://issuetracker.google.com/issues/116842164 has been fixed by Google.

compman723 · 2020-02-10T13:53:43Z

I can't seem to locate the Bad ids file while on windows. Can someone point me to the location of this file? Thanks!

gilesknap · 2020-02-10T13:59:28Z

Hi, If you are running a recent version of gphotos-sync then there is no badids file any more, this is because the bug in google api is fixed.

If gphotos sync gets an error it always reports it now, rather than collecting a list of bad files that will never work.

compman723 · 2020-02-10T14:10:14Z

Thank you for the reply. Is there a way to retrigger a download if there is a failure? or retry all the failures? Also view all the failures?

…

On Mon, Feb 10, 2020 at 7:59 AM Giles Knap ***@***.***> wrote: Hi, If you are running a recent version of gphotos-sync then there is no badids file any more, this is because the bug in google api is fixed. If gphotos sync gets an error it always reports it now, rather than collecting a list of bad files that will never work. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#125?email_source=notifications&email_token=AOFZ3NHZH4GQE3LLTUY6BEDRCFMUDA5CNFSM4IVAUMJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELITFXI#issuecomment-584135389>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOFZ3NCN2HK45XRGFBMP643RCFMUDANCNFSM4IVAUMJA> .

wills721 · 2020-02-10T14:14:27Z

Also, just so I'm sure I have it, what version of gphotos-sync stopped writing a badids file and what is the most recent version?

gilesknap · 2020-02-10T17:12:59Z

The current version is 2.12.4 (but ironically the version number reporting is broken in this release).
2.10.4 Was the first release without bad ids.
The bad ids file would always have appeared in the root of your target folder.

To retrigger scan and downloads use the --rescan option.

gilesknap mentioned this issue Sep 10, 2019

Handle Intermittent Google Service failures #99

Closed

gilesknap mentioned this issue Sep 10, 2019

Google API Bugs #119

Open

gilesknap closed this as completed Sep 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bad IDs #125

Bad IDs #125

wills721 commented Sep 9, 2019

gilesknap commented Sep 10, 2019 via email

wills721 commented Sep 10, 2019

gilesknap commented Sep 10, 2019 via email

wills721 commented Sep 10, 2019 •

edited

Loading

gilesknap commented Sep 10, 2019 via email

wills721 commented Sep 10, 2019

gilesknap commented Sep 10, 2019

wills721 commented Sep 10, 2019 via email •

edited

Loading

wills721 commented Sep 10, 2019

gilesknap commented Sep 10, 2019

gilesknap commented Sep 10, 2019

wills721 commented Sep 11, 2019

gilesknap commented Sep 11, 2019 via email

wills721 commented Sep 11, 2019

gilesknap commented Sep 12, 2019

wills721 commented Sep 12, 2019

gilesknap commented Nov 3, 2019

compman723 commented Feb 10, 2020

gilesknap commented Feb 10, 2020

compman723 commented Feb 10, 2020 via email

wills721 commented Feb 10, 2020

gilesknap commented Feb 10, 2020

Bad IDs #125

Bad IDs #125

Comments

wills721 commented Sep 9, 2019

gilesknap commented Sep 10, 2019 via email

wills721 commented Sep 10, 2019

gilesknap commented Sep 10, 2019 via email

wills721 commented Sep 10, 2019 • edited Loading

gilesknap commented Sep 10, 2019 via email

wills721 commented Sep 10, 2019

gilesknap commented Sep 10, 2019

wills721 commented Sep 10, 2019 via email • edited Loading

wills721 commented Sep 10, 2019

gilesknap commented Sep 10, 2019

gilesknap commented Sep 10, 2019

wills721 commented Sep 11, 2019

gilesknap commented Sep 11, 2019 via email

wills721 commented Sep 11, 2019

gilesknap commented Sep 12, 2019

wills721 commented Sep 12, 2019

gilesknap commented Nov 3, 2019

compman723 commented Feb 10, 2020

gilesknap commented Feb 10, 2020

compman723 commented Feb 10, 2020 via email

wills721 commented Feb 10, 2020

gilesknap commented Feb 10, 2020

wills721 commented Sep 10, 2019 •

edited

Loading

wills721 commented Sep 10, 2019 via email •

edited

Loading