downloader fails #139

huangbiyong · 2019-06-12T01:38:23Z

Traceback (most recent call last):
File "/Users/chhu02/anaconda3/bin/icloudpd", line 10, in
sys.exit(main())
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/icloudpd/base.py", line 229, in main
client_id=os.environ.get("CLIENT_ID"),
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/icloudpd/authentication.py", line 32, in authenticate
client_id=client_id)
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/pyicloud_ipd/base.py", line 193, in init
self.authenticate()
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/pyicloud_ipd/base.py", line 216, in authenticate
raise PyiCloudFailedLoginException(msg, error)
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Failed to validate the credentials from cookie',))

jkalamarz · 2019-06-12T03:42:01Z

I got the same. It seems some change on Apple side. HomeAssistant's comment:

++++ Breaking News +++++

FYI - It has been reported that Apple changed something on there end that is breaking the pyicloud.py program that sits between HA components and Apple. There is some issue where authorizing the access to the Apple iCloud account fails. It is happening on all versions of HA and on icloud3 (all versions), along with the base icloud component that is part of HA. I’ve tried it on 0.92.0, 0.93.0, 0.93.2, 0.94.0 & 0.94.1.

Everything works, or seems to, until the account needs to be reauthorized. I’m now keeping up with what is going on and am dead in the water until this issue is resolved.

Gary aka GeeksterGary

menkej · 2019-06-12T13:45:37Z

I got the same. It seems some change on Apple side.
I can confirm this, too. Lets hope pyicloud can fix this soon. I hope I find some time this weekend to work on this as well, if still needed.

huangbiyong · 2019-06-13T01:20:45Z

apple change login logic , resolve:

vipinbeni · 2019-06-13T04:10:51Z

Can any one please implement in java for https://github.com/viable-hartman/InflatableDonkey/tree/TwoFAHacky please help

menkej · 2019-06-13T08:19:45Z

apple change login logic , resolve: ...

Hi @huangbiyong ,
Can you please rephrase what you are meaning with your comment/screenshots? Do you have a fix already for this issue? If so you could send me the files so that I can review/test/checkin.

cheers,
Jörg

huangbiyong · 2019-06-13T10:20:47Z

Can refer to my github: https://github.com/huangbiyong/Apple-iCloud-lastest-API

danielfmolnar · 2019-06-18T12:08:41Z

Hey!
I've managed to make it work. You should get this version of pyicloud
https://github.com/PeterHedley94/pyicloud
and then you should comment out these lines from icloudpd/base.py:

        # if skip_videos and photo.item_type != "image":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos." % photo.filename
        #     )
        #     break
        # if photo.item_type != "image" and photo.item_type != "movie":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos and videos. "
        #         "(Item type was: %s)" % (photo.filename, photo.item_type)
        #     )
        #     break

These lines have to be commented out because otherwise I get the error that the PhotoAsset object has no attribute called "item_type".
It works now, it is downloading photos perfectly.

schorschie · 2019-06-18T17:20:42Z

Hi @danielfmolnar ,

I cloned the repository and run sudo python setup.py install in the pyicloud directory. I also I commented out the mentioned lines.

        # if skip_videos and photo.item_type != "image":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos." % photo.filename
        #     )
        #     break
        # if photo.item_type != "image" and photo.item_type != "movie":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos and videos. "
        #         "(Item type was: %s)" % (photo.filename, photo.item_type)
        #     )
        #     break

But I still get the "same" error message:

2019-06-18 19:14:31 DEBUG    Authenticating...
iCloud Password: 
Traceback (most recent call last):
  File "/usr/local/bin/icloudpd", line 11, in <module>
    load_entry_point('icloudpd==1.4.3', 'console_scripts', 'icloudpd')()
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "build/bdist.linux-x86_64/egg/icloudpd/base.py", line 229, in main
  File "build/bdist.linux-x86_64/egg/icloudpd/authentication.py", line 39, in authenticate
  File "/usr/local/lib/python2.7/dist-packages/pyicloud_ipd-0.9.9-py2.7.egg/pyicloud_ipd/base.py", line 193, in __init__
    self.authenticate()
  File "/usr/local/lib/python2.7/dist-packages/pyicloud_ipd-0.9.9-py2.7.egg/pyicloud_ipd/base.py", line 216, in authenticate
    raise PyiCloudFailedLoginException(msg, error)
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError(u'Failed to validate the credentials from cookie',))
2019-06-18 19:14:34 DEBUG    Authenticating...
Traceback (most recent call last):
  File "/usr/local/bin/icloudpd", line 11, in <module>
    load_entry_point('icloudpd==1.4.3', 'console_scripts', 'icloudpd')()
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "build/bdist.linux-x86_64/egg/icloudpd/base.py", line 229, in main
  File "build/bdist.linux-x86_64/egg/icloudpd/authentication.py", line 32, in authenticate
  File "/usr/local/lib/python2.7/dist-packages/pyicloud_ipd-0.9.9-py2.7.egg/pyicloud_ipd/base.py", line 193, in __init__
    self.authenticate()
  File "/usr/local/lib/python2.7/dist-packages/pyicloud_ipd-0.9.9-py2.7.egg/pyicloud_ipd/base.py", line 216, in authenticate
    raise PyiCloudFailedLoginException(msg, error)
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError(u'Failed to validate the credentials from cookie',))

Do you have any Idea what went wrong and could help me with that?

hartmark · 2019-06-20T11:38:47Z

I also have problems with icloudpd, does anyone have any working workarounds?

hartmark · 2019-06-20T11:42:38Z

@danielfmolnar 's fix does it work? If so can anyone build a new docker image for it so I can update my icloud-sync job

ndbroadbent · 2019-06-20T12:06:25Z

Hi everyone, sorry I don't have any time to investigate this issue or work on a fix, but I'm very happy to merge a PR and release a new version + docker image. Thanks!

ndbroadbent · 2019-06-20T12:09:24Z

It looks like we might need a change in the pyicloud library, so here's my fork of pyicloud that icloud_photos_downloader uses (also I use the pyicloud-ipd branch in this repo.)

If someone could please send a PR to update the pyicloud-ipd branch for my pyicloud fork with the required changes, and then make sure that the test suite is passing for icloud_photos_downloader. Then I'm happy to push out the new versions

danielfmolnar · 2019-06-20T12:12:49Z

@ndbroadbent
Hey!
I have forked both libraries. They work. I merged the branch from PeterHedley94.

danielfmolnar · 2019-06-20T12:14:47Z

So, apparently his code managed to fix the authentication issue, but it did not work with your photo downloader library. I had to uncomment a paragraph. Now it works perfectly like it used to.

internationils · 2019-06-20T15:29:50Z

@danielfmolnar can you do a PR for this to https://github.com/ndbroadbent/pyicloud/tree/pyicloud-ipd ? Or at least post diffs of the changes to this thread? Thanks...

DJBenson · 2019-06-20T20:43:53Z

Hey!
I've managed to make it work. You should get this version of pyicloud
https://github.com/PeterHedley94/pyicloud
and then you should comment out these lines from icloudpd/base.py:
        # if skip_videos and photo.item_type != "image":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos." % photo.filename
        #     )
        #     break
        # if photo.item_type != "image" and photo.item_type != "movie":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos and videos. "
        #         "(Item type was: %s)" % (photo.filename, photo.item_type)
        #     )
        #     break
These lines have to be commented out because otherwise I get the error that the PhotoAsset object has no attribute called "item_type".
It works now, it is downloading photos perfectly.

I cloned that version of pyicloud and it doesn't work for me;

Traceback (most recent call last): File "/usr/local/bin/icloud", line 11, in <module> load_entry_point('pyicloud==0.9.1', 'console_scripts', 'icloud')() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/cmdline.py", line 202, in main password.strip() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/base.py", line 194, in __init__ self.authenticate() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/base.py", line 208, in authenticate myICloud = hack.PyiCloudService() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/hack.py", line 232, in __init__ super(PyiCloudService, self).__init__(self) TypeError: super() argument 1 must be type, not classobj

danielfmolnar · 2019-06-21T09:21:14Z

Hey!
I've managed to make it work. You should get this version of pyicloud
https://github.com/PeterHedley94/pyicloud
and then you should comment out these lines from icloudpd/base.py:
        # if skip_videos and photo.item_type != "image":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos." % photo.filename
        #     )
        #     break
        # if photo.item_type != "image" and photo.item_type != "movie":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos and videos. "
        #         "(Item type was: %s)" % (photo.filename, photo.item_type)
        #     )
        #     break
These lines have to be commented out because otherwise I get the error that the PhotoAsset object has no attribute called "item_type".
It works now, it is downloading photos perfectly.
I cloned that version of pyicloud and it doesn't work for me;

Traceback (most recent call last): File "/usr/local/bin/icloud", line 11, in <module> load_entry_point('pyicloud==0.9.1', 'console_scripts', 'icloud')() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/cmdline.py", line 202, in main password.strip() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/base.py", line 194, in __init__ self.authenticate() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/base.py", line 208, in authenticate myICloud = hack.PyiCloudService() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/hack.py", line 232, in __init__ super(PyiCloudService, self).__init__(self) TypeError: super() argument 1 must be type, not classobj

Hey!
You should clone my repo, that version should work. Actually, PeterHedley94's version in itself did not work for me either, that's why I had to merge his changes into ndbroadbent's pyicloud library. Both of them are necessary because ndbroadbent's repo is fully compatible with his photo downloader code and PeterHedley94's changes (adding hack.py) solves the authentication issue.
So I recommend that you pull my forks:
https://github.com/danielfmolnar/icloud_photos_downloader
https://github.com/danielfmolnar/pyicloud
Please let me know if they don't work properly.

danielfmolnar · 2019-06-21T09:29:05Z

@danielfmolnar can you do a PR for this to https://github.com/ndbroadbent/pyicloud/tree/pyicloud-ipd ? Or at least post diffs of the changes to this thread? Thanks...

Hey!
I've created the pull requests for both repos.

DJBenson · 2019-06-21T14:58:03Z

Hey!
I've managed to make it work. You should get this version of pyicloud
https://github.com/PeterHedley94/pyicloud
and then you should comment out these lines from icloudpd/base.py:
        # if skip_videos and photo.item_type != "image":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos." % photo.filename
        #     )
        #     break
        # if photo.item_type != "image" and photo.item_type != "movie":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos and videos. "
        #         "(Item type was: %s)" % (photo.filename, photo.item_type)
        #     )
        #     break
These lines have to be commented out because otherwise I get the error that the PhotoAsset object has no attribute called "item_type".
It works now, it is downloading photos perfectly.
I cloned that version of pyicloud and it doesn't work for me;
Traceback (most recent call last): File "/usr/local/bin/icloud", line 11, in <module> load_entry_point('pyicloud==0.9.1', 'console_scripts', 'icloud')() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/cmdline.py", line 202, in main password.strip() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/base.py", line 194, in __init__ self.authenticate() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/base.py", line 208, in authenticate myICloud = hack.PyiCloudService() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/hack.py", line 232, in __init__ super(PyiCloudService, self).__init__(self) TypeError: super() argument 1 must be type, not classobj
Hey!
You should clone my repo, that version should work. Actually, PeterHedley94's version in itself did not work for me either, that's why I had to merge his changes into ndbroadbent's pyicloud library. Both of them are necessary because ndbroadbent's repo is fully compatible with his photo downloader code and PeterHedley94's changes (adding hack.py) solves the authentication issue.
So I recommend that you pull my forks:
https://github.com/danielfmolnar/icloud_photos_downloader
https://github.com/danielfmolnar/pyicloud
Please let me know if they don't work properly.

Hi @danielfmolnar - no luck I'm afraid.

I've pulled and installed (sudo python setup.py install) both forks but I still get the authentication error. If I run the icloud binary itself I get the repeated prompt to authenticate.

gregbert42 · 2019-06-22T13:53:15Z

This worked for me once I install FIRST icloud_photos_downloader, and THEN pyicloud..
before, i did it in that order, i kept getting the authentication error.

Edit: I am now able to authenticate and create a cookie using icloud - but icloudpd still gives me the error above ( PyiCloudAPIResponseError('Failed to validate the credentials from cookie'))

this is SURELY not the way to do it... but the way I finally made this work was to copy the base.py from the 0.9.1 installation to the 0.10.0 installation

cp /lib/python3.7/site-packages/pyicloud-0.9.1-py3.7.egg/pyicloud/base.py /lib/python3.7/site-packages/pyicloud_ipd-0.10.0-py3.7.egg/pyicloud_ipd/

I am so ashamed... someone please tell me the right way to do this...

DJBenson · 2019-06-22T23:01:18Z

This worked for me once I install FIRST icloud_photos_downloader, and THEN pyicloud..
before, i did it in that order, i kept getting the authentication error.

Edit: I am now able to authenticate and create a cookie using icloud - but icloudpd still gives me the error above ( PyiCloudAPIResponseError('Failed to validate the credentials from cookie'))

this is SURELY not the way to do it... but the way I finally made this work was to copy the base.py from the 0.9.1 installation to the 0.10.0 installation

cp /lib/python3.7/site-packages/pyicloud-0.9.1-py3.7.egg/pyicloud/base.py /lib/python3.7/site-packages/pyicloud_ipd-0.10.0-py3.7.egg/pyicloud_ipd/

I am so ashamed... someone please tell me the right way to do this...

That is indeed working for me too. I'm not to bothered about how it works so long as it does work 😄

simnsdt · 2019-06-23T15:46:57Z

This worked for me once I install FIRST icloud_photos_downloader, and THEN pyicloud..
before, i did it in that order, i kept getting the authentication error.

Edit: I am now able to authenticate and create a cookie using icloud - but icloudpd still gives me the error above ( PyiCloudAPIResponseError('Failed to validate the credentials from cookie'))

this is SURELY not the way to do it... but the way I finally made this work was to copy the base.py from the 0.9.1 installation to the 0.10.0 installation

cp /lib/python3.7/site-packages/pyicloud-0.9.1-py3.7.egg/pyicloud/base.py /lib/python3.7/site-packages/pyicloud_ipd-0.10.0-py3.7.egg/pyicloud_ipd/

I am so ashamed... someone please tell me the right way to do this...

Following those steps I'm getting this error message:
Traceback (most recent call last): File "/usr/local/bin/icloudpd", line 11, in <module> load_entry_point('icloudpd==1.4.4', 'console_scripts', 'icloudpd')() File "/usr/local/lib/python3.7/dist-packages/click-6.7-py3.7.egg/click/core.py", line 722, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/click-6.7-py3.7.egg/click/core.py", line 697, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.7/dist-packages/click-6.7-py3.7.egg/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.7/dist-packages/click-6.7-py3.7.egg/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/home/simon/.local/lib/python3.7/site-packages/icloudpd/base.py", line 277, in main directory = os.path.normpath(directory) File "/usr/lib/python3.7/posixpath.py", line 340, in normpath path = os.fspath(path) TypeError: expected str, bytes or os.PathLike object, not NoneType

schorschie · 2019-06-23T17:25:20Z

Hi @danielfmolnar

So I recommend that you pull my forks:
https://github.com/danielfmolnar/icloud_photos_downloader
https://github.com/danielfmolnar/pyicloud
Please let me know if they don't work properly.

Hi @danielfmolnar - no luck I'm afraid.

I've pulled and installed (sudo python setup.py install) both forks but I still get the authentication error. If I run the icloud binary itself I get the repeated prompt to authenticate.

same for me. I did sudo pip uninstall pyicloud followed by sudo python setup.py install in your repository, and still get the following error:

icloudpd --directory tttest --username [email protected] --recent 50
2019-06-23 19:20:11 DEBUG    Authenticating...
iCloud Password: 
Traceback (most recent call last):
  File "/usr/local/bin/icloudpd", line 11, in <module>
    load_entry_point('icloudpd==1.4.4', 'console_scripts', 'icloudpd')()
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "build/bdist.linux-x86_64/egg/icloudpd/base.py", line 243, in main
  File "build/bdist.linux-x86_64/egg/icloudpd/authentication.py", line 39, in authenticate
  File "/usr/local/lib/python2.7/dist-packages/pyicloud_ipd-0.10.0-py2.7.egg/pyicloud_ipd/base.py", line 193, in __init__
    self.authenticate()
  File "/usr/local/lib/python2.7/dist-packages/pyicloud_ipd-0.10.0-py2.7.egg/pyicloud_ipd/base.py", line 216, in authenticate
    raise PyiCloudFailedLoginException(msg, error)
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError(u'Failed to validate the credentials from cookie',))

The password was correct, I'm sure.

biGdada · 2019-06-26T10:18:12Z

after pulling danielfmolnars forks and copying base.py according to gregbert42s suggestion,
i'm getting 2fa window on my iphone every time i run icloudpd. no matter if i allow or decline the request, icloudpd connects and downloads. any solution?

gregbert42 · 2019-06-26T10:44:17Z

Same happens for me. Important to set it for awake hours :-)

…

On Jun 26, 2019, at 12:18 PM, biGdada ***@***.***> wrote: after pulling danielfmolnars forks and copying base.py according to gregbert42s suggestion, i'm getting 2fa window on my iphone every time i run icloudpd. no matter if i allow or decline the request, icloudpd connects and downloads. any solution? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

andersjohansson · 2019-06-28T09:54:01Z

I believe I have solved this in a somewhat clean way.

If you download and install the versions from my two branches: (start with pyicloud-ipd), it should work.
https://github.com/andersjohansson/pyicloud/tree/pyicloud-ipd-auth-fixes
https://github.com/andersjohansson/icloud_photos_downloader/tree/fix-doc-and-version
(download, extract, and run python setup.py install [--user])

As @ndbroadbent, says, icloud_photos_downloader uses the renamed pyicloud-ipd from the pyicloud-ipd branch (https://github.com/ndbroadbent/pyicloud/tree/pyicloud-ipd). I have merged the changes from @PeterHedleyJHA to this branch, and it all seems to work.

(I also documented the previously undocumented change that download directory has to be specified as --directory, instead as first argument, confused me for a while...)

andersjohansson · 2019-06-28T10:01:37Z

It still seems to launch the 2FA window everytime on the phone though, no idea how to solve that.

hartmark · 2019-06-28T10:12:31Z

Good work! Having the 2FA pop up all the time is not possible for me as I don't have physical access to the device. I'm hoping we can work aound that limitation

ndbroadbent · 2019-09-03T08:26:53Z

Hi everyone! It would be great if someone was able to submit a PR with these changes, where all the tests are passing, and with 100% test coverage (and no problem if you need to delete some code to remove some features. We can re-add something later if there is enough demand.) Cheers!

ndbroadbent · 2019-09-03T08:28:34Z

I think the problem with the current workaround is that it’s a bit of a hack, and it breaks some functionality that other users are depending on. But either way, it would be better to have a working script than a broken one!

ghost · 2019-09-03T08:33:42Z

Fully agree on that point @ndbroadbent =)

boredazfcuk · 2019-09-03T14:48:41Z

I think everyone might be on a wild goose chase...

I have this working in a docker container and I haven't applied any fixes at all. The only issue is this error when generating the cookie:

Traceback (most recent call last):
  File "/usr/bin/icloudpd", line 11, in <module>
    load_entry_point('icloudpd==1.4.4', 'console_scripts', 'icloudpd')()
  File "/usr/lib/python3.7/site-packages/click-6.7-py3.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/click-6.7-py3.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.7/site-packages/click-6.7-py3.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.7/site-packages/click-6.7-py3.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/icloudpd-1.4.4-py3.7.egg/icloudpd/base.py", line 277, in main
  File "/usr/lib/python3.7/posixpath.py", line 340, in normpath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

But I only get the 2FA prompt on my iDevice when I generate my cookie. Every sync after that works as expected.

ghost · 2019-09-03T15:48:12Z

I’m gonna have to double check but I’m pretty sure that’s not what I’m getting.

Is this the same version that’s in @ndbroadbent s container?

biGdada · 2019-09-03T15:59:31Z

what is the status of 2FA prompt right now? has anyone succeeded to to make it work without triggering the 2FA dialog?

boredazfcuk · 2019-09-03T16:01:27Z

I’m gonna have to double check but I’m pretty sure that’s not what I’m getting.

Is this the same version that’s in @ndbroadbent s container?

No, it's using a container I built... Sorry, I should have linked it: https://github.com/boredazfcuk/docker-icloud_photos_downloader

boredazfcuk · 2019-09-03T16:01:48Z

what is the status of 2FA prompt right now? has anyone succeeded to to make it work without triggering the 2FA dialog?

Yes: https://github.com/boredazfcuk/docker-icloud_photos_downloader

boredazfcuk · 2019-09-03T16:04:00Z

Sorry, about line 24. I've not had chance to break it out into it's own file yet as I write most of my stuff in Portainer's web editor and I can't feed additional files into it...

biGdada · 2019-09-03T16:05:54Z

what is the status of 2FA prompt right now? has anyone succeeded to to make it work without triggering the 2FA dialog?

Yes: https://github.com/boredazfcuk/docker-icloud_photos_downloader

what is the right set of commands to install it directly on linux/freebsd (without the docker)?

boredazfcuk · 2019-09-03T17:03:33Z

The dockerfile is just an install script which has all the commands needed to install it on Alpine Linux.

You should be able to adapt it to install on whatever distribution you like.

boredazfcuk · 2019-09-03T20:39:10Z

I've finished tidying the code. I've split out the launch script to a separate file now so it's easier to read.

I've re-built the Docker image and the container from scratch three or four times now and it definitely doesn't have the 2FA prompting issue.

This issue must be some sort of system issue, rather than code problem. If people would verify my findings and report any issues that would be great, thanks.

ghost · 2019-09-04T08:58:43Z

Ok @boredazfcuk, so there's a bunch of things unfortunately.

Volumes:
The --volume statement in your example is hardcoded to boredazfcuk but the script actually checks for /home/${USER}/iCloud. Suggestion would be to use something generic such as /data or /photos instead.

Script init:
During script init you check for /home/${USER}/iCloud/.mounted but that file is never created (unless you do it manually)

CLIOPTIONS:
It seems the script only allows passing of one set of parameters, when I tried to pass both smtp-username and smtp-password it failed.

Finally, and sadly:
Even after fixing those things, it does not sync any photos at all, my photos directory is empty (except for .mounted). See output below:
/ # /usr/local/bin/sync-icloud.sh
2019-09-04 10:44:58 INFO Local account: hyper-1024:sc-media-65537
2019-09-04 10:44:58 INFO Apple account: zzzz@yyyy
2019-09-04 10:44:58 INFO Cookie directory: /config
2019-09-04 10:44:58 INFO iCloud directory: /home/hyper/iCloud
2019-09-04 10:44:58 INFO Command line options:
2019-09-04 10:44:58 INFO Syncronisation for hyper started
2019-09-04 10:44:58 INFO Sync hyper iCloud...
Usage: icloudpd

Error: Got unexpected extra argument ()
/usr/local/bin/sync-icloud.sh: line 79: 2019-09-04 10:45:03 INFO Exit code non-zero - Error: 0: not found
2019-09-04 10:45:03 INFO Set correct file time stamp from exif data...
2019-09-04 10:45:03 INFO Set group and permissions of downloaded files...
2019-09-04 10:45:03 INFO Syncronisation for hyper complete
2019-09-04 10:45:03 INFO Next syncronisation at 16:45

boredazfcuk · 2019-09-04T10:28:49Z

Hi,
Thanks for testing. Thanks for pointing the bits you're having issue with... hopefully some answers:

Volumes point 1:
The --volume statement in the readme has hardcoded values because it's only an example. It's the exact command I used to create my working container. I will expand the readme more to show that.

Volumes point 2: I use /home/${USER}/iCloud as the download destination within the container as I have 3 different accounts that I sync (me, Mrs & kids). Having them sync to the user's home folder means I have an easy visual guide as to which container I'm operating in if I am connected via the shell. They also map to the same location on the host machine for consistency. I could change the download path within the container to something generic, but I see that as a drawback as I'd lose user visibility. Also, it doesn't really matter where they're stored within the container as the destination folder is a bind mount to a different location on the host.

Script init:
Sorry, I should have made it clear that the .mounted file needs to be created manually. I can't automate it as the container has no way of knowing if the underlying volume is mounted or not. I've updated the readme to reflect this.

CLIOPTIONS:
What options did you use? I'm guessing that it's probably a string expansion issue as I tested multiple options with

 --env CLIOPTIONS="--folder-structure={:%Y} -- recent 50"

Edit: just seen you tried passing the SMTP options... You're probably just best leaving them out for the time being. I'm going to look at implementing notification options within my docker file rather than relying on icloudpd. The problem with passing the notification options directly to icloudpd is that you have no way of entering the 2FA code when you get prompted to renew it... This will cause the container to hang indefinitely.

On my to do list is configuring a Healthcheck on the container so that it will show as unhealthy if the cookie has expired... Interestingly the cookie expiry date is visible within the cookie so I'm going to amend my script to extract that and display it on each sync so you get plenty of warning.

I should be able to fire off notifications about cookie expiry in advance so you get a week or two's notice rather than just a "someone's trying to access your account" type message randomly.

ghost · 2019-09-04T13:02:25Z

Hey,

Ah, got it, thanks.
After running the script inside the container I get the following

/usr/local/bin # /usr/bin/icloudpd --directory \"/home/${USER}/iCloud\" --cookie-directory=\"${CONFIGDIR}\" --username
=\"${APPLEID}\" --password=\"${APPLEPASSWORD}\" \""${CLIOPTIONS}"\""
> "
Usage: icloudpd <options>

Error: Invalid value for "-d" / "--directory": Path ""/home/hyper/iCloud"" does not exist.

Could there be an issue with the escaping?

Ps. I also noticed that sometimes you use mix space and = when assigning the parameters (ie. --directory /home/... and --cookie-directory=$CONFIGDIR), both works of course, just my narcissistic brain reacted =)
Ds.

boredazfcuk · 2019-09-04T14:11:39Z

This error is saying that the destination folder for the downloads does not exist inside the container. Are you sure your container was created with two volumes and not just the first one?

--volume icloudpd_boredazfcuk_config:/config \
--volume /home/boredazfcuk/iCloud:/home/boredazfcuk/iCloud \

The first volume is for storing the authentication token so it's available between restarts.
The second volume is basically:

"<path to the folder on the host>:<path to the folder inside the container>"

So yours should look something like:

--voume /home/hyper/iCloud:/home/hyper/iCloud \

The "/home/hyper/iCloud" folder should automatically be created when you specify that second volume.

I've updated the readme so it should be clearer now.

ghost · 2019-09-04T14:13:40Z

Hey,

Yupp, /home/hyper/iCloud most definitely exists:

/home/hyper/iCloud # ls -al
total 0
drwxrwx--- 1 hyper sc-media 28 Sep  4 10:41 .
drwxr-sr-x 1 hyper sc-media 12 Sep  4 10:42 ..
-rw-rw---- 1 hyper sc-media  0 Sep  4 10:41 .mounted
drwxrwx--- 1 hyper sc-media  8 Aug 28 08:45 @eaDir
/home/hyper/iCloud #

... and this is from inside the container of course..
Edit:
I tested some more things. If I remove all escaped quotes (") the command runs like a charm..

boredazfcuk · 2019-09-04T14:19:13Z

Path ""/home/hyper/iCloud"" does not exist.

I think it's something odd is happening with the quotes. I think those double quotes means it's actually using "/home/hyper/iCloud" (quote included) as the path

boredazfcuk · 2019-09-04T15:05:08Z

I suppose I can really remove those, as usernames with spaces in them are not allowed. I'm just in the habit of quoting all my variables so I can deal with spaces.

I also noticed that sometimes you use mix space and = when assigning the parameters

I copy/pasted a lot of the script from one I created to run in an LXC container. That was when the instructions included '='. They've since been removed. See here: #119

boredazfcuk · 2019-09-04T15:05:42Z

If I remove all escaped quotes (") the command runs like a charm..

Just to confirm, you don't get the 2FA prompting issue?

ghost · 2019-09-04T15:08:23Z

No issues with the 2FA!

boredazfcuk · 2019-09-04T15:11:20Z

Winner! Winner!

mdbraber · 2019-09-05T18:42:41Z

Thanks @boredazfcuk and @hyper-ex for building this. Downloading photos now works great (without 2FA prompt) for all "album-Albums" (i.e. named albums I've created my self). For the "official" Apple albums like All Photos, Slo-mo. Screenshots or others it doesn't work ("0 photos"). Any idea what could be going on or how to best debug? Thanks!

boredazfcuk · 2019-09-16T11:06:30Z

@mdbraber Sorry for the late reply, I was busy building a bunch of other containers and forgot to respond...

I've never used the --album functionality to download named albums. I just run it without that option and it pulls "All Photos" which includes all the screen shots, videos etc.

Also, I've now pushed this to Ducker hub so the image can be downloaded with:

docker pull boredazfcuk/icloudpd

rather than having to build it from scratch with the dockerfile

gregbert42 · 2019-09-16T12:57:31Z

I am super appreciative of this... So happy to have it working again seamlessly.

…

On Mon, Sep 16, 2019 at 1:06 PM boredazfcuk ***@***.***> wrote: @mdbraber <https://github.com/mdbraber> Sorry for the late reply, I was busy building a bunch of other containers and forgot to respond... I've never used the --album functionality to download named albums. I just run it without that option and it pulls "All Photos" which includes all the screen shots, videos etc. Also, I've now pushed this to Ducker hub so the image can be downloaded with: docker pull boredazfcuk/icloudpd — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#139>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEBNBYOROYJ5Y2AGXDN5XK3QJ5SDZANCNFSM4HXETO7Q> .

markuskkkl · 2019-11-27T15:36:47Z

Very happy! Great job and thanks @boredazfcuk @hyper-ex !

Compared to my previous build I'm now missing the --album parameter. Please correct me, hope I'm just wrong and the feature to download specific albums is still available.
edit: my fault, album is working :)

menkej · 2020-09-29T18:05:28Z

This issue is resolved, I guess, @ndbroadbent

ndbroadbent · 2020-09-30T16:17:09Z

Thanks @menkej, will close!

This was referenced Jun 28, 2019

Include code to workaround auth problems icloud-photos-downloader/pyicloud#3

Open

Fix doc and auth issue #141

Closed

4gra mentioned this issue Sep 16, 2019

Authentication (2FA) fails #142

Closed

ndbroadbent closed this as completed Sep 30, 2020

downloader fails #139

downloader fails #139

Comments

huangbiyong commented Jun 12, 2019

jkalamarz commented Jun 12, 2019 • edited Loading

menkej commented Jun 12, 2019

huangbiyong commented Jun 13, 2019

vipinbeni commented Jun 13, 2019

menkej commented Jun 13, 2019

huangbiyong commented Jun 13, 2019

danielfmolnar commented Jun 18, 2019 • edited Loading

schorschie commented Jun 18, 2019 • edited Loading

hartmark commented Jun 20, 2019

hartmark commented Jun 20, 2019

ndbroadbent commented Jun 20, 2019

ndbroadbent commented Jun 20, 2019

danielfmolnar commented Jun 20, 2019 • edited Loading

danielfmolnar commented Jun 20, 2019

internationils commented Jun 20, 2019

DJBenson commented Jun 20, 2019

danielfmolnar commented Jun 21, 2019

danielfmolnar commented Jun 21, 2019

DJBenson commented Jun 21, 2019

gregbert42 commented Jun 22, 2019 • edited Loading

DJBenson commented Jun 22, 2019

simnsdt commented Jun 23, 2019

schorschie commented Jun 23, 2019

biGdada commented Jun 26, 2019

gregbert42 commented Jun 26, 2019 via email

andersjohansson commented Jun 28, 2019 • edited Loading

andersjohansson commented Jun 28, 2019

hartmark commented Jun 28, 2019

ndbroadbent commented Sep 3, 2019

ndbroadbent commented Sep 3, 2019

ghost commented Sep 3, 2019

boredazfcuk commented Sep 3, 2019 • edited Loading

ghost commented Sep 3, 2019 • edited by ghost Loading

biGdada commented Sep 3, 2019

boredazfcuk commented Sep 3, 2019

boredazfcuk commented Sep 3, 2019

boredazfcuk commented Sep 3, 2019

biGdada commented Sep 3, 2019

boredazfcuk commented Sep 3, 2019

boredazfcuk commented Sep 3, 2019

ghost commented Sep 4, 2019 • edited by ghost Loading

boredazfcuk commented Sep 4, 2019 • edited Loading

ghost commented Sep 4, 2019 • edited by ghost Loading

boredazfcuk commented Sep 4, 2019 • edited Loading

ghost commented Sep 4, 2019 • edited by ghost Loading

boredazfcuk commented Sep 4, 2019

boredazfcuk commented Sep 4, 2019

boredazfcuk commented Sep 4, 2019

ghost commented Sep 4, 2019

boredazfcuk commented Sep 4, 2019

mdbraber commented Sep 5, 2019

boredazfcuk commented Sep 16, 2019 • edited Loading

gregbert42 commented Sep 16, 2019 via email

markuskkkl commented Nov 27, 2019 • edited Loading

menkej commented Sep 29, 2020

ndbroadbent commented Sep 30, 2020

jkalamarz commented Jun 12, 2019 •

edited

Loading

danielfmolnar commented Jun 18, 2019 •

edited

Loading

schorschie commented Jun 18, 2019 •

edited

Loading

danielfmolnar commented Jun 20, 2019 •

edited

Loading

gregbert42 commented Jun 22, 2019 •

edited

Loading

andersjohansson commented Jun 28, 2019 •

edited

Loading

boredazfcuk commented Sep 3, 2019 •

edited

Loading

ghost commented Sep 3, 2019 •

edited by ghost

Loading

ghost commented Sep 4, 2019 •

edited by ghost

Loading

boredazfcuk commented Sep 4, 2019 •

edited

Loading

ghost commented Sep 4, 2019 •

edited by ghost

Loading

boredazfcuk commented Sep 4, 2019 •

edited

Loading

ghost commented Sep 4, 2019 •

edited by ghost

Loading

boredazfcuk commented Sep 16, 2019 •

edited

Loading

markuskkkl commented Nov 27, 2019 •

edited

Loading