Duplicate files when syncing

[Alano-i]

Duplicate files when syncing When I checked the synchronization log, I found that there were duplicate files. As shown in the figure, such a large file as 3445.MOV may not be completely synchronized during the first synchronization. When synchronized again, duplicate files appeared. The previously synced files were not deleted, and the newly synced files were not renamed correctly.

Please, how can I solve this problem。
The system I use is truenas scale.

[boredazfcuk]

Sorry, I must have missed this.

Does it do this every time? If you delete those four files and then wait for it to re-run, does it behave the same way?

[Alano-i]

Deleting the resync sometimes works, sometimes the same error occurs. It may be that some video files are too large and cannot be downloaded completely after the first synchronization.

It may be necessary to compare the online and local files for consistency before synchronizing. If it is consistent, skip the download. If it is inconsistent, delete the local file and download it again.

Of course, it is best if you can resume the transmission from a breakpoint.

Thank you for your dedication.

[boredazfcuk]

The underlying application downloads files to a temporary location, and once the download is complete, renames the download to the file that you see. You shouldn't see partially downloaded files in your file list.

If a download is not successful, the temporary file will remain. Upon the next synchronisation, the underlying application will delete the temporary files and start the download again.

There is no support for resuming downloads.

Have you attempted to play one of the smaller files? Is it definitely corrupt? You don't have two files with the same name that require de-duplication?

[boredazfcuk]

See this post:
icloud-photos-downloader/icloud_photos_downloader#155 (comment)

[Alano-i]

The underlying application downloads files to a temporary location, and once the download is complete, renames the download to the file that you see. You shouldn't see partially downloaded files in your file list.

If a download is not successful, the temporary file will remain. Upon the next synchronisation, the underlying application will delete the temporary files and start the download again.

There is no support for resuming downloads.

Have you attempted to play one of the smaller files? Is it definitely corrupt? You don't have two files with the same name that require de-duplication?

I've tried playing smaller files, but it doesn't work.The larger file can be played normally.

Checked the icloud website, there is no file with the same name.

[boredazfcuk]

This is odd. I've not seen this. I'll record some larger videos and see if I can replicate it.

[Alano-i]

This is odd. I've not seen this. I'll record some larger videos and see if I can replicate it.

Thanks. Is this problem related to permissions?

There is another question, is it possible to set crontab task to run the sync instead of setting a time interval.

Because I will cooperate with other scripts to manage photos, if the time interval is set, there is no guarantee that the photos will be up to date when the script for managing photos is running.

[boredazfcuk]

Thanks. Is this problem related to permissions?

It shouldn't be, my container sets the required permissions on files and folders many times during its run, to make sure the correct permisisons are applied.

There is another question, is it possible to set crontab task to run the sync instead of setting a time interval.

Unfortunately not. Running cron inside conatiners is bad practice. It's actually disabled in a lot of the Docker OS images. Docker advises that cron jobs should be run on the host eg: 00 8-20 * * docker run --rm <image/name> </path/to/script.sh> to run a script inside a container and then remove the container afterwards. My container however, is designed to always be running, as it has error checking and healthchecks built into it.

[boredazfcuk]

I'm going to look into having a mode which exits after a single synchronisation run. I've had a couple more people request this feature, so it makes sense. This would allow you to schedule the times you want my container to run via cron on the host computer... Might take a while though as it's probably going to mean a pretty big re-write and I need to think about knock on effect on things like health-checks and stuff.

[Alano-i]

great, thanks.

[boredazfcuk]

I've added a new variable single_pass. Please see the readme for details. You should no be able to use cron on the host machine to start the container at specific times of day.

[Alano-i]

Thanks.

If single_pass = true is set,
then when the container is started at the specified time through cron, each sync will check all icloud photos, and then sync new photos to the local or delete deleted photos.
Is that right？

[boredazfcuk]

Hi,

Yes, if single_pass is set to True the script will just execute once and then exit. This means the container needs to be created with restart set to no so something like:
docker create --restart "no" <rest of command>
or
restart: "no"
if you are using docker compose.

Then you should be able to start it from cron with docker container start <name of container>

[Alano-i]

Thanks for your help.