Bug: Resync required everytime when using skip_file on CLI

[theh0lyc0w]

Describe the bug

Observed behaviour: When setting the skip_file config option through the CLI, onedrive sees this as a CLI override of the config file even if the CLI provided option is the same. The application exits and requires starting it with the resync option.
The config file seems to be created automatically by the application and contains only the default sync_dir.

Expected behaviour: When the same CLI provided skip_file options is used, a resync is not required.

I run onedrive using the docker image and prefer setting the config option through environment variables / CLI instead of modifying the config file. This used to work in versions prior to 2.5.

Operating System Details

== Host OS ==
Linux nuc 5.15.0-127-generic #137-Ubuntu SMP Fri Nov 8 15:21:01 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

Client Installation Method

From Distribution Package

OneDrive Account Type

Personal

What is your OneDrive Application Version

onedrive v2.5.3-27-g71a71da

What is your OneDrive Application Configuration

== Container ==
Application version                          = onedrive v2.5.3-27-g71a71da
Compiled with                                = LDC 2100
Curl version                                 = libcurl/8.11.1 OpenSSL/3.0.15 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3 OpenLDAP/2.5.13
User Application Config path                 = /root/.config/onedrive
System Application Config path               = /etc/onedrive
Applicable Application 'config' location     = /root/.config/onedrive/config
Configuration file found in config location  = false - using application defaults
Applicable 'sync_list' location              = /root/.config/onedrive/sync_list
Applicable 'items.sqlite3' location          = /root/.config/onedrive/items.sqlite3
Config option 'drive_id'                     = 
Config option 'sync_dir'                     = ~/OneDrive
Config option 'enable_logging'               = false
Config option 'log_dir'                      = /var/log/onedrive
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'permanent_delete'             = false
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.3-27-g71a71da
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 0
Config option 'threads'                      = 8
Config option 'max_curl_idle'                = 120
Compile time option --enable-notifications   = false

Selective sync 'sync_list' configured        = false

Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false

What is your 'curl' version

== Container ==
curl 8.11.1 (x86_64-pc-linux-gnu) libcurl/8.11.1 GnuTLS/3.7.9 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 libssh2/1.10.0 nghttp2/1.52.0 ngtcp2/1.6.0 nghttp3/1.4.0 librtmp/2.3 OpenLDAP/2.5.13
Release-Date: 2024-12-11, security patched: 8.11.1-1~bpo12+1
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

== Container ==
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/DG2S37ZG7B7SZYO3BW7FJBILI4:/var/lib/docker/overlay2/l/3F5WWGZD4SJMV3LQWQNA63GGRS:/var/lib/docker/overlay2/l/3ZLVL7B76FBFWHAZR3OLTHI6WY:/var/lib/docker/overlay2/l/TLSYZSKFSPPOHPGIYCZNRKRPBY:/var/lib/docker/overlay2/l/DTUZQP7L6YNVF54C4HTCAQ4OIJ:/var/lib/docker/overlay2/l/TFM7LQ3G5J6IOJRMAFTGB5BIDH:/var/lib/docker/overlay2/l/GZF6PNXNW5OYDANZC6WF2NB6AO:/var/lib/docker/overlay2/l/3SZIK5RAMC62SV3WZGFFLNCGM5:/var/lib/docker/overlay2/l/QNV3MAMRGZI6R2X7STYO6JTSEL:/var/lib/docker/overlay2/l/AA4PJ45F3XHE4GFMMORAPQLKQF:/var/lib/docker/overlay2/l/6T6ZZBKRZCFZU7LOTRNQOKMXZQ,upperdir=/var/lib/docker/overlay2/b066f235d48e08d739c0901bb67ac7068d4480beeb5d01c0f7e911a43296c7a1/diff,workdir=/var/lib/docker/overlay2/b066f235d48e08d739c0901bb67ac7068d4480beeb5d01c0f7e911a43296c7a1/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup type cgroup2 (ro,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,inode64)
/dev/sda2 on /mnt/mountcheck type exfat (ro,nosuid,nodev,noexec,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,iocharset=utf8,errors=remount-ro)
/dev/sda2 on /onedrive/data type exfat (rw,nosuid,nodev,noexec,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,iocharset=utf8,errors=remount-ro)
/dev/mapper/ubuntu--vg-ubuntu--lv on /onedrive/conf type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /etc/resolv.conf type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /etc/hostname type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /etc/hosts type ext4 (rw,relatime)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/asound type tmpfs (ro,relatime,inode64)
tmpfs on /proc/acpi type tmpfs (ro,relatime,inode64)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
tmpfs on /proc/scsi type tmpfs (ro,relatime,inode64)
tmpfs on /sys/firmware type tmpfs (ro,relatime,inode64)
tmpfs on /sys/devices/virtual/powercap type tmpfs (ro,relatime,inode64)

What are all your local file system partition types

NAME        FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0                                              
loop1                                              
loop2                                              
loop3                                              
loop4                                              
loop5                                              
sda                                                
|-sda1                                             
`-sda2                               675.5G    27% /onedrive/data
                                                   /mnt/mountcheck
nvme0n1                                            
|-nvme0n1p1                                        
|-nvme0n1p2                                        
`-nvme0n1p3

How do you use 'onedrive'

OneDrive account/folder is used on multiple systems.

The onedrive application syncs this for backup purposes once every hour (download only with cleanup). The application syncs only one account.

Steps to reproduce the behaviour

1. remove existing config file
2. start container with skip_file and resync options set through CLI
3. let the sync complete
4. stop container
5. start container with same skip_file option
6. application exits with message: "An application configuration change has been detected where a --resync is required"

Complete Verbose Log Output

# Running container as user: onedrive
# Base Args: --monitor
# We are performing debug output
# Adding --verbose --verbose
# We are synchronizing in download-only mode
# Adding --download-only
# We are cleaning up local files that are not present online
# Adding --cleanup-local-files
# We are skipping files based on the provided pattern
# Adding --skip-file ARG
# We are setting the monitor interval
# Adding --monitor-interval ARG
# We are setting the monitor log frequency
# Adding --monitor-log-frequency ARG
# Using arguments: --monitor-log-frequency 1 --monitor-interval 3600 --skip-file ~*|.~*|*.tmp|*.swp|*.partial|*.mp4 --cleanup-local-files --download-only --verbose --verbose --monitor --confdir /onedrive/conf --syncdir /onedrive/data
# Changing ownership permissions on /onedrive/data and /onedrive/conf to onedrive:onedrive
# Launching 'onedrive' as onedrive via gosu
DEBUG: Application started
DEBUG: Process ID: 7F2F5EF0EA00
DEBUG: User UID:   1000
DEBUG: User GID:   1000
DEBUG: User Name:  onedrive
DEBUG: Application Version: onedrive v2.5.3-27-g71a71da
DEBUG: Application Compiled With: LDC 2100
DEBUG: Passed in 'cliArgs': ["/usr/local/bin/onedrive", "--monitor-log-frequency", "1", "--monitor-interval", "3600", "--skip-file", "~*|.~*|*.tmp|*.swp|*.partial|*.mp4", "--cleanup-local-files", "--download-only", "--monitor", "--syncdir", "/onedrive/data"]
DEBUG: Note: --confdir and --verbose are not listed in 'cliArgs' array
DEBUG: Passed in --confdir if present: /onedrive/conf
DEBUG: Passed in --verbose count if present: 2
DEBUG: runtime_environment: HOME environment variable detected, expansion of '~' should be possible
DEBUG: runtime_environment: Calculated defaultHomePath: /home/onedrive
DEBUG: configDirName: CLI override to set configDirName to: /onedrive/conf
DEBUG: refreshTokenFilePath =   /onedrive/conf/refresh_token
DEBUG: deltaLinkFilePath =      /onedrive/conf/delta_link
DEBUG: databaseFilePath =       /onedrive/conf/items.sqlite3
DEBUG: databaseFilePathDryRun = /onedrive/conf/items-dryrun.sqlite3
DEBUG: uploadSessionFilePath =  /onedrive/conf/session_upload
DEBUG: userConfigFilePath =     /onedrive/conf/config
DEBUG: syncListFilePath =       /onedrive/conf/sync_list
DEBUG: systemConfigFilePath =   /etc/onedrive/config
DEBUG: configBackupFile =       /onedrive/conf/.config.backup
DEBUG: configHashFile =         /onedrive/conf/.config.hash
DEBUG: syncListHashFile =       /onedrive/conf/.sync_list.hash
DEBUG: Reading configuration file: /onedrive/conf/config
DEBUG: Configuration file successfully loaded
DEBUG: Configuring Client Side Filtering (Selective Sync)
DEBUG: Configuring skip_dir ...
DEBUG: skip_dir: 
DEBUG: Selective Sync Directory Mask: Regex!char("^$", "i")
DEBUG: Configuring skip_dir_strict_match ...
DEBUG: skip_dir_strict_match: false
DEBUG: Configuring skip_dotfiles ...
DEBUG: skip_dotfiles: false
DEBUG: Configuring skip_file ...
DEBUG: skip_file: ~*|.~*|*.tmp|*.swp|*.partial|*.mp4
DEBUG: Selective Sync File Mask: Regex!char("^~.*$|^\\.~.*$|^.*\\.tmp$|^.*\\.swp$|^.*\\.partial$|^.*\\.mp4$", "i")
DEBUG: sync_dir: Setting runtimeSyncDirectory from config value 'sync_dir'
DEBUG: sync_dir: Using configured 'sync_dir' path as-is as however SHELL or USER environment variable configuration detected - should be placed in USER home directory
DEBUG: use 'sync_dir' as is - no touch
DEBUG: sync_dir: runtimeSyncDirectory set to: /onedrive/data
DEBUG: --------------- Application Runtime Configuration ---------------
DEBUG: Application version                          = onedrive v2.5.3-27-g71a71da
DEBUG: Compiled with                                = LDC 2100
DEBUG: Curl version                                 = libcurl/8.11.1 OpenSSL/3.0.15 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3 OpenLDAP/2.5.13
DEBUG: User Application Config path                 = /onedrive/conf
DEBUG: System Application Config path               = /etc/onedrive
DEBUG: Applicable Application 'config' location     = /onedrive/conf/config
DEBUG: Configuration file found in config location  = true - using 'config' file values to override application defaults
DEBUG: Applicable 'sync_list' location              = /onedrive/conf/sync_list
DEBUG: Applicable 'items.sqlite3' location          = /onedrive/conf/items.sqlite3
DEBUG: Config option 'drive_id'                     = 
DEBUG: Config option 'sync_dir'                     = /onedrive/data
DEBUG: Config option 'enable_logging'               = false
DEBUG: Config option 'log_dir'                      = /var/log/onedrive
DEBUG: Config option 'disable_notifications'        = false
DEBUG: Config option 'skip_dir'                     = 
DEBUG: Config option 'skip_dir_strict_match'        = false
DEBUG: Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial|*.mp4
DEBUG: Config option 'skip_dotfiles'                = false
DEBUG: Config option 'skip_symlinks'                = false
DEBUG: Config option 'monitor_interval'             = 3600
DEBUG: Config option 'monitor_log_frequency'        = 1
DEBUG: Config option 'monitor_fullscan_frequency'   = 12
DEBUG: Config option 'read_only_auth_scope'         = false
DEBUG: Config option 'dry_run'                      = false
DEBUG: Config option 'upload_only'                  = false
DEBUG: Config option 'download_only'                = true
DEBUG: Config option 'local_first'                  = false
DEBUG: Config option 'check_nosync'                 = false
DEBUG: Config option 'check_nomount'                = false
DEBUG: Config option 'resync'                       = false
DEBUG: Config option 'resync_auth'                  = false
DEBUG: Config option 'cleanup_local_files'          = true
DEBUG: Config option 'classify_as_big_delete'       = 1000
DEBUG: Config option 'disable_upload_validation'    = false
DEBUG: Config option 'disable_download_validation'  = false
DEBUG: Config option 'bypass_data_preservation'     = false
DEBUG: Config option 'no_remote_delete'             = false
DEBUG: Config option 'remove_source_files'          = false
DEBUG: Config option 'sync_dir_permissions'         = 700
DEBUG: Config option 'sync_file_permissions'        = 600
DEBUG: Config option 'space_reservation'            = 52428800
DEBUG: Config option 'permanent_delete'             = false
DEBUG: Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
DEBUG: Config option 'azure_ad_endpoint'            = 
DEBUG: Config option 'azure_tenant_id'              = 
DEBUG: Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.3-27-g71a71da
DEBUG: Config option 'force_http_11'                = false
DEBUG: Config option 'debug_https'                  = false
DEBUG: Config option 'rate_limit'                   = 0
DEBUG: Config option 'operation_timeout'            = 3600
DEBUG: Config option 'dns_timeout'                  = 60
DEBUG: Config option 'connect_timeout'              = 10
DEBUG: Config option 'data_timeout'                 = 60
DEBUG: Config option 'ip_protocol_version'          = 0
DEBUG: Config option 'threads'                      = 8
DEBUG: Config option 'max_curl_idle'                = 120
DEBUG: Compile time option --enable-notifications   = false
DEBUG: 
DEBUG: Selective sync 'sync_list' configured        = false
DEBUG: 
DEBUG: Config option 'sync_business_shared_items'   = false
DEBUG: 
DEBUG: Config option 'webhook_enabled'              = false
DEBUG: 
DEBUG: --------------------DEVELOPER_OPTIONS----------------------------
DEBUG: Config option 'force_children_scan'          = false
DEBUG: 
DEBUG: -----------------------------------------------------------------
DEBUG: Configuring default new folder permissions as: 700
DEBUG: Configuring default new file permissions as: 600
DEBUG: skip_file: CLI override of config file option, --resync needed
DEBUG: 
DEBUG: An application configuration change has been detected where a --resync is required
DEBUG: 
DEBUG: Exit scope was called
DEBUG: performSynchronisedExitProcess called by: exitScope
DEBUG: CurlEngine releaseAllCurlInstances() called
DEBUG: CurlEngine curlEnginePool size to release: 0
DEBUG: CurlEngine releaseAllCurlInstances() completed
DEBUG: Shutting down Client Side Filtering instance
DEBUG: Shutdown of Client Side Filtering instance is complete
DEBUG: Shutting down Application Configuration instance
DEBUG: Shutdown of Application Configuration instance is complete
DEBUG: Application is exiting
DEBUG: Shutting down Application Logging instance

Screenshots

No response

Other Log Information or Details

No response

Additional context

I used the latest edge version of the container from december 16th 2024. In my setup the entrypoint.sh does have some minor modifications to support additional CLI options through environment variables and check whether the mount point with the data is correctly mounted.

skip file option used: *|.|.tmp|.swp|.partial|*.mp4

[abraunegg]

This is not a bug.

You are "changing* the defaults when using --skip-file via the CLI.

Please read:

https://github.com/abraunegg/onedrive/blob/master/docs/usage.md#using-client-side-filtering-rules-to-determine-what-should-be-synced-with-microsoft-onedrive

[theh0lyc0w]

Thanks for your reply. Maybe the description was not clear enough.

I understand resync is required once when changing from the default to a custom skip_file setting.

But a resync is required everytime the application starts, even when the custom skip_file setting does not change. This behaviour was different pre v2.5.

When changing skip_file in the config file I need to do a resync only once and after that it is not required anymore.

I would expect the same behaviour when setting skip_file through the CLI?

[abraunegg]

When changing skip_file in the config file I need to do a resync only once and after that it is not required anymore

Corrrct because the application configuration has been set, updated and this is what is used.

I would expect the same behaviour when setting skip_file through the CLI?

No because you are overriding the configuration. The application order of options are:

Application Defaults -> config file -> CLI options

When you change things with the CLI use that require a resync, this is then required

This behaviour was different pre v2.5.

The non requirement of resync when using CLI skip file option was a bug in v2.4.x that you were exploiting. This bug has been fixed in v2.5.x

[theh0lyc0w]

Okay. Thanks. So to set a skip_file filter without requiring a resync each time the application starts, requires this doing this through the config file.

Using the skip_file option on the CLI always requires a resync if it's different from the application default*. Even if you use the same pattern everytime you start the application?

*since I have not configured it in the config file atm.

[abraunegg]

Even if you use the same pattern everytime you start the application?

The application makes checks to ensure there is valid change before making a resync requirement.

[theh0lyc0w]

I don't that is happening in my case. I tried to describe it in Steps to reproduce the behaviour
remove existing config file, but let me expand it a bit more.

1. start container with --skip_file "~*|.~*|*.tmp|*.swp|*.partial|*.mp4" and --resync options set through CLI using environment variables
2. let the resync complete successfully
3. stop and remove container
4. start new container with same skip_file option --skip_file "~*|.~*|*.tmp|*.swp|*.partial|*.mp4", but without resync
5. application exits with message: "An application configuration change has been detected where a --resync is required"

[abraunegg]

You are changing the application defaults by use of --skip-file

The defaults are being changed. This is why resync is being required.

In your scenario when you first initiate the resync - because you changed the application defaults.

The second is the exact same situation you are changing the application defaults.

When you configure the skip_file option in the configuration file - this becomes the new defaults thus on subsequent use does not trigger the --resync requirement.