Bug: A database consistency issue has been caught. #3072

linkwood17 · 2025-01-14T20:45:53Z

Describe the bug

Each sync ends up with a database consistency issue has been caught. Resync runs without problems.
I use that ubuntu machine with onedrive to sync two different accounts. The issue is only seen with one account.
Completely setup new ubuntu VM (different ubuntu version and different curl version) and synced from scratch but still have the same problem.
Also tried with ip_protocol_version=1 and force_http_11=true: same problem

Operating System Details

root@lx43:~# uname -a
Linux lx43 6.8.0-51-generic #52-Ubuntu SMP PREEMPT_DYNAMIC Thu Dec  5 13:09:44 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@lx43:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.1 LTS
Release:        24.04
Codename:       noble

Client Installation Method

From Distribution Package

OneDrive Account Type

Personal

What is your OneDrive Application Version

onedrive v2.5.3-1+np1+1.1

What is your OneDrive Application Configuration

root@lx43:~# /usr/bin/onedrive --confdir=/onedrive/config/nt.hotmail/ --display-config
Reading configuration file: /onedrive/config/nt.hotmail/config
Configuration file successfully loaded
WARNING: Required environment variables required to enable GUI Notifications are not present

WARNING: Your cURL/libcurl version (8.5.0) has known HTTP/2 bugs that impact the use of this client.
         Please report this to your distribution, requesting an update to a newer cURL version, or consider upgrading it yourself for optimal stability.
         Downgrading all client operations to use HTTP/1.1 to ensure maximum operational stability.
         Please read https://github.com/abraunegg/onedrive/blob/master/docs/usage.md#compatibility-with-curl for more information.

Application version                          = onedrive v2.5.3-1+np1+1.1
Compiled with                                = LDC 2106
Curl version                                 = libcurl/8.5.0 OpenSSL/3.0.13 zlib/1.3 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.7 libpsl/0.21.2 (+libidn2/2.3.7) libssh/0.10.6/openssl/zlib nghttp2/1.59.0 librtmp/2.3 OpenLDAP/2.6.7
User Application Config path                 = /onedrive/config/nt.hotmail/
System Application Config path               = /etc/onedrive
Applicable Application 'config' location     = /onedrive/config/nt.hotmail/config
Configuration file found in config location  = true - using 'config' file values to override application defaults
Applicable 'sync_list' location              = /onedrive/config/nt.hotmail/sync_list
Applicable 'items.sqlite3' location          = /onedrive/config/nt.hotmail/items.sqlite3
Config option 'drive_id'                     = 
Config option 'sync_dir'                     = /onedrive/nt.hotmail
Config option 'enable_logging'               = true
Config option 'log_dir'                      = /var/log/onedrive/nt.hotmail/
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = _Davos|_Family
Config option 'skip_dir_strict_match'        = true
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = true
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = true
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 755
Config option 'sync_file_permissions'        = 644
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.3-1+np1+1.1
Config option 'force_http_11'                = true
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 0
Config option 'threads'                      = 8
Config option 'max_curl_idle'                = 120
Environment var 'XDG_RUNTIME_DIR'            = false
Environment var 'DBUS_SESSION_BUS_ADDRESS'   = false
Config option 'notify_file_actions'          = false

Selective sync 'sync_list' configured        = false

Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false

What is your 'curl' version

root@lx43:~# curl --version
curl 8.5.0 (x86_64-pc-linux-gnu) libcurl/8.5.0 OpenSSL/3.0.13 zlib/1.3 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.7 libpsl/0.21.2 (+libidn2/2.3.7) libssh/0.10.6/openssl/zlib nghttp2/1.59.0 librtmp/2.3 OpenLDAP/2.6.7
Release-Date: 2023-12-06, security patched: 8.5.0-2ubuntu10.6
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Network

What are all your system 'mount points'

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=987140k,nr_inodes=246785,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=200940k,mode=755,inode64)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/vda1 on / type ext4 (rw,relatime,discard,errors=remount-ro,commit=30)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=32,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2296)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/vda16 on /boot type ext4 (rw,relatime)
/dev/vda15 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
192.168.0.1:/volume1/public/ on /public type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.1,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=192.168.0.1)
192.168.0.1:/volume1/onedrive/ on /onedrive type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.1,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=192.168.0.1)
tmpfs on /run/user/1101 type tmpfs (rw,nosuid,nodev,relatime,size=200936k,nr_inodes=50234,mode=700,uid=1101,gid=100,inode64)

What are all your local file system partition types

root@lx43:~# lsblk -f
NAME    FSTYPE  FSVER LABEL           UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sr0     iso9660       cidata          2025-01-13-22-08-27-00                              
vda                                                                                       
|-vda1  ext4    1.0   cloudimg-rootfs 372462ea-137d-4e94-9465-a2fbb2a7f4ee   16.1G    12% /
|-vda14                                                                                   
|-vda15 vfat    FAT32 UEFI            A4F8-12D8                              98.2M     6% /boot/efi
`-vda16 ext4    1.0   BOOT            0de9fa52-b0fa-4de2-9fd3-df23fb104826  707.3M    13% /boot

How do you use 'onedrive'

I use that VM to "download" the onedrive files to create a backup with a history.
Download only and then I create a daily snapshot.

Steps to reproduce the behaviour

new ubuntu and onedrive installation.
Run the onedrive as a service with your service file.
First run downloads all data properly.
Second run then produces the error.

Complete Verbose Log Output

root@lx43:~# /usr/bin/onedrive --synchronize --verbose --confdir=/onedrive/config/nt.hotmail/
Reading configuration file: /onedrive/config/nt.hotmail/config
Configuration file successfully loaded
Using 'user' configuration path for application config and state data: /onedrive/config/nt.hotmail/
WARNING: Required environment variables required to enable GUI Notifications are not present
WARNING: The required environment variables to enable GUI Notifications are not available; GUI notifications are disabled

WARNING: Your cURL/libcurl version (8.5.0) has known HTTP/2 bugs that impact the use of this client.
         Please report this to your distribution, requesting an update to a newer cURL version, or consider upgrading it yourself for optimal stability.
         Downgrading all client operations to use HTTP/1.1 to ensure maximum operational stability.
         Please read https://github.com/abraunegg/onedrive/blob/master/docs/usage.md#compatibility-with-curl for more information.

Using the following path to store the runtime application log: /var/log/onedrive/nt.hotmail/

DEPRECIATION WARNING: --synchronize has been deprecated in favour of --sync or -s
DEPRECIATION WARNING: Deprecated commands will be removed in a future release.

Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...

WARNING: Application has been configured to cleanup local files that are not present online.
WARNING: Local data loss MAY occur in this scenario if you are expecting data to remain archived locally.

Downgrading all HTTP operations to HTTP/1.1 due to user configuration
Application Version:  onedrive v2.5.3-1+np1+1.1
Account Type:         personal
Default Drive ID:     fa0759806e59af9b
Default Root ID:      FA0759806E59AF9B!103
Remaining Free Space: 1042.80 GB (1119697974067 bytes)
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /onedrive/nt.hotmail
Generating a /delta response from the OneDrive API from this Item ID: FA0759806E59AF9B!103
Adding OneDrive folder details for processing
Adding 85 OneDrive items for processing from the OneDrive 'root' Folder
...
<snip as there is some sensitive data>
...
Adding 8 OneDrive items for processing from weihnachtskarten
Skipping path - The Microsoft OneNote Notebook '/Documents/Notizbuch von Natascha' is not supported by this client
Finished processing self generated /delta JSON response from the OneDrive API
Processing 9044 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/19] to ensure consistent local state
Processing OneDrive JSON item batch [2/19] to ensure consistent local state
Processing OneDrive JSON item batch [3/19] to ensure consistent local state
Processing OneDrive JSON item batch [4/19] to ensure consistent local state
Processing OneDrive JSON item batch [5/19] to ensure consistent local state
Processing OneDrive JSON item batch [6/19] to ensure consistent local state
Processing OneDrive JSON item batch [7/19] to ensure consistent local state
Processing OneDrive JSON item batch [8/19] to ensure consistent local state
Processing OneDrive JSON item batch [9/19] to ensure consistent local state
Processing OneDrive JSON item batch [10/19] to ensure consistent local state
Processing OneDrive JSON item batch [11/19] to ensure consistent local state
Processing OneDrive JSON item batch [12/19] to ensure consistent local state
Processing OneDrive JSON item batch [13/19] to ensure consistent local state
Processing OneDrive JSON item batch [14/19] to ensure consistent local state
Processing OneDrive JSON item batch [15/19] to ensure consistent local state
Processing OneDrive JSON item batch [16/19] to ensure consistent local state
Processing OneDrive JSON item batch [17/19] to ensure consistent local state
Processing OneDrive JSON item batch [18/19] to ensure consistent local state
Processing OneDrive JSON item batch [19/19] to ensure consistent local state
Skipping path - excluded by skip_dir config: _Davos
Skipping path - excluded by skip_dir config: _Family
Syncing this OneDrive Personal Shared Folder: TSV Bonstetten Chränzli 2025
Generating a /delta response from the OneDrive API from this Item ID: 24470056F5C3E43!14891
Adding OneDrive folder details for processing
Adding 11 OneDrive items for processing from the OneDrive Shared Folder
...
<snip as there is some sensitive information>
...
Finished processing self generated /delta JSON response from the OneDrive API
Processing 56 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
The following generated a broken database tree query:
Drive ID: 024470056f5c3e43
Item ID: 24470056F5C3E43!671
ERROR: A database consistency issue has been caught. A --resync is needed to rebuild the database.

Screenshots

No response

Other Log Information or Details

No response

Additional context

No response

abraunegg · 2025-01-14T21:09:42Z

@linkwood17
Please follow the Basic Troubleshooting Steps: https://github.com/abraunegg/onedrive?tab=readme-ov-file#basic-troubleshooting-steps

Essentially, build your client from 'master' and re-test.

linkwood17 · 2025-01-14T21:48:51Z

Hello @abraunegg,
I just compiled the master version and also upgrade curl to the newer version curl 8.11.0

Still see the application crashing. journal -f shows:

Jan 14 22:40:33 lx43 onedrive[55106]: Finished processing self generated /delta JSON response from the OneDrive API
Jan 14 22:40:51 lx43 onedrive[55106]: Processing 9044 applicable JSON items received from Microsoft OneDrive ............
Jan 14 22:40:51 lx43 onedrive[55106]: Syncing this OneDrive Personal Shared Folder: ./<***dir***>
Jan 14 22:40:51 lx43 onedrive[55106]: Generating a /delta response from the OneDrive API from this Drive ID: 24470056f5c3e43 and Item ID: 24470056F5C3E43!14891 .Attempting to perform a database vacuum to optimise database
Jan 14 22:40:51 lx43 onedrive[55106]: Database vacuum is complete
Jan 14 22:40:52 lx43 onedrive[55106]: core.exception.AssertError@src/itemdb.d(892): Assertion failure
Jan 14 22:40:52 lx43 onedrive[55106]: ----------------
Jan 14 22:40:52 lx43 onedrive[55106]: ??:? _d_assertp [0x5893e28fa920]
Jan 14 22:40:52 lx43 onedrive[55106]: ??:? immutable(char)[] itemdb.ItemDatabase.computePath(const(char)[], const(char)[]) [0x5893e28c11eb]
Jan 14 22:40:52 lx43 onedrive[55106]: ??:? immutable(char)[] syncEngine.SyncEngine.computeItemPath(immutable(char)[], immutable(char)[]) [0x5893e288e4cb]
Jan 14 22:40:52 lx43 onedrive[55106]: ??:? bool syncEngine.SyncEngine.checkJSONAgainstClientSideFiltering(std.json.JSONValue) [0x5893e28937af]
Jan 14 22:40:52 lx43 onedrive[55106]: ??:? std.json.JSONValue syncEngine.SyncEngine.generateDeltaResponse(immutable(char)[]) [0x5893e28a87b1]
Jan 14 22:40:52 lx43 onedrive[55106]: ??:? void syncEngine.SyncEngine.fetchOneDriveDeltaAPIResponse(immutable(char)[], immutable(char)[], immutable(char)[]) [0x5893e2880227]
Jan 14 22:40:52 lx43 onedrive[55106]: ??:? void syncEngine.SyncEngine.syncOneDriveAccountToLocalDisk() [0x5893e287d50d]
Jan 14 22:40:52 lx43 onedrive[55106]: ??:? void main.performStandardSyncProcess(immutable(char)[], monitor.Monitor) [0x5893e2733953]
Jan 14 22:40:52 lx43 onedrive[55106]: ??:? _Dmain [0x5893e273185e]
Jan 14 22:40:52 lx43 systemd[1]: onedrive-nt.hotmail.service: Main process exited, code=exited, status=1/FAILURE
Jan 14 22:40:52 lx43 systemd[1]: onedrive-nt.hotmail.service: Failed with result 'exit-code'.
Jan 14 22:40:52 lx43 systemd[1]: onedrive-nt.hotmail.service: Consumed 28.330s CPU time, 66.8M memory peak, 0B memory swap peak.
Jan 14 22:40:55 lx43 systemd[1]: onedrive-nt.hotmail.service: Scheduled restart job, restart counter is at 2.
Jan 14 22:40:55 lx43 systemd[1]: Started onedrive-nt.hotmail.service - OneDrive Free Client for <***>.
Jan 14 22:40:55 lx43 onedrive[55147]: Reading configuration file: /onedrive/config/nt.hotmail/config
Jan 14 22:40:55 lx43 onedrive[55147]: Configuration file successfully loaded
Jan 14 22:40:55 lx43 onedrive[55147]: Using IPv4 and IPv6 (if configured) for all network operations
Jan 14 22:40:55 lx43 onedrive[55147]: Attempting to contact Microsoft OneDrive Login Service
Jan 14 22:40:56 lx43 onedrive[55147]: Successfully reached Microsoft OneDrive Login Service
Jan 14 22:40:56 lx43 onedrive[55147]: Configuring Global Azure AD Endpoints
Jan 14 22:40:56 lx43 onedrive[55147]: WARNING: Application has been configured to cleanup local files that are not present online.
Jan 14 22:40:56 lx43 onedrive[55147]: WARNING: Local data loss MAY occur in this scenario if you are expecting data to remain archived locally.
Jan 14 22:40:57 lx43 onedrive[55147]: OneDrive synchronisation interval (seconds): 300
Jan 14 22:40:57 lx43 onedrive[55147]: Attempting to contact Microsoft OneDrive Login Service
Jan 14 22:40:58 lx43 onedrive[55147]: Successfully reached Microsoft OneDrive Login Service
Jan 14 22:40:58 lx43 onedrive[55147]: Starting a sync with Microsoft OneDrive

I also could send the full log, but preferable on a private channel
Many thanks

abraunegg · 2025-01-14T21:52:25Z

Please send a complete verbose debug log to the support email address.

To generate the verbose debug log, please read:

https://github.com/abraunegg/onedrive/wiki/Generate-debug-log-for-support

In addition to this, please disable any onedrive systemd service so that it does not interfere with the debug log generation.

abraunegg · 2025-01-15T19:01:04Z

@linkwood17
As per email conversation, your issue is dealing with OneDrive Shared Folders, however this is not a bug that I can fix easily as this is a Microsoft API Bug.

Let me explain this part. For reference please use the debug log as provided.

Your OneDrive Account Drive ID looks like the following:

When the Shared Folder is 'Processed' to be created, (reference: Processing OneDrive Item 8929 of 9046 from API Response Bundle 0) - this JSON is correctly determined to be a Shared Folder, however the JSON data provided by Microsoft is inconsistent:

Your DriveID: fa0759806e59af9b - 16 Characters
Shared Folder DriveID: 24470056f5c3e43 - 15 Characters
SharePoint URL ID: 024470056f5c3e43 - 16 Characters

The OneDrive API is dropping the leading zero in their API response.

When the Shared Folder gets created (reference Processing OneDrive JSON item 427 of 500 as part of JSON Item Batch 18 of 19) the provided JSON again has this 15 character DriveID, thus this is entered into the database as such:

Item("fa0759806e59af9b", "FA0759806E59AF9B!12932", "XXXXXXXX", "", remote, "aRkEwNzU5ODA2RTU5QUY5QiExMjkzMi4y", "adDpGQTA3NTk4MDZFNTlBRjlCITEyOTMyLjYzODcyNDAyMDk4ODYwMDAwMA", 2025-Jan-13 21:54:58.8360888Z, "FA0759806E59AF9B!103", "", "", "24470056f5c3e43", "", "24470056F5C3E43!14891", dir, "Y", "")

When the 'root' (the remote shared folder) details are retrieved, the correct Drive ID Length is provided:

The application failure occurs due to the following:

DEBUG: Attempting to calculate local filesystem path for 024470056f5c3e43 and 24470056F5C3E43!14891
DEBUG: Initial calculated path = XXXXXXXXX
DEBUG: The path we are trying to calculate extends to a OneDrive Shared Folder .. need to perform multiple calculations to calculate the full true local path

This is because when the API data for your Shared Folder is generated, it is looking for a DriveID of '024470056f5c3e43' as per the root details, but your 'Shared Folder' details is incorrect from the API response, thus this is why the application is crashing.

This OneDrive API bug can potentially be worked around, but I need to think of the flow on impacts.

If you have paid support for Microsoft OneDrive, please raise a bug report with them as their API is broken and is the root cause of your issue here.

abraunegg · 2025-01-15T20:07:20Z

@linkwood17
Please can you test the following PR to potentially resolve your issue.

I cannot replicate your issue, thus I need you to test this PR to work around the API Bug.

To test the PR, first install all the require platform dependencies to build the client on your respective platforms. Please read https://github.com/abraunegg/onedrive/blob/master/docs/install.md#building-from-source---high-level-requirements and then follow correctly for your platform.

Once this is done, to clone the PR to resolve your issue, you can use a script like the following:

#!/bin/bash

PR=3075

rm -rf ./onedrive-pr${PR}
git clone https://github.com/abraunegg/onedrive.git onedrive-pr${PR}
cd onedrive-pr${PR}
git fetch origin pull/${PR}/head:pr${PR}
git checkout pr${PR}

# Configure and Build
./configure --enable-debug --enable-notifications; make clean; make;
./onedrive --version

This script will create a local folder called onedrive-pr3075 with the PR version.

To run the PR, you need to run the client from the PR build directory:

./onedrive <any other options needed>

To install the PR, you will need to perform sudo make install to install the PR version to your system.

When running the PR, your version should be: onedrive v2.5.3-34-g021febf or greater.

linkwood17 · 2025-01-15T21:08:06Z

Hi @abraunegg,
I've found multiple threads about a Microsoft bug that is around for over a half a year now. Linked folders are shown as *.url in the Mac/Win OneDrive app. Thats true for the newer shares I've accepted and since then the ondrive crashes.
Funny is that shares that are made a year or more ago, still show as folders and they sync well without a problem.

I've now adapted the config to skip the directory and after a resync the onedrive does not crash anymore.
That is for me a good workaround already, as I do not need that shared directory to be downloaded.

I'm now going to test with your bug fix and tell you the result asap.
(of course, without skipping that directory)

best regards

linkwood17 · 2025-01-15T22:01:30Z

Hi @abraunegg,
PR3075 is now running for quite a while without any problems or error.
It does not sync the contents of the shared folder but this is most probably the MS bug.

By the way, the workaround mentioned before with the skiped directory did not work. It took longer to crash but stil it does.

Thanks for your help and many regards

abraunegg · 2025-01-15T22:04:22Z

Hi @abraunegg, PR3075 is now running for quite a while without any problems or error. It does not sync the contents of the shared folder but this is most probably the MS bug.

By the way, the workaround mentioned before with the skiped directory did not work. It took longer to crash but stil it does.

Thanks for your help and many regards

The Shared Folder should be syncing if you have configured it to do so ....

The PR version works around the API bug - but the data contents should sync if there is no configuration to exlcude it.

abraunegg · 2025-01-16T08:18:15Z

@linkwood17
Please can you provide an updated verbose debug log when using PR3075 so that I can validate a number of items, including why the Shared Folder is not syncing, despite the bug being fixed.

abraunegg · 2025-01-16T19:35:26Z

@linkwood17
Your email system is rejecting email responses:

On the other account, the shared folder was created but not its contents.

As requested I need this debug log from this instance .......

I run the onedrive as systemd service with the given service file, that starts it with –monitor.
Systemctl status onedrive* shows that the service is running but the uptime is always less than 5minutes and the process id is >changing all the time. My monitoring (Zabbix) also mentions that the daemon has been restarted as it’s runtime is less than 10minutes.
I didn’t noticed that before the “upgrade”, when it was running with the ubuntu version of onedrive.

The systemd services will restart itself on failure .. so you have a failure going on. Additionally, when doing debugging and problem analysis, as advised above you must disable any systemd services.

abraunegg · 2025-01-16T19:38:31Z

@linkwood17
From your second log, you have a Microsoft Account Issue:

Please read #2957

I cannot solve this issue - you MUST speak to Microsoft.

abraunegg · 2025-01-16T20:56:13Z

@linkwood17

The 3rd debug log you have provided for the Shared Folder issue is 100% not applicable as it is using the 'master' client version.

The client version you are using is still: DEBUG: Application Version: onedrive v2.5.3-33-g1d00a52

When running the PR debug version you must:

Stop all systemd services
Run the client manually from the PR directory

The PR Client version is onedrive v2.5.3-34-g021febf

Please can you:

Clone the PR client version correctly
Run the PR client correctly as required

abraunegg · 2025-01-16T21:55:20Z

@linkwood17
Thankyou for the updated log - the client is still failing with the original issue. As such I have updated the PR to adjust.

Please can you rebuild your PR Client Version and retest client version onedrive v2.5.3-36-g35c901e or greater

linkwood17 · 2025-01-16T22:31:53Z

Hello @abraunegg,
I've pulled the pr3075 again and it updated some files. But the compiled version is still onedrive v2.5.3-34-g021febf.

abraunegg · 2025-01-16T22:35:04Z

@linkwood17

I've pulled the pr3075 again and it updated some files. But the compiled version is still onedrive v2.5.3-34-g021febf.

Please can you provide the debug log and/or revalidate the process you are using to build the PR.

The script provided above should always delete the old PR directory and pull in the new client version.

linkwood17 · 2025-01-16T23:28:35Z

@abraunegg,
I now could compile it. Somehow the compiler did not work anymore. Maybe some caching? After booting the VM all worked fine.
The new version is now running but it looks like it does not run trough.
I'm debuging now for a half an hour or more and it does not stop.
Also the other account running as service does not generate any logs anymore.
I'll let it run over night and send you an update in a couple of hours.
Best regards

abraunegg · 2025-01-16T23:48:19Z

@linkwood17

The new version is now running but it looks like it does not run trough.
...
I'm debuging now for a half an hour or more and it does not stop.

What do you mean by this ? its running, but not running .. but does not stop ... that indicates to me it is working .. ??

Also the other account running as service does not generate any logs anymore.

The other account has account issues ... that is something this application cannot solve for .....

My advice to you is:

Whilst you are debugging this Shared Folder issue, as per previous advice - disable all your systemd services / other application instances so you can focus on the problem at hand
Ensure you are building the PR client version the correct way
Ensure you are running the PR client in the correct manner as per: https://github.com/abraunegg/onedrive/wiki/Generate-debug-log-for-support

abraunegg · 2025-01-17T21:08:49Z

@linkwood17
Thanks for the confirmation of fix.

Please can you run this PR for an extended period, run multiple times to make sure there are no other issues / side effects from the change done to support the broken Microsoft API response.

abraunegg · 2025-01-18T20:46:36Z

@linkwood17
In reviewing the last provided log file, whilst the application is working for you, I have this nagging itch that there is a potential problem lurking.

As such, I have added more validation in the entire application to check for and validate the root cause (OneDrive Personal DriveId Length != 16 characters) and correct it where possible.

Please can you test this new PR that has all of this expanded validation. This is 100% different from the existing PR you have been currently testing.

Once this is done, to clone the PR to resolve your issue, you can use a script like the following:

#!/bin/bash

PR=3081

rm -rf ./onedrive-pr${PR}
git clone https://github.com/abraunegg/onedrive.git onedrive-pr${PR}
cd onedrive-pr${PR}
git fetch origin pull/${PR}/head:pr${PR}
git checkout pr${PR}

# Configure and Build
./configure --enable-debug --enable-notifications; make clean; make;
./onedrive --version

This script will create a local folder called onedrive-pr3081 with the PR version.

To run the PR, you need to run the client from the PR build directory:

./onedrive <any other options needed>

To install the PR, you will need to perform sudo make install to install the PR version to your system.

When running the PR, your version should be: onedrive v2.5.3-37-g45e6cb9 or greater.

If you can provide your response via GitHub or email that would be greatly appreciated.

abraunegg · 2025-01-18T22:31:21Z

@linkwood17
Thankyou for the updated log using the newer PR version.

As per the debug log, the extra corrections are being used within the application and with any API query.

Please continue to use the updated PR (3081) for your access to Microsoft OneDrive.

abraunegg · 2025-01-20T19:31:48Z

@linkwood17
Any further update regarding this PR ?

abraunegg · 2025-01-20T19:49:02Z

@linkwood17
There is nothing in the log that shows any issue.

Please can you:

Stop the respective systemd service
Edit that systemd service to add debug logging (--verbose --verbose)
Manually perform a resync for that configuration before you restart the systemd service
Restart the respective systemd service

abraunegg · 2025-01-21T18:32:55Z

@linkwood17
Any update on the above in regards to validating client operations without systemd, performing a resync then re-enabling the systemd service?

abraunegg · 2025-01-21T19:18:52Z

@linkwood17
Confirmed 2 x emails received. I think you need to look at a potential issue with how you have things configured .. there is nothing fancy here - DKIM keys are in place & via DNS and I have never had an issue the provided email platform communicating with others via gmx either .... something for you to look into I feel.

abraunegg · 2025-01-21T19:28:41Z

@linkwood17
In both your provided debug log files you have this line:

Applicable 'items.sqlite3' location

Line 53 | Line 26

Please can you provide for me the sqlite file that is specified.

abraunegg · 2025-01-21T19:53:00Z

@linkwood17
Please can you find another way to archive / send that file.

If it resides on an encrypted partition, please move / unencrypt it before archiving as the file as sent cannot be opened:

abraunegg · 2025-01-21T20:27:54Z

@linkwood17
I have received the file now without issue - will analyse and cross check with my test system

abraunegg · 2025-01-22T07:31:24Z

@linkwood17
The latest issue with the PR has been identified with thanks to the provided file.

Please can you rebuild your PR 3081 client. When rebuilt your client version should be onedrive v2.5.3-42-g58e7f65 or greater.

abraunegg · 2025-01-22T07:48:32Z

@linkwood17
When running the updated PR , please also utilise --resync to correct the offending DB entries.

abraunegg · 2025-01-22T18:55:15Z

@linkwood17
~~A further review of yesterdays work has highlighted another issue with the Graph API responses from your debug logs provided 2 days ago which has been corrected this morning.~~

~~Please can you rebuild your client to onedrive v2.5.3-43-g2ee3930 and retest using --resync~~

~~Also I received zero emails, so please can you check your system.~~

abraunegg · 2025-01-22T19:18:49Z

@linkwood17
A further review of yesterdays work has highlighted another issue with the Graph API responses from your debug logs provided 2 days ago which has been corrected this morning.

Please can you rebuild your client to onedrive v2.5.3-44-gf565390 and retest using --resync

Also I received zero emails, so please can you check your system.

abraunegg · 2025-01-22T20:15:05Z

@linkwood17
Thankyou for testing onedrive v2.5.3-44-gf565390 and now your Shared Folder data is being queried - this is a good outcome.

However, this latest crash is happening is that for your main drive, all your Graph API responses are for a lowercase driveid as expected, however, the Graph API responding with uppercase driveid values for your Shared Folder .. which is not correct, and does not align with all the other testing.

As such, I have had to tweak the PR yet again to work around this API response issue.

Please can you rebuild your client and test onedrive v2.5.3-45-g99eab68

abraunegg · 2025-01-22T20:54:30Z

@linkwood17
I have reviewed your debug log feom onedrive v2.5.3-45-g99eab68 and all looks OK.

The true test of this change to support the API issue will be when this is merged into 'master'

abraunegg · 2025-01-23T22:08:50Z

@linkwood17
In reviewing PR #3081 I am 100% uncomfortable with forcing some of the change in this PR to align with how all other debug logs / account data has been.

I have no knowledge why, in your case (this bug report):

Your primary account driveId is lowercase (as per all other accounts I have tested, all other Personal Account debug logs I have reviewed from past issues
Your Shared Folder driveId is lowercase and is not equal to 16 characters when the path details is queried via the API
Your Shared Folder driveId is uppercase when the Shared Folder is actually queried

In PR #3081 , the solution is to force everything to lowercase, and, where required, pad any 15 character driveId with zeros.

As such, PR #3081 I believe (whilst solving your issue) is potentially the incorrect way to solve this problem and has unknown potential blast radius failure for other users.

From https://learn.microsoft.com/

In Microsoft Graph API, the driveId is a unique identifier assigned to each drive, such as a user's OneDrive. This ID is essential for accessing and managing drive resources programmatically.

The driveId should be treated as an opaque string, meaning its internal structure or format should not be relied upon or assumed in application development. This approach ensures that applications remain robust and compatible, even if Microsoft changes the underlying implementation of these identifiers.

Unfortunately Microsoft does not publicly disclose the exact specifications of the driveId, including its length or character composition for any account type. I can only use historic data (bug reports and other) to determine that usually a Personal Account driveId is 16 characters in length and is usually lowercase.

As such, in PR #3081 - forcing any compliance to this (16 characters, lowercase) potentially may have undesired results and impacts.

To combat this, I have created PR #3086 to negate some of the forced changes that #3081 performs, whilst still checking and validating the driveId length and case (which is your actual issue with this bug report)

Please can you test PR #3086 to help validate that this PR resolves your issue - and, if this does, then this is a much better approach to solving the actual issue as this treats the driveId as immutable identifiers and avoid making assumptions about their format or content. This ensures that the application will continue to function correctly, regardless of any changes Microsoft may make to the driveId structure in the future.

To test PR #3086 , you can use a script like the following:

#!/bin/bash

PR=3086

rm -rf ./onedrive-pr${PR}
git clone https://github.com/abraunegg/onedrive.git onedrive-pr${PR}
cd onedrive-pr${PR}
git fetch origin pull/${PR}/head:pr${PR}
git checkout pr${PR}

# Configure and Build
./configure --enable-debug --enable-notifications; make clean; make;
./onedrive --version

This script will create a local folder called onedrive-pr3086 with the PR version.

To run the PR, you need to run the client from the PR build directory:

./onedrive <any other options needed>

To install the PR, you will need to perform sudo make install to install the PR version to your system.

When running the PR, your version should be (at this point in time): onedrive v2.5.3-40-g0a75a21. A version greater than this may indicate you are running a different PR build.

If you could provide confirmation via email including any debug log if this PR fails to operate correctly that would be greatly appreciated.

abraunegg · 2025-01-24T08:51:22Z

@linkwood17
Thanks for the email and debug log. This confirms the new PR #3086 is working as intended:

DEBUG: ONEDRIVE PERSONAL API BUG: The provided 'remoteDriveId' is not 16 Characters in length - fetching correct value from Microsoft Graph API via getDriveIdRoot call
DEBUG: CurlEngine getCurlInstance() called
DEBUG: CurlEngine curlEnginePool current size: 7
DEBUG: CurlEngine broOu5qeMJKQ8jaX time since last use: 1 minute, 20 secs, 783 ms, 37 μs, and 7 hnsecs
DEBUG: CurlEngine was in a valid state - returning existing CurlEngine instance
DEBUG: Using CurlEngine instance ID: broOu5qeMJKQ8jaX
DEBUG: Read token from appConfig
DEBUG: Authorised State: true
DEBUG: Request URL = https://graph.microsoft.com/v1.0/drives/24470056f5c3e43/root
DEBUG: Existing Microsoft OneDrive Access Token Expires: 2025-Jan-24 09:17:56.8870869
DEBUG: CurlEngine cleanup() called on instance id: broOu5qeMJKQ8jaX
DEBUG: OneDrive API releaseCurlEngine() Called
DEBUG: CurlEngine releaseEngine() called on instance id: broOu5qeMJKQ8jaX
DEBUG: CurlEngine curlEnginePool size before release: 6
DEBUG: Release Timestamp for CurlEngine broOu5qeMJKQ8jaX: 2025-Jan-24 07:21:49.7835928Z
DEBUG: CurlEngine cleanup() called on instance id: broOu5qeMJKQ8jaX
DEBUG: CurlEngine curlEnginePool size after release: 7
DEBUG:  - old newDatabaseItem.remoteDriveId = 24470056f5c3e43
DEBUG:  - new newDatabaseItem.remoteDriveId = 024470056F5C3E43

and

DEBUG: ONEDRIVE PERSONAL API BUG: The provided ['parentReference']['driveId'].str from the JSON is not 16 Characters in length - fetching correct value from Microsoft Graph API via getDriveIdRoot call
DEBUG: CurlEngine getCurlInstance() called
DEBUG: CurlEngine curlEnginePool current size: 6
DEBUG: CurlEngine PBUt5FYA8TlyriXR time since last use: 1 minute, 22 secs, 703 ms, 654 μs, and 3 hnsecs
DEBUG: CurlEngine was in a valid state - returning existing CurlEngine instance
DEBUG: Using CurlEngine instance ID: PBUt5FYA8TlyriXR
DEBUG: Read token from appConfig
DEBUG: Authorised State: true
DEBUG: Request URL = https://graph.microsoft.com/v1.0/drives/24470056f5c3e43/root
DEBUG: Existing Microsoft OneDrive Access Token Expires: 2025-Jan-24 09:17:56.8870869
DEBUG: CurlEngine cleanup() called on instance id: PBUt5FYA8TlyriXR
DEBUG: OneDrive API releaseCurlEngine() Called
DEBUG: CurlEngine releaseEngine() called on instance id: PBUt5FYA8TlyriXR
DEBUG: CurlEngine curlEnginePool size before release: 5
DEBUG: Release Timestamp for CurlEngine PBUt5FYA8TlyriXR: 2025-Jan-24 07:21:51.4443577Z
DEBUG: CurlEngine cleanup() called on instance id: PBUt5FYA8TlyriXR
DEBUG: CurlEngine curlEnginePool size after release: 6
DEBUG:  - old newDatabaseItem.remoteDriveId = 24470056f5c3e43 <-- LOGGING OUTPUT NEEDS CHANGE
DEBUG:  - new newDatabaseItem.remoteDriveId = 024470056F5C3E43 <-- LOGGING OUTPUT NEEDS CHANGE
DEBUG: Downgrading all children for this searchItem.driveId (024470056F5C3E43) and searchItem.id (24470056F5C3E43!14891) to an out-of-sync state

I am going to make a slight logging output change, then will commit this PR to 'master'

Many thanks your your assistance and extreme patience with testing the PR releases to get to the right outcome for this issue.

linkwood17 added the Bug Something isn't working label Jan 14, 2025

abraunegg added the Waiting on Response label Jan 14, 2025

abraunegg added OneDrive Personal - Shared Folders OneDrive API Inconsistency OneDrive API Bug OneDrive API Bug and removed Waiting on Response labels Jan 15, 2025

abraunegg added the Unable to reproduce label Jan 15, 2025

abraunegg linked a pull request Jan 15, 2025 that will close this issue

Add OneDrive Personal Drive ID and Remote Drive ID length check #3075

Closed

abraunegg added PR Provided - Needs Testing Waiting on Response labels Jan 15, 2025

abraunegg added this to the v2.5.4 milestone Jan 17, 2025

This comment has been minimized.

Sign in to view

abraunegg added Fixed and removed PR Provided - Needs Testing Waiting on Response labels Jan 17, 2025

abraunegg linked a pull request Jan 19, 2025 that will close this issue

Add OneDrive Personal Drive ID and Remote Drive ID length check #3081

Closed

abraunegg removed a link to a pull request Jan 19, 2025

Add OneDrive Personal Drive ID and Remote Drive ID length check #3075

Closed

abraunegg added In Progress Currently being worked on and removed Fixed labels Jan 24, 2025

abraunegg linked a pull request Jan 24, 2025 that will close this issue

Add OneDrive Personal Drive ID and Remote Drive ID length check #3086

Merged

abraunegg removed a link to a pull request Jan 24, 2025

Add OneDrive Personal Drive ID and Remote Drive ID length check #3081

Closed

abraunegg added Fixed and removed In Progress Currently being worked on labels Jan 24, 2025

abraunegg closed this as completed in #3086 Jan 24, 2025

Bug: A database consistency issue has been caught. #3072

Bug: A database consistency issue has been caught. #3072

Comments

linkwood17 commented Jan 14, 2025

Describe the bug

Operating System Details

Client Installation Method

OneDrive Account Type

What is your OneDrive Application Version

What is your OneDrive Application Configuration

What is your 'curl' version

Where is your 'sync_dir' located

What are all your system 'mount points'

What are all your local file system partition types

How do you use 'onedrive'

Steps to reproduce the behaviour

Complete Verbose Log Output

Screenshots

Other Log Information or Details

Additional context

abraunegg commented Jan 14, 2025

linkwood17 commented Jan 14, 2025 • edited by abraunegg Loading

abraunegg commented Jan 14, 2025

abraunegg commented Jan 15, 2025 • edited Loading

abraunegg commented Jan 15, 2025

linkwood17 commented Jan 15, 2025

linkwood17 commented Jan 15, 2025

abraunegg commented Jan 15, 2025

abraunegg commented Jan 16, 2025

abraunegg commented Jan 16, 2025

abraunegg commented Jan 16, 2025

abraunegg commented Jan 16, 2025

abraunegg commented Jan 16, 2025 • edited Loading

linkwood17 commented Jan 16, 2025

abraunegg commented Jan 16, 2025

linkwood17 commented Jan 16, 2025

abraunegg commented Jan 16, 2025

This comment has been minimized.

This comment has been minimized.

abraunegg commented Jan 17, 2025

abraunegg commented Jan 18, 2025

abraunegg commented Jan 18, 2025

abraunegg commented Jan 20, 2025

abraunegg commented Jan 20, 2025

abraunegg commented Jan 21, 2025

abraunegg commented Jan 21, 2025

abraunegg commented Jan 21, 2025

abraunegg commented Jan 21, 2025

abraunegg commented Jan 21, 2025

abraunegg commented Jan 22, 2025

abraunegg commented Jan 22, 2025

abraunegg commented Jan 22, 2025 • edited Loading

abraunegg commented Jan 22, 2025 • edited Loading

abraunegg commented Jan 22, 2025

abraunegg commented Jan 22, 2025

abraunegg commented Jan 23, 2025

abraunegg commented Jan 24, 2025

linkwood17 commented Jan 14, 2025 •

edited by abraunegg

Loading

abraunegg commented Jan 15, 2025 •

edited

Loading

abraunegg commented Jan 16, 2025 •

edited

Loading

abraunegg commented Jan 22, 2025 •

edited

Loading

abraunegg commented Jan 22, 2025 •

edited

Loading