Bug: Upload|Download Loop for AIP Protected File in Monitor Mode #3070

erenoglu · 2025-01-13T10:46:08Z

Describe the bug

Today I noticed that I'm having a loop of a new pptx file being uploaded then downloaded over and over again in monitor mode due to "enrichment" feature of Onedrive, and this keeps on going till I break it.

Operating System Details

up to date Arch linux installation

Client Installation Method

From Source

OneDrive Account Type

Business

What is your OneDrive Application Version

onedrive v2.5.3-29-gc7037f83

What is your OneDrive Application Configuration

sync_dir = "/home/myhome/OneDrive"
disable_upload_validation = "true"
skip_symlinks = "true"
skip_dotfiles = "true"
disable_download_validation = "true"
skip_file = "~*|.~*|*.tmp|*.swp|*.partial|.directory"
notify_file_actions = "true"

What is your 'curl' version

curl 8.11.1 (x86_64-pc-linux-gnu) libcurl/8.11.1 OpenSSL/3.4.0 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libpsl/0.21.5 libssh2/1.11.1 nghttp2/1.64.0 nghttp3/1.7.0
Release-Date: 2024-12-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

home is mounted as a btrfs subvolume

What are all your local file system partition types

btrfs

How do you use 'onedrive'

Onedrive is shared with Windows systems and also web editing, but none were active at the time of the bug report (computers off)

Steps to reproduce the behaviour

compile onedrive from source, do --sync --resync
When completed, do --monitor
Add a pptx file to a folder
Observe that onedrive client will upload it, then detect enrichment, so re-download it, then detect it as a new file, upload it, then download it again due to enrichment, and this goes on in a loop till interrupted.

Complete Verbose Log Output

Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching /delta response from the OneDrive API for Drive ID: ....
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0
Finished processing /delta JSON response from the OneDrive API
No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Sync with Microsoft OneDrive is complete
[M] New local file added: new-file.pptx
New items to upload to Microsoft OneDrive: 1
Total New Data to Upload:        3 MB
Uploading new file: new-file.pptx ... done
WARNING: Skipping upload integrity check for: new-file.pptx
WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded.
WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details.
WARNING: Skipping download integrity check for: new-file.pptx
Downloading file: new-file.pptx ... done
[M] Total number of local file(s) added or changed: 1
[M] Local file changed: new-file.pptx
Uploading modified file: new-file.pptx ... done
WARNING: Skipping upload integrity check for: new-file.pptx
WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded.
WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details.
WARNING: Skipping download integrity check for: new-file.pptx
Downloading file: new-file.pptx ... done
[M] Total number of local file(s) added or changed: 1
[M] Local file changed: new-file.pptx
Uploading modified file: new-file.pptx ... done
WARNING: Skipping upload integrity check for: new-file.pptx
WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded.
WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details.
WARNING: Skipping download integrity check for: new-file.pptx
Downloading file: new-file.pptx ... done
[M] Total number of local file(s) added or changed: 1
[M] Local file changed: new-file.pptx
Uploading modified file: new-file.pptx ... done
WARNING: Skipping upload integrity check for: new-file.pptx
WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded.
WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details.
WARNING: Skipping download integrity check for: new-file.pptx
Downloading file: new-file.pptx ... done
[M] Total number of local file(s) added or changed: 1
[M] Local file changed: new-file.pptx
Uploading modified file: new-file.pptx ... done
WARNING: Skipping upload integrity check for: new-file.pptx
WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded.
WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details.
^C
Received termination signal, attempting to cleanly shutdown application
Monitored directory removed: ....
...
Monitored directory removed: 
Waiting for any existing upload|download process to complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

Screenshots

No response

Other Log Information or Details

No response

Additional context

No response

erenoglu · 2025-01-13T11:12:40Z

By the way, the real file name is something like below, in case & character is messing up stuff:
"ABC D&E F1-GHI Jklmno Prstu_vyz.pptx"

abraunegg · 2025-01-13T19:22:11Z

@erenoglu
I have setup a test system to validate this and I cannot at this stage reproduce this on a stock standard Fedora platform:

[M] New local file added: ./new_pptx_testing_issue_3070.pptx
New items to upload to Microsoft OneDrive: 1
Total New Data to Upload:        32 KB
Uploading new file: ./new_pptx_testing_issue_3070.pptx ... done
WARNING: Skipping upload integrity check for: ./new_pptx_testing_issue_3070.pptx
WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded.
WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details.
WARNING: Skipping download integrity check for: ./new_pptx_testing_issue_3070.pptx
Downloading file: ./new_pptx_testing_issue_3070.pptx ... done
[M] Total number of local file(s) added or changed: 1
[M] Total number of local file(s) added or changed: 1

The new pptx file is added to the local directory, is uploaded, enriched, then downloaded and that is the end of that.

I will setup an Arch Linux system to validate there.

erenoglu · 2025-01-13T19:39:08Z

Thank you. I did a few test on my side now. But it seems related to Azure Information Protection (AIP) protected files.
In Monitor mode:

if I add a simple PPTX file to the folder, it works normally
if I add an AIP encrypted XLSX file, bug occurs, similarly to AIP protected PPTX file
If I add a normal XLSX file, it works normally
I also observe the creation of .partial file with the problem file types.

So it seems something is happening if the file is AIP protected which makes Onedrive believe it's a new file.
Such files cannot be opened in Linux, only from a corporate PC or Office 365 web.

abraunegg · 2025-01-13T19:50:36Z

@erenoglu
Can you please email me a full debug log of this.

I need to look at the JSON data specifically of the response from the OneDrive API to see if there is anything different in that JSON response post the enrichment after upload that the file is AIP protected.

abraunegg · 2025-01-14T19:26:06Z

@erenoglu
Looking at your other issues, when they have been dealing with AIP protected files:

From #3057

From #3066

I can only assume at this point that the JSON response for the downloaded enriched item lacks either the correct size, hash or both in the response, or, the values are different to that what has been downloaded.

To assist with debugging, and to negate needing a full debug log, please can you run the following PR - #3071

First install all the require platform dependencies to build the client on your respective platforms. Please read https://github.com/abraunegg/onedrive/blob/master/docs/install.md#building-from-source---high-level-requirements and then follow correctly for your platform.

Once this is done, to clone the PR to resolve your issue, you can use a script like the following:

#!/bin/bash

PR=3071

rm -rf ./onedrive-pr${PR}
git clone https://github.com/abraunegg/onedrive.git onedrive-pr${PR}
cd onedrive-pr${PR}
git fetch origin pull/${PR}/head:pr${PR}
git checkout pr${PR}

# Configure and Build
./configure --enable-debug --enable-notifications; make clean; make;
./onedrive --version

This script will create a local folder called onedrive-pr3071 with the PR version.

To run the PR, you need to run the client from the PR build directory:

./onedrive <any other options needed>

To install the PR, you will need to perform sudo make install to install the PR version to your system.

When running the PR, your version should be: onedrive v2.5.3-33-g28848e9 or greater.

This will print out:

The response JSON
The values which are then used in the database from the response JSON and the actual local file calculations post download

This will help identify what is going on with these AIP files, and, specifically if there is anything in the JSON response that identifies the file as being AIP protected.

Please can you test this PR to provide the needed information.

erenoglu · 2025-01-14T21:36:47Z

Thanks. Here is the log output with the PR, let me know if you'd need more debugging:
onedrive --version: onedrive v2.5.3-33-g28848e9
Ran a --sync and then ran with --monitor for below log:

No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Sync with Microsoft OneDrive is complete
New items to upload to Microsoft OneDrive: 1
Uploading new file: movements.xlsx ... done
WARNING: Skipping upload integrity check for: movements.xlsx
WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded.
WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details.
ISSUE 3070 DEBUG - Response JSON: {"@microsoft.graph.downloadUrl":"https:\/\/corporate-my.sharepoint.com\/personal\/myemail_corporate_com\/_layouts\/15\/download.aspx?UniqueId=12345678&Translate=false&tempauth=v1.eyJzaXRlaWQiOiJhMzIyOGJiMC0wYjY3LTRiMDctODU0OS03MDczODAwOWNiYmEiLCJhcHBfZGlzcGxheW5hbWUiOiJPbmVEcml2ZSBDbGllbnQgZm9yIExpbnV4IiwiYXBwaWQiOiJkNTBjYTc0MC1jODNmLTRkMWItYjYxNi0xMmM1MTkzODRmMGMiLCJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvbm9raWEtbXkuc2hhcmVwb2ludC5jb21ANWQ0NzE3NTEtOTY3NS00MjhkLTkxN2ItNzBmNDRmOTYzMGIwIiwiZXhwIjoiMTczNjg5MzU3MSJ9.CgoKBHNuaWQSAjY0EgsI2qLmmoO72j0QBRoNMjAuMTkwLjE5MC45OSoseXM2aTJXZEg5SFFGRm9INnVMTDhPUzJVNWRYUkR5aFBBcHFZVll5Rkxjdz0wmAE4AUIQoXeOKFpQAKDbSNPKat3bm0oQaGFzaGVkcHJvb2Z0b2tlblILWyJkdmNfY21wIl1yKTBoLmZ8bWVtYmVyc2hpcHwxMDAzM2ZmZjk1OWM2YjM3QGxpdmUuY29tegEyggESCVEXR111lo1CEZF7cPRPljCwkgEERW1yZZoBCEVyZW5vZ2x1ogEXZW1yZS5lcmVub2dsdUBub2tpYS5jb22qARAxMDAzM0ZGRjk1OUM2QjM3sgE5bXlmaWxlcy53cml0ZSBhbGxmaWxlcy53cml0ZSBhbGxzaXRlcy5yZWFkIGFsbHNpdGVzLndyaXRlyAEB.h5Ph1pe8s4VlTaodG5JsqfVN35duP5e8tIwDLRM2E0Q&ApiVersion=2.0","@odata.context":"https:\/\/graph.microsoft.com\/v1.0\/$metadata#drives('b%21sIsio2cLB0uFSXBzgAnLut-WmSm0fgdDjza9jGNENgE-DHr4-94gQor4P4sATA3d')\/items\/$entity","cTag":"\"c:{41B4579B-852A-4B01-89D5-2FCFA45003AB},1\"","createdBy":{"application":{"displayName":"OneDrive Client for Linux","id":"d50ca740-c83f-4d1b-b616-12c519384f0c"},"user":{"displayName":"Name Surname (Corporate)","email":"[email protected]","id":"id1234567"}},"createdDateTime":"2025-01-14T21:26:11Z","eTag":"\"{41B4579B-852A-4B01-89D5-2FCFA45003AB},1\"","file":{"hashes":{"quickXorHash":"ybkqfgZ8y5qJSBe3xOSEMZaKd0g="},"mimeType":"application\/vnd.openxmlformats-officedocument.spreadsheetml.sheet"},"fileSystemInfo":{"createdDateTime":"2025-01-14T21:26:11Z","lastModifiedDateTime":"2025-01-14T21:26:11Z"},"id":"01Y3SVTJM3K62ECKUFAFFYTVJPZ6SFAA5L","lastModifiedBy":{"application":{"displayName":"OneDrive Client for Linux","id":"d50ca740-c83f-4d1b-b616-12c519384f0c"},"user":{"displayName":"Name Surname (Corporate)","email":"[email protected]","id":"id1234567"}},"lastModifiedDateTime":"2025-01-14T21:26:11Z","name":"movements.xlsx","parentReference":{"driveId":"drive_id_mine","driveType":"business","id":"123456","name":"X01","path":"\/drives\/drive_id_mine\/root:\/mypath","siteId":"abcdert123456a"},"size":3969632,"webUrl":"https:\/\/corporate-my.sharepoint.com\/personal\/myemail_corporate_com\/_layouts\/15\/Doc.aspx?sourcedoc=%7B41B4579B-852A-4B01-89D5-2FCFA45003AB%7D&file=movements.xlsx&action=default&mobileredirect=true"}
ISSUE 3070 DEBUG - Online XOR   : ybkqfgZ8y5qJSBe3xOSEMZaKd0g=
ISSUE 3070 DEBUG - Online Size  : 3969632
ISSUE 3070 DEBUG - Local XOR    : DDf5FKXMZ42NBpVGIxTcFE94r2s=
ISSUE 3070 DEBUG - Local Size   : 4038144
Downloading file: movements.xlsx ... done
Uploading modified file: movements.xlsx ... done
WARNING: Skipping upload integrity check for: movements.xlsx
WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded.
WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details.
ISSUE 3070 DEBUG - Response JSON: {"@microsoft.graph.downloadUrl":"https:\/\/corporate-my.sharepoint.com\/personal\/myemail_corporate_com\/_layouts\/15\/download.aspx?UniqueId=12345678&Translate=false&tempauth=v1.eyJzaXRlaWQiOiJhMzIyOGJiMC0wYjY3LTRiMDctODU0OS03MDczODAwOWNiYmEiLCJhcHBfZGlzcGxheW5hbWUiOiJPbmVEcml2ZSBDbGllbnQgZm9yIExpbnV4IiwiYXBwaWQiOiJkNTBjYTc0MC1jODNmLTRkMWItYjYxNi0xMmM1MTkzODRmMGMiLCJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvbm9raWEtbXkuc2hhcmVwb2ludC5jb21ANWQ0NzE3NTEtOTY3NS00MjhkLTkxN2ItNzBmNDRmOTYzMGIwIiwiZXhwIjoiMTczNjg5MzU4MyJ9.CgoKBHNuaWQSAjY0EgsIzqH0j4S72j0QBRoNMjAuMTkwLjE5MC45OSoseXM2aTJXZEg5SFFGRm9INnVMTDhPUzJVNWRYUkR5aFBBcHFZVll5Rkxjdz0wmAE4AUIQoXeOK2xgALBNd44PpnVEmEoQaGFzaGVkcHJvb2Z0b2tlblILWyJkdmNfY21wIl1yKTBoLmZ8bWVtYmVyc2hpcHwxMDAzM2ZmZjk1OWM2YjM3QGxpdmUuY29tegEyggESCVEXR111lo1CEZF7cPRPljCwkgEERW1yZZoBCEVyZW5vZ2x1ogEXZW1yZS5lcmVub2dsdUBub2tpYS5jb22qARAxMDAzM0ZGRjk1OUM2QjM3sgE5bXlmaWxlcy53cml0ZSBhbGxmaWxlcy53cml0ZSBhbGxzaXRlcy5yZWFkIGFsbHNpdGVzLndyaXRlyAEB.BJcq9owexd5k8UOLJMYS7Y8B-qGL8R-uZs12bB58frc&ApiVersion=2.0","@odata.context":"https:\/\/graph.microsoft.com\/v1.0\/$metadata#drives('b%21sIsio2cLB0uFSXBzgAnLut-WmSm0fgdDjza9jGNENgE-DHr4-94gQor4P4sATA3d')\/items\/$entity","cTag":"\"c:{41B4579B-852A-4B01-89D5-2FCFA45003AB},2\"","createdBy":{"application":{"displayName":"OneDrive Client for Linux","id":"d50ca740-c83f-4d1b-b616-12c519384f0c"},"user":{"displayName":"Name Surname (Corporate)","email":"[email protected]","id":"id1234567"}},"createdDateTime":"2025-01-14T21:26:11Z","eTag":"\"{41B4579B-852A-4B01-89D5-2FCFA45003AB},2\"","file":{"hashes":{"quickXorHash":"ybkqfgZ8y5qJSBe3xOSEMZaKd0g="},"mimeType":"application\/vnd.openxmlformats-officedocument.spreadsheetml.sheet"},"fileSystemInfo":{"createdDateTime":"2025-01-14T21:26:11Z","lastModifiedDateTime":"2025-01-14T21:26:24Z"},"id":"01Y3SVTJM3K62ECKUFAFFYTVJPZ6SFAA5L","lastModifiedBy":{"application":{"displayName":"OneDrive Client for Linux","id":"d50ca740-c83f-4d1b-b616-12c519384f0c"},"user":{"displayName":"Name Surname (Corporate)","email":"[email protected]","id":"id1234567"}},"lastModifiedDateTime":"2025-01-14T21:26:24Z","name":"movements.xlsx","parentReference":{"driveId":"drive_id_mine","driveType":"business","id":"123456","name":"X01","path":"\/drives\/drive_id_mine\/root:\/mypath","siteId":"abcdert123456a"},"size":3969632,"webUrl":"https:\/\/corporate-my.sharepoint.com\/personal\/myemail_corporate_com\/_layouts\/15\/Doc.aspx?sourcedoc=%7B41B4579B-852A-4B01-89D5-2FCFA45003AB%7D&file=movements.xlsx&action=default&mobileredirect=true"}
ISSUE 3070 DEBUG - Online XOR   : ybkqfgZ8y5qJSBe3xOSEMZaKd0g=
ISSUE 3070 DEBUG - Online Size  : 3969632
ISSUE 3070 DEBUG - Local XOR    : NlsLo4ePWFSG+L49WT6r4SGZ0H8=
ISSUE 3070 DEBUG - Local Size   : 4038144
Downloading file: movements.xlsx ... done
Uploading modified file: movements.xlsx ... done
WARNING: Skipping upload integrity check for: movements.xlsx
WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded.
WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details.
ISSUE 3070 DEBUG - Response JSON: {"@microsoft.graph.downloadUrl":"https:\/\/corporate-my.sharepoint.com\/personal\/myemail_corporate_com\/_layouts\/15\/download.aspx?UniqueId=12345678&Translate=false&tempauth=v1.eyJzaXRlaWQiOiJhMzIyOGJiMC0wYjY3LTRiMDctODU0OS03MDczODAwOWNiYmEiLCJhcHBfZGlzcGxheW5hbWUiOiJPbmVEcml2ZSBDbGllbnQgZm9yIExpbnV4IiwiYXBwaWQiOiJkNTBjYTc0MC1jODNmLTRkMWItYjYxNi0xMmM1MTkzODRmMGMiLCJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvbm9raWEtbXkuc2hhcmVwb2ludC5jb21ANWQ0NzE3NTEtOTY3NS00MjhkLTkxN2ItNzBmNDRmOTYzMGIwIiwiZXhwIjoiMTczNjg5MzU5NSJ9.CgoKBHNuaWQSAjY0EgsIoo7E_4S72j0QBRoNMjAuMTkwLjE5MC45OSoseXM2aTJXZEg5SFFGRm9INnVMTDhPUzJVNWRYUkR5aFBBcHFZVll5Rkxjdz0wmAE4AUIQoXeOLmQwAKDbSNSOYJr9wUoQaGFzaGVkcHJvb2Z0b2tlblILWyJkdmNfY21wIl1yKTBoLmZ8bWVtYmVyc2hpcHwxMDAzM2ZmZjk1OWM2YjM3QGxpdmUuY29tegEyggESCVEXR111lo1CEZF7cPRPljCwkgEERW1yZZoBCEVyZW5vZ2x1ogEXZW1yZS5lcmVub2dsdUBub2tpYS5jb22qARAxMDAzM0ZGRjk1OUM2QjM3sgE5bXlmaWxlcy53cml0ZSBhbGxmaWxlcy53cml0ZSBhbGxzaXRlcy5yZWFkIGFsbHNpdGVzLndyaXRlyAEB.2LebmC1AKy9tqexqfEZ1JLGFxNQYjoVVZJ-bSt1labE&ApiVersion=2.0","@odata.context":"https:\/\/graph.microsoft.com\/v1.0\/$metadata#drives('b%21sIsio2cLB0uFSXBzgAnLut-WmSm0fgdDjza9jGNENgE-DHr4-94gQor4P4sATA3d')\/items\/$entity","cTag":"\"c:{41B4579B-852A-4B01-89D5-2FCFA45003AB},3\"","createdBy":{"application":{"displayName":"OneDrive Client for Linux","id":"d50ca740-c83f-4d1b-b616-12c519384f0c"},"user":{"displayName":"Name Surname (Corporate)","email":"[email protected]","id":"id1234567"}},"createdDateTime":"2025-01-14T21:26:11Z","eTag":"\"{41B4579B-852A-4B01-89D5-2FCFA45003AB},3\"","file":{"hashes":{"quickXorHash":"ybkqfgZ8y5qJSBe3xOSEMZaKd0g="},"mimeType":"application\/vnd.openxmlformats-officedocument.spreadsheetml.sheet"},"fileSystemInfo":{"createdDateTime":"2025-01-14T21:26:11Z","lastModifiedDateTime":"2025-01-14T21:26:36Z"},"id":"01Y3SVTJM3K62ECKUFAFFYTVJPZ6SFAA5L","lastModifiedBy":{"application":{"displayName":"OneDrive Client for Linux","id":"d50ca740-c83f-4d1b-b616-12c519384f0c"},"user":{"displayName":"Name Surname (Corporate)","email":"[email protected]","id":"id1234567"}},"lastModifiedDateTime":"2025-01-14T21:26:36Z","name":"movements.xlsx","parentReference":{"driveId":"drive_id_mine","driveType":"business","id":"123456","name":"X01","path":"\/drives\/drive_id_mine\/root:\/mypath","siteId":"abcdert123456a"},"size":3969632,"webUrl":"https:\/\/corporate-my.sharepoint.com\/personal\/myemail_corporate_com\/_layouts\/15\/Doc.aspx?sourcedoc=%7B41B4579B-852A-4B01-89D5-2FCFA45003AB%7D&file=movements.xlsx&action=default&mobileredirect=true"}
ISSUE 3070 DEBUG - Online XOR   : ybkqfgZ8y5qJSBe3xOSEMZaKd0g=
ISSUE 3070 DEBUG - Online Size  : 3969632
ISSUE 3070 DEBUG - Local XOR    : ifNJzQAJ5q7sBMEq3lxUkpqfK80=
ISSUE 3070 DEBUG - Local Size   : 4038144
Downloading file: movements.xlsx ... done
Uploading modified file: movements.xlsx ... done
WARNING: Skipping upload integrity check for: movements.xlsx
WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded.
WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details.
^C
Received termination signal, attempting to cleanly shutdown application
Waiting for any existing upload|download process to complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

abraunegg · 2025-01-14T21:47:03Z

Thanks

OK so there is 100% zero identifier that this is AIP protected in the response, but the downloaded file is totally different size and content wise.

The only fix that can be done here is to update the response JSON with actual values from disk if they are different.

When that response JSON is then inserted to the DB, the existing on disk values match the DB, thus, will not be flagged as a locally changed file, thus will not be re-uploaded.

erenoglu · 2025-01-14T21:54:43Z

Yeah tough situation, if I added --verbose to --monitor command, would it create more logs? Indeed if it's not possible to identify AIP protected files, how can you differentiate :(

abraunegg · 2025-01-14T22:17:53Z

The logging level does not differenciate / change the API response on an uploaded file ... thats where a change would be needed ... and that is a bug for Microsoft.

Whilst not ideal, these need to be true:

Business Account
Disabling download integrity checks
- When not used, the offending file is deleted locally
File size is different
Hash is different

So if all of these 4 items are true, its a high potential to be an AIP file.

Now .. AIP can applied to a number of file types - and when 'protected' can even change the file extention, but in your example the extention is not changing ... so using that as a further identifier cant be used

Using the mimetype in the JSON response also is not going to be helpful.

This is the only solution here:

Update the JSON when the above 4 is true
Do not use the disable download integrity check and let the AIP files fail and be deleted .. as you said yourself they cannot be opened on Linux anyway.

erenoglu · 2025-01-14T22:32:58Z

The thing is, in a normal workday, I may be receiving these from colleagues through email, then save some of them in my local Onedrive folder (in linux) for future reference, and later open them when I need using a local Windows VM or online Onedrive. How can we deal with this? I would prefer all files to be downloaded whether they may be opened locally or not, frankly speaking, so I'd prefer "Update the JSON" option. In not disabling download integrity check, I'm not sure, I would not want to have an incomplete folder as it may confuse me later on seeking where my files may be (I'm sure I wont remember which files had AIP and which file not)

erenoglu · 2025-01-14T22:39:29Z

Maybe as a best of both world, can we enable download integrity check so we have this protection for normal fles, but as we detect above 4 is true, instead of deleting, just update JSON and not delete the file? While errors may happen during transfer, I assume it's not very common.

abraunegg · 2025-01-14T22:44:18Z

The challenge is - enable / disabls integrity checks ... a non-AIP file , that meets then the 4 criteria, will get the JSON updated .. and it may be a valid bad download that then is treated as a good download (JSON fudging)

I would rather only fudge the JSON data only if disable download integrity checking has been enabled ...

It is still not ideal ... but this is the best that is possible due to the Microsoft API constraint in their reaponse JSON.

erenoglu · 2025-01-14T23:11:24Z

Makes sense. and if a user wants the integrity check, he can enable it in expense of losing downloads on AIP files. works...
Maybe also good to document this behavior in the documentation section for the parameter.

abraunegg · 2025-01-15T20:27:03Z

@erenoglu
Please can you rebuild PR #3071 and use application version onedrive v2.5.3-34-gd15b7b8 and test.

This specifically updates the JSON data from Microsoft in this specific use case as discussed.

Please can you provide some feedback.

abraunegg · 2025-01-16T20:57:49Z

@erenoglu
Please can you respond in regards to the updated PR

erenoglu · 2025-01-17T09:18:48Z

Hi, will do hopefully today, just returned from a business trip. but when I compile, the application version is now:
onedrive v2.5.3-39-g9d52d53

with this, here's the log.I did a --sync first, got to a clean state (deleted online & local unnecesary files), did another --sync, then --monitor, then placed an AIP file into the local folder. Do you need --verbose?

Sync with Microsoft OneDrive is complete
New items to upload to Microsoft OneDrive: 1
Uploading new file: movements.xlsx ... done
WARNING: Skipping upload integrity check for: movements.xlsx
WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded.
WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details.
ISSUE 3070 - CHANGING SOURCE JSON DATA BASED ON DOWNLOADED VALUES (quickXorHash,size)
Downloading file: movements.xlsx ... done

abraunegg · 2025-01-17T09:21:29Z

Hi, will do hopefully today, just returned from a business trip. but when I compile, the application version is now: onedrive v2.5.3-39-g9d52d53

with this, here's the log.I did a --sync first, got to a clean state (deleted online & local unnecesary files), did another --sync, then --monitor, then placed an AIP file into the local folder. Do you need --verbose?
Sync with Microsoft OneDrive is complete
New items to upload to Microsoft OneDrive: 1
Uploading new file: movements.xlsx ... done
WARNING: Skipping upload integrity check for: movements.xlsx
WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded.
WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details.
ISSUE 3070 - CHANGING SOURCE JSON DATA BASED ON DOWNLOADED VALUES (quickXorHash,size)
Downloading file: movements.xlsx ... done

Thats the expected output.

So no more continual upload / download ?

erenoglu · 2025-01-17T12:54:52Z

For the one file I tested, it's OK. I'll do a few more tests tonight, maybe also put some files online and see how they propagate locally.

erenoglu · 2025-01-17T13:34:14Z

Some more logs after waiting for some time and Onedrive detecting online changes. What's confusing is some files download with a counter from 0% to 100%, and the ISSUE 3070 message comes in between, not clear for which file (programs.xls)
Also, disable download validation is still true. Shall I change it to "false" and --resync?

Sync with Microsoft OneDrive is complete
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Starting a sync with Microsoft OneDrive
Performing a full scan of online data to ensure consistent local state
Fetching items from the OneDrive API for Drive ID: .... ........CAUTION:    The JSON element transmitted by the Microsoft OneDrive API includes HTML URL encoded items, which may complicate pattern matching and potentially lead to synchronisation problems for this item.
....
Processing 2335 applicable JSON items received from Microsoft OneDrive ..
Number of items to download from Microsoft OneDrive: 4
Downloading file: programs.xlsx ... done
Downloading: meeting-slides ... 0%   |  ETA    --:--:--
Downloading: meeting-slides ... 5%   |  ETA    00:00:57
Downloading: meeting-slides ... 10%  |  ETA    00:00:27
ISSUE 3070 - CHANGING SOURCE JSON DATA BASED ON DOWNLOADED VALUES (quickXorHash,size)
Downloading file: SC MS ENT/Reporting/2025/P01/movements copy 2.xlsx ... done
Downloading: meeting-slides ... 15%  |  ETA    00:00:17
Downloading: meeting-slides ... 20%  |  ETA    00:00:12
Downloading: meeting-slides ... 25%  |  ETA    00:00:09
ISSUE 3070 - CHANGING SOURCE JSON DATA BASED ON DOWNLOADED VALUES (quickXorHash,size)
Downloading file: final-slides ... done
Downloading: meeting-slides ... 30%  |  ETA    00:00:10
Downloading: meeting-slides ... 35%  |  ETA    00:00:08
Downloading: meeting-slides ... 40%  |  ETA    00:00:06
Downloading: meeting-slides ... 45%  |  ETA    00:00:05
Downloading: meeting-slides ... 50%  |  ETA    00:00:04
Downloading: meeting-slides ... 55%  |  ETA    00:00:04
Downloading: meeting-slides ... 60%  |  ETA    00:00:03
Downloading: meeting-slides ... 65%  |  ETA    00:00:03
Downloading: meeting-slides ... 70%  |  ETA    00:00:02
Downloading: meeting-slides ... 75%  |  ETA    00:00:02
Downloading: meeting-slides ... 80%  |  ETA    00:00:01
Downloading: meeting-slides ... 85%  |  ETA    00:00:01
Downloading: meeting-slides ... 90%  |  ETA    00:00:01
Downloading: meeting-slides ... 95%  |  ETA    00:00:01
Downloading: meeting-slides ... 100% | DONE in 00:00:05
Downloading file: meeting-slides ... done
Performing a database consistency and integrity check on locally stored data ..
Scanning the local file system '/home/myhome/OneDrive' for new data to upload .
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching items from the OneDrive API for Drive ID: .... .
No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Sync with Microsoft OneDrive is complete

abraunegg · 2025-01-17T19:46:45Z

Some more logs after waiting for some time and Onedrive detecting online changes. What's confusing is some files download with a counter from 0% to 100%, and the ISSUE 3070 message comes in between

This is because the code change for this has not been cleaned up ... the application output is brute force at the moment to ensure the code change is being hit, to ensure the change JSON structure is happening

I will start cleaning up the PR

If you stop using --disable-download-validation your AIP files will report as failed downloads.

Please read:

abraunegg · 2025-01-17T19:49:10Z

@erenoglu
The PR is now cleaned up.

Please can you perform one last validation using the updated PR - the client version for the PR should be onedrive v2.5.3-42-g91d9739 or greater.

Please can you also read and validate the documentation changes.

Once you confirm, I will merged this PR into 'master'

erenoglu · 2025-01-17T20:36:41Z

OK, now onedrive version is onedrive v2.5.3-43-gbfbd788.

with this, when synced and then monitor mode, and without disabling download validation:

Pasted an AIP protected file into the folder locally, file got uploaded, then downloaded, due to download hash mismatch, the local file got deleted. Now this file is only available in Onedrive online, I assume this will be intended behavior when download validation is enabled.

then I disabled download validation, did a --sync --resync and tried --monitor again:

Pasted an AIP protected file into the folder locally, file got uploaded, then downloaded, no issues.
Uploaded an AIP protected file into Onedrive online with browser, waited for Client to catch up, Client downloaded the file, no issues.
Deleted an AIP protected file locally, while file was open in Online Browser. Client attempted to delete the file and no errors received, file was still open in browser. File did not get deleted from Onedrive. When browser closed, file remained available online. In next sync, file was detected and downloaded.

I checked the documents, they are OK. Hope MS can implement a data field to inform if a file is AIP protected so we could enable the download validation again.

abraunegg · 2025-01-17T20:46:26Z

@erenoglu
Thanks for your confirmation on the fix and your patience whilst fixing the AIP handling for the client.

abraunegg · 2025-01-25T01:47:13Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

erenoglu added the Bug Something isn't working label Jan 13, 2025

abraunegg added the Investigating label Jan 13, 2025

abraunegg changed the title ~~Bug: Upload Download Loop for New File in Monitor Mode~~ Bug: Upload Download Loop for AIP Protected File in Monitor Mode Jan 13, 2025

abraunegg added the Unable to reproduce label Jan 13, 2025

abraunegg added PR Provided - Needs Testing Waiting on Response and removed Investigating labels Jan 15, 2025

abraunegg linked a pull request Jan 17, 2025 that will close this issue

Fix Upload|Download Loop for AIP Protected File in Monitor Mode #3071

Merged

abraunegg added this to the v2.5.4 milestone Jan 17, 2025

abraunegg removed the PR Provided - Needs Testing label Jan 17, 2025

abraunegg added Fixed and removed Waiting on Response labels Jan 17, 2025

abraunegg changed the title ~~Bug: Upload Download Loop for AIP Protected File in Monitor Mode~~ Bug: Upload|Download Loop for AIP Protected File in Monitor Mode Jan 17, 2025

abraunegg closed this as completed in #3071 Jan 17, 2025

Repository owner locked as resolved and limited conversation to collaborators Jan 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug: Upload|Download Loop for AIP Protected File in Monitor Mode #3070

Bug: Upload|Download Loop for AIP Protected File in Monitor Mode #3070

erenoglu commented Jan 13, 2025 •

edited

Loading

erenoglu commented Jan 13, 2025

abraunegg commented Jan 13, 2025

erenoglu commented Jan 13, 2025

abraunegg commented Jan 13, 2025

abraunegg commented Jan 14, 2025 •

edited

Loading

erenoglu commented Jan 14, 2025 •

edited

Loading

abraunegg commented Jan 14, 2025

erenoglu commented Jan 14, 2025

abraunegg commented Jan 14, 2025

erenoglu commented Jan 14, 2025

erenoglu commented Jan 14, 2025

abraunegg commented Jan 14, 2025

erenoglu commented Jan 14, 2025

abraunegg commented Jan 15, 2025

abraunegg commented Jan 16, 2025

erenoglu commented Jan 17, 2025 •

edited

Loading

abraunegg commented Jan 17, 2025

erenoglu commented Jan 17, 2025

erenoglu commented Jan 17, 2025

abraunegg commented Jan 17, 2025

abraunegg commented Jan 17, 2025 •

edited

Loading

erenoglu commented Jan 17, 2025 •

edited

Loading

abraunegg commented Jan 17, 2025

abraunegg commented Jan 25, 2025

Bug: Upload|Download Loop for AIP Protected File in Monitor Mode #3070

Bug: Upload|Download Loop for AIP Protected File in Monitor Mode #3070

Comments

erenoglu commented Jan 13, 2025 • edited Loading

Describe the bug

Operating System Details

Client Installation Method

OneDrive Account Type

What is your OneDrive Application Version

What is your OneDrive Application Configuration

What is your 'curl' version

Where is your 'sync_dir' located

What are all your system 'mount points'

What are all your local file system partition types

How do you use 'onedrive'

Steps to reproduce the behaviour

Complete Verbose Log Output

Screenshots

Other Log Information or Details

Additional context

erenoglu commented Jan 13, 2025

abraunegg commented Jan 13, 2025

erenoglu commented Jan 13, 2025

abraunegg commented Jan 13, 2025

abraunegg commented Jan 14, 2025 • edited Loading

From #3057

From #3066

erenoglu commented Jan 14, 2025 • edited Loading

abraunegg commented Jan 14, 2025

erenoglu commented Jan 14, 2025

abraunegg commented Jan 14, 2025

erenoglu commented Jan 14, 2025

erenoglu commented Jan 14, 2025

abraunegg commented Jan 14, 2025

erenoglu commented Jan 14, 2025

abraunegg commented Jan 15, 2025

abraunegg commented Jan 16, 2025

erenoglu commented Jan 17, 2025 • edited Loading

abraunegg commented Jan 17, 2025

erenoglu commented Jan 17, 2025

erenoglu commented Jan 17, 2025

abraunegg commented Jan 17, 2025

abraunegg commented Jan 17, 2025 • edited Loading

erenoglu commented Jan 17, 2025 • edited Loading

abraunegg commented Jan 17, 2025

abraunegg commented Jan 25, 2025

erenoglu commented Jan 13, 2025 •

edited

Loading

abraunegg commented Jan 14, 2025 •

edited

Loading

erenoglu commented Jan 14, 2025 •

edited

Loading

erenoglu commented Jan 17, 2025 •

edited

Loading

abraunegg commented Jan 17, 2025 •

edited

Loading

erenoglu commented Jan 17, 2025 •

edited

Loading