S3 synchronization errors after upgrade to v2.2.6

[abjrcode]

S3 synchronization stopped working after upgrade to v2.2.6 and although it worked for me flawlessly for a over a year now.

Environment

Joplin version: v2.2.6
Platform: Windows 10
OS specifics: OS Build 19043.1110

Steps to reproduce

I just try to synchronize with my existing S3 bucket in eu-central-1 which I have been using already for more than a year.
I am using the following URL for S3: https://s3.amazonaws.com/ and I tried switching to the regional endpoint https://s3.eu-central-1.amazonaws.com/ and it didn't solve it.

Describe what you expected to happen

S3 synchronization should work normally.

Logfile

• Redacted my personal information such as the bucket and API key.

FetchError: redirect location header missing at: https://s3.amazonaws.com/<MY-BUCKET-NAME>/info.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=<MY_AWS_KEY_ID>%2F20210810%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210810T205440Z&X-Amz-Expires=3600&X-Amz-Signature=a43ce1568ea1975bf1167de6748242652e2d1ca7d6bbb7518ab49315c90c69eb&X-Amz-SignedHeaders=host&x-id=GetObject
    at ClientRequest.<anonymous> (C:\Users\<MY-USER-NAME>\AppData\Local\Programs\Joplin\resources\app.asar\node_modules\@joplin\lib\node_modules\node-fetch\index.js:152:13)
    at ClientRequest.emit (events.js:310:20)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:596:27)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:119:17)
    at TLSSocket.socketOnData (_http_client.js:469:22)
    at TLSSocket.emit (events.js:310:20)
    at addChunk (_stream_readable.js:286:12)
    at readableAddChunk (_stream_readable.js:268:9)
    at TLSSocket.Readable.push (_stream_readable.js:209:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:186:23)

[laurent22]

It's most likely due to this change: #5212

@leematos does the error makes sense to you?

[plerohellec]

I just upgraded to 2.2.6 from 2.1.9 on MacOS and I also have synchronization errors but they are different. I use S3 as well.
In the sidebar it says:
Last error: SyntaxError: Unexpected token < in JSON at position 0

and in log.txt:

2021-08-10 14:07:04: Synchronizer: "Sync: starting: Starting synchronisation to target 8... supportsAccurateTimestamp = false; supportsMultiPut = false [1628629624406]"
2021-08-10 14:07:04: Synchronizer: "Indexing resources..."
2021-08-10 14:07:04: "ResourceService::indexNoteResources: Start"
2021-08-10 14:07:04: "ResourceService::indexNoteResources: Completed"
2021-08-10 14:07:06: RevisionService: "RevisionService::maintenance: Starting..."
2021-08-10 14:07:06: RevisionService: "RevisionService::maintenance: Service is enabled"
2021-08-10 14:07:07: RevisionService: "RevisionService::collectRevisions: Created revisions for 0 notes"
2021-08-10 14:07:07: RevisionService: "RevisionService::maintenance: Done in 198ms"
2021-08-10 14:07:07: Synchronizer: "SyntaxError: Unexpected token < in JSON at position 0
SyntaxError: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at MigrationHandler.<anonymous> (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/services/synchronizer/MigrationHandler.js:44:31)
    at Generator.next (<anonymous>)
    at fulfilled (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/services/synchronizer/MigrationHandler.js:5:58)"
2021-08-10 14:07:07: Synchronizer: "Sync: finished: Synchronisation finished [1628629624406]"
2021-08-10 14:07:07: Synchronizer: "Operations completed: "
2021-08-10 14:07:07: Synchronizer: "Total folders: 11"
2021-08-10 14:07:07: Synchronizer: "Total notes: 74"
2021-08-10 14:07:07: Synchronizer: "Total resources: 1"
2021-08-10 14:07:07: Synchronizer: "There was some errors:"
2021-08-10 14:07:07: Synchronizer: "SyntaxError: Unexpected token < in JSON at position 0
SyntaxError: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at MigrationHandler.<anonymous> (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/services/synchronizer/MigrationHandler.js:44:31)
    at Generator.next (<anonymous>)
    at fulfilled (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/services/synchronizer/MigrationHandler.js:5:58)"
2021-08-10 14:07:07: "Updating all notifications..."

[abjrcode]

Actually I had a similar error to your @plerohellec when I used the regional S3 endpoint.

[leematos]

Hrm.

@laurent22 are there new migrations/did the sync version change in 2.2.6?

I just upgraded myself and it works on Mac so I'm drawing a blank but happy to keep debugging.

@abjrcode @plerohellec

If you'd be willing to debug with me you can email me at:

find dot creativity at github username dot com

and I'm happy to take a look with you both as I just implemented this 😓

[leematos]

It looks like ya'll are failing here:

joplin/packages/lib/services/synchronizer/MigrationHandler.ts

Lines 41 to 47 in c3f10d3

	const syncTargetInfoText = await this.api_.get('info.json');
	// Returns version 0 if the sync target is empty
	let output: SyncTargetInfo = { version: 0 };
	if (syncTargetInfoText) {
	output = JSON.parse(syncTargetInfoText);

@plerohellec are you also using AWS directly or a different provider?

[leematos]

If we are able to sync, I'm gonna get some logging set up on your environments, and then see if we can dump the output of syncTargetInfoText to see what the actual failure is.

[plerohellec]

@leematos I'm using another provider (UpCloud) so my AWS S3 URL is https://documents.us-sjo1.upcloudobjects.com.

I wonder if it's possible that the info.json file never got created during the migration? I took a quick look at the bucket and didn't see it but I would have to paginate a lot to be sure.

Note that I also have another joplin-desktop instance running that is still on 2.1.8 and that is connected to that same bucket.

[leematos]

@plerohellec, nah, I suspect it's there as your other instance would be upset about the info.json file not existing.

If you are around and want to sync, I'm at my desk for the next hour or so.

If not, you could also email me a test bucket on that provider and I can try it in my env to see if I get the same error (I'm on a Mac.)

[plerohellec]

@leematos thanks, I sent you an email with the credentials for a test bucket at UpCloud.

[leematos]

I've connected with @plerohellec and I have a test bucket that I am looking at now.

I'm curious, while I look into that, can @plerohellec and @abjrcode can you both try a "check config" and see if that shows an error, It may get us the useful bits quickly.

[plerohellec]

This is what I get for check config:

Error. Please check that URL, username, password, etc. are correct and that the sync target is accessible. The reported error was:

and nothing after that.

[leematos]

I was able to pull out the actual error:

<?xml version="1.0" encoding="UTF-8"?> <Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region is wrong; expecting &#39;us-sjo1&#39;.</Message><Key>info.json</Key><BucketName>testing-bucket</BucketName><Resource>/testing-bucket/info.json</Resource><Region>us-sjo1</Region><RequestId>169A191402CE4517</RequestId><HostId>0477af3d-107f-4ea2-8337-54f90d3abbc2</HostId></Error>
`

@abjrcode I suspect yours is similar.

The new SDK requires a region to be set, but it should be overriding it automatically is my current understanding.

I'm gonna look into what logic is needed to get it squared away.

[leematos]

@laurent22

I think two things need to happen if we want to proceed forward:

1. More robust error handling for xml error. I'm gonna try that now just to get more vis to the errors.
2. We may need to add a setting to AWS S3 for region.

It's unclear to me if this will require me to make DB migrations? Do you have any example commits from maybe the making of joplin server that I can reference for adding a new setting for the AWS S3 target?

Fastest fix: Revert.

I'm off of my day job tomorrow so I will have some time to investigate then as well.

[HugoReeves]

Just reporting that I am also seeing the same error using the AWS S3 API for synchronization. I'm using a real S3 bucket for storage as well, not an S3 compliant endpoint.

[leematos]

Did a little more digging.

1. there are two types of XML parsing problems:
   1. Because of the challenges with the New API around getting blobs, fetch'd Pre-signed URLs don't have XML error handling. That's our problem. I'm looking into the simplest way to address that.
   2. The second, is there is a problem with AWS-SDK where it is not parsing some XML and generating a useful error message. This is why we get blank errors and I'm not sure if that's a 3rd party provider malformed XML problem, or an AWS SDK problem.

@HugoReeves would you be able to make me an S3 test bucket so I can validate if your issue is the same? My email is here: #5287 (comment)

@laurent22 I'm not sure if you can revoke/revert and roll foward the package, but I'd probably suggest it at this point to avoid further impact and I can keep looking into it.

Sorry for the trouble everyone. I was attempting to fix an iOS bug, upgrade the AWS-sdk and shrink the package size. 😓

If you go back to the prior package your syncing should work as normal. Any version below 2.2.6 should not exhibit the problem: https://github.com/laurent22/joplin/releases?after=android-v2.2.3

[leematos]

To be more specific:

• the get info.json problem looks to need region overrides to solve it, and then XML error parsing.

• The Sync config tester throwing blank may be solved by that, but could be something else.

[leematos]

I need to get to sleep, but I:

• Was able to add a region sync field and do preliminary tests on desktop. It looks to be working.
• I was able to build rudimentary XML error parsing, but would like another look at it with fresh eyes as it's def got some flaws.

At this point sync works with an UpCloud bucket. I'd like to also test and confirm with an AWS bucket if possible and then Laurent can decide how he wants to proceed.

[abjrcode]

Sorry @leematos I live in a different time zone and wasn't around to help you out but I see you got to the bottom of it.

[laurent22]

@leematos, I've set back desktop 2.2 to a pre-release so that less people are affected by the issue. So if I understand correctly it doesn't affect actual AWS but compatible implementations like UpCloud, is that correct?

I'm not sure why the info.json GET call is failing as it should be using the same AWS driver as any other call. Or maybe in those specific cases, all calls would fail anyway? (and it turns out the first one is for info.json)

If I can help with something let me know.

[abjrcode]

Hi @laurent22 I am using AWS S3 directly so I think the issue is applicable there too.

[spoit]

Hi @laurent22 I had the issue too directly connecting to AWS S3. Rolling back to 2.2.5 solved the sync issue for now

[laurent22]

I have reverted the change for now so if you're having this issue please try the latest version:

• Desktop: https://github.com/laurent22/joplin/releases/
• Android: https://github.com/laurent22/joplin-android/releases

It's not reverted on iOS and CLI yet.

[spoit]

I installed 2.2.7 and can confirm S3 sync works again (to a real s3 origin)

[leematos]

@spoit @abjrcode any way either of you could create an AWS testing bucket I could use? Email here: #5287 (comment)

I know one half of the "real AWS s3" bucket issue is the region and that's solved, but I think the second half of empty check config error output is non-aws implementations having a space before the XML Error output, and I want to confirm if AWS has the same or not.

[spoit]

@leematos I created a test bucket and mailed you the credentials

[stale]

Hey there, it looks like there has been no activity on this issue recently. Has the issue been fixed, or does it still require the community's attention? This issue may be closed if no further activity occurs. You may comment on the issue and I will leave it open. Thank you for your contributions.

[spoit]

Hi, my current version is Joplin 2.3.5 (prod, win32) and everything works. I think that commit 6927335 fixed things for me

[leeh11]

Looking forward to the new version on IOS!

[stale]

Hey there, it looks like there has been no activity on this issue recently. Has the issue been fixed, or does it still require the community's attention? This issue may be closed if no further activity occurs. You may comment on the issue and I will leave it open. Thank you for your contributions.

[RobertBohn]

I'm still having this problem. Running version 2.6.10 (prod, linux)

[leematos]

@RobertBohn -- If you could comment on #5875 I think that will be the right place to continue debugging.

Please share your S3 target and any logs that you have showing an error on that issue.

[bloodyburger]

Same issue with 2.6.10, but now it works on Windows but not on Mac.

[leematos]

@subzero355 -- If you could comment on #5875 I think that will be the right place to continue debugging.

Please share your S3 target and any logs that you have showing an error on that issue. (If it's working in one place and not the other, then I suspect your sync settings are different.)

[github-actions]

Hey there, it looks like there has been no activity on this issue recently. Has the issue been fixed, or does it still require the community's attention? This issue may be closed if no further activity occurs. You may comment on the issue and I will leave it open. Thank you for your contributions.

[zcmack]

i also started experiencing this issue on 2.6.10 (Darwin). I've been syncing with my minio server successfully up to this point. Now developer tools is throwing an error because I don't have a region set.

18:20:27: Error: Region is missing at Object.resolveRegionConfig (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/node_modules/@aws-sdk/config-resolver/dist-cjs/regionConfig/resolveRegionConfig.js:9) at new S3Client (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/node_modules/@aws-sdk/client-s3/dist-cjs/S3Client.js:20) at SyncTargetAmazonS3.api (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/SyncTargetAmazonS3.js:61) at SyncTargetAmazonS3.initFileApi (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/SyncTargetAmazonS3.js:137) at SyncTargetAmazonS3.<anonymous> (BaseSyncTarget.ts:89) at Generator.next (<anonymous>) at /Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/BaseSyncTarget.js:8 at new Promise (<anonymous>) at __awaiter (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/BaseSyncTarget.js:4) at SyncTargetAmazonS3.fileApi (BaseSyncTarget.ts:87)

I realize I was running with an unsupported option that was recently broken for sake of getting S3 out of beta. kind of a bummer though as I want to keep things self hosted.

[leematos]

@zcmack -- you should be able to set a region and in theory self-hosted minio should work. Depending on your minio config it could be gibberish (blah as a region) or meaningful (subdomain), but the S3 v3 API needs a region whereas the older client would infer it.

[github-actions]

Hey there, it looks like there has been no activity on this issue recently. Has the issue been fixed, or does it still require the community's attention? This issue may be closed if no further activity occurs. You may comment on the issue and I will leave it open. Thank you for your contributions.

[github-actions]

Closing this issue after a prolonged period of inactivity. If this issue is still present in the latest release, feel free to create a new issue with up-to-date information.