S3 sync issues in 2.6

[azazzel]

after update to 2.6.9. on macOS Monterey, the sync stoped working, pressing sync doesn't do anything, no errors, no nothing

[Daeraxa]

Can you at least provide details as to what sync target you are using and provide the log files as requested on the issue template?

[azazzel]

S3 for sync target. Found the issue, after update the S3 region was blank. It's back to normal after inputing the region

[lanyun1103]

S3 for sync target. Found the issue, after update the S3 region was blank. It's back to normal after inputing the region

Oh, I thought that this option does not need to be filled in, I just fill in one and can use it, thanks

[chenkehao1998]

I faced the same problem.
I have downgrade the version to 2.5 by update the data of sqlite and reinstall it.

[chenkehao1998]

The config of s3 is differernt in version 2.6 which require region,I added it,but the sync problem is still existed.
I found a mistake in development mode.

But I am not good at js . So I don't know what makes this error occured.

[chenkehao1998]

The same problem occured in windows 10, too. Not just at macos

[leeh11]

Same issue occured.

I used Oracle Cloud Object Storage with Amazon S3 Compatibility API. The old Joplin version worked fine.

I filled the region, but the problem is still existed.

[laurent22]

Please let us know what URL you use to connect to S3. We need the complete URL to replicate the issue (you can replace the domain by example.com though).

@leematos, perhaps you would know what might be the issue?

[chenkehao1998]

my URL is s3-cn-east-2.qiniucs.com
I used Qiniu Cloud Object Storage with Amazon S3 Compatibility API. The oldder Joplin version 2.5.12 worked fine,too

[leeh11]

Please let us know what URL you use to connect to S3. We need the complete URL to replicate the issue (you can replace the domain by example.com though).

@leematos, perhaps you would know what might be the issue?

My URL is like this：mynamespace.compat.objectstorage.us-phoenix-1.oraclecloud.com

[laurent22]

Could try adding "https://" in front of the URL to see if the validation passes?

[leeh11]

Could try adding "https://" in front of the URL to see if the validation passes?

It worked! But a new problem arises:
Hostname/IP does not match certificate's altnames: Host: joplin.mynamespace.compat.objectstorage.ap-seoul-1.oraclecloud.com. is not in the cert's altnames: DNS:swiftobjectstorage.ap-seoul-1.oraclecloud.com (Code ERR_TLS_CERT_ALTNAME_INVALID)

[chenkehao1998]

Thank u very much! My problem has solved.
I tried http:// before. But it did't worked. However,https is really valid.
And I didn't face the problem leeh11 said. Maybe,it's different between qiniu cloud and oracle.

[laurent22]

It worked! But a new problem arises: Hostname/IP does not match certificate's altnames: Host: joplin.mynamespace.compat.objectstorage.ap-seoul-1.oraclecloud.com. is not in the cert's altnames: DNS:swiftobjectstorage.ap-seoul-1.oraclecloud.com (Code ERR_TLS_CERT_ALTNAME_INVALID)

@leeh11, if you type the URL directly in a browser, does it work? Or perhaps this service only supports plain http (not https)?

[leeh11]

It worked! But a new problem arises: Hostname/IP does not match certificate's altnames: Host: joplin.mynamespace.compat.objectstorage.ap-seoul-1.oraclecloud.com. is not in the cert's altnames: DNS:swiftobjectstorage.ap-seoul-1.oraclecloud.com (Code ERR_TLS_CERT_ALTNAME_INVALID)

@leeh11, if you type the URL directly in a browser, does it work? Or perhaps this service only supports plain http (not https)?

I tried both http and https in browser, no response.
Let me check the oracle cloud documentation to see if I need to add a certificate some where.

[anemyte]

@leeh11 I'm using OCI as well. Adding region from the endpoint URL resolved the issue for me (at least on Linux). Here is my settings:

[leeh11]

@leeh11 I'm using OCI as well. Adding region from the endpoint URL resolved the issue for me (at least on Linux). Here is my settings:

I have the same config, but not worked on macOS and iOS(old version worked fine). If you have a iOS device please test it, tks.

[tessus]

Afaik, the region is part of the S3 specs. However, maybe there's a way to retrieve and use a default region, if one is set for a user. Maybe @leematos can answer this, since he probably know the SDK and S3 API better.

Regarding the endpoint: IMO the schema always belongs to the URL, even though an unencrypted endpoint makes not much sense, except for testing S3 locally. Thus automatically prefixing the URL with https:// is not a good idea, unless it is only done, iff the URL does not start with a schema.

[hoonnn]

Same problem if you are using MinIO.

[tessus]

@hoonnn can you be a bit more specific? does it work when you enter a proper URL with scheme and a region?

[hoonnn]

@tessus Same error on android and mac.
If you downgrade to v2.5.12 with the same settings, it is normal.

tks🙂

[up-umop]

I'm also experiencing this issue on Windows 10 and mobile using the AWS S3 (beta) sync target. The sync button is present but doesn't do anything in Windows, and the sync errors on mobile.

Note: 👆 before the update the AWS region was blank. I supplied a value based on the troubleshooting tips above.

Note: 👆I originally supplied us-west-2 in the config, but changed it based on this message. I get this same error after supplying us-east-1 and saving.

[laurent22]

I'm going to guess it's caching some auth token and not clearing it properly. Maybe you'll have to delete the app and reinstall it.

[leematos]

I'm the Dev that worked on this.

To anyone still having problems, attempt to toggle forcePathStyle on/off and do a test sync. See if that changes your behavior.

@up-umop In your screenshot i see "Success! Sync appears to be correct." which means on that device your settings are working and your client was able to auth and connect to your S3 provider. Use those settings on other clients and you should be successful.

---

I'll look to add OCI to our examples in the docs based on what the users above found.

[up-umop]

@up-umop In your screenshot i see "Success! Sync appears to be correct." which means on that device your settings are working and your client was able to auth and connect to your S3 provider. Use those settings on other clients and you should be successful.

Correct! Based on @laurent22's comment I was able to clear my mobile app cache, which allowed the verified credentials to work correctly. Once that went though, I updated the region in the Windows desktop app which again began to successfully sync.

Thanks!

[aflip]

Update: The sync works, I can see the objects created in s3. But the sync message has this error attached to it. The data seems to be all synced. have ##ed out identifiable info.

Last error: redirect location header missing at: https://s3.amazonaws.com/##/info.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=###ap-south-1%2Fs3%2Faws4_request&X-Amz-Date=20211227T162017Z&X-Amz-Expires=3600&X-Amz-Signature=####-Amz-SignedHeaders=host&x-id=GetObject

[leematos]

@aflip thanks. I recall seeing a variant of this in my testing. Are you using force path style toggled on or off? Can you try the opposite of whatever you are using?

basically S3 exposes buckets in 2 ways (I can't remember off the top of my head the specific differences) but that toggle switches how requests are sent.

[aflip]

@leematos I had it toggled on, and then i sync'd again with it toggled off, and the sync happens, but the error remains.

[leematos]

@aflip would it be possible for you to make a test bucket I could use to see if I can generate this error to further debug?

my email is find (dot) creativity (at) githubusername (dot) com

[shinglyu]

@laurent22 sorry I was compiling the code in the wrong way so my changes were not picked up. I fixed it and created the PR #5923

[aflip]

@leematos I restarted the app a few times and now the error has disappeared. Thank you very much for the support. Also, i don't really know how to make a test bucket. I made this bucket etc. following a guide I found online, step by step. [SOLVED]

[TencentSZ]

I also encountered the same issue after updating to 2.6.10. I use Baidu Cloud as an alternative to Amazon S3. I filled a string into 'AWS region' which I think is relative to the region.

I do not feel this step makes any sense. Anyway, my problem seems solved.
I agree with Shinglyu, a good error message like "The region is missing, please set it in the configuration screen" would be sufficient.

[sbkg0002]

Results in a successfull test:(sorry Dutch)

Succes! Configuratie van synchronisatie lijkt correct te zijn.

But sync results in an error:

Laatst gemelde fout: <?xml version="1.0" encoding="UTF-8"?><Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'sos-de-fra-1' is wrong; expecting 'de-fra-1'.</Message><Region>de-fra-1</Region></Error>

Setting the URL to https://sos.exo.io results in the same error.

Version: Joplin 2.6.10 (prod, darwin)

[leematos]

@sbkg0002 -- can you try toggling force path style on/off and trying a test sync and a real sync?

Did you ever attempt to set the region to sos-de-fra-1? I'm wondering if somewhere that setting is saved and hasn't been overridden based on your successful test sync.

[vichbb]

Joplin cos（tencent Cloud Object Storage,Storage object service compatible with S3 API) synchronization directory level error:
in mac: a have b c,when sync to windows:a,b,c

[leematos]

@vichbb -- I'm not sure I understand. Are you saying one folder won't sync? Can you share logs?

[vichbb]

@vichbb -- I'm not sure I understand. Are you saying one folder won't sync? Can you share logs?

There is no error, just a problem with the folder level, just like this, I create a new notebook in MAC: MyNote, and then create two new notebooks under MyNote: Note1 and Note2. Note1 has a note testA, note2 also has a note testB. Now the two note levels are myNote/Note1 /testA and myNote/Note2 /testB. But when I fully synced into cosine of s3. Then use Windows to synchronize again and become Note1 /testA and Note2 /testB with an empty notebook myNote. Now the hierarchy is Note1 and Note2 are not in MyNote, they're in the same layer.

[leematos]

@vichbb thanks for this info. Let me do a test real quick:

• Parent Notebook
  • Child Notebook 1
    • Note 1
  • Child Notebook 2
    • Note 2

[vichbb]

@vichbb thanks for this info. Let me do a test real quick:

• Parent Notebook

  • Child Notebook 1

    • Note 1

  • Child Notebook 2

    • Note 2

Yes, that's it!

[leematos]

@vichbb what version of joplin are you using on Windows?

I just ran this test on Mac/iOS with both on latest and I did not see the effect you are talking about. Steps I followed:

On Mac:

make notebook p1.
Inside of p1 make notebook c1. Inside of c1 make note note-c1.
Inside of p1 make notebook c2. inside of c2 make note note-c2.
sync.

On iOS
make new note in c1 note-c1-2.
make new note in c2 note-c2-2.
sync.

On ios
edit notes.
sync.

The folder structure remained.

At this point I think it's either an issue with Tencent COS or something outside of this syncing change, or 2.5 and 2.6 versions of joplin making different structures.

[sbkg0002]

@sbkg0002 -- can you try toggling force path style on/off and trying a test sync and a real sync?

Did you ever attempt to set the region to sos-de-fra-1? I'm wondering if somewhere that setting is saved and hasn't been overridden based on your successful test sync.

Not that I can remember. It keeps bugging me with that message about sos-de-fra-1, while I didn't use that as a regio. Any ideas on how to fix it?

[vichbb]

@vichbb what version of joplin are you using on Windows?

I just ran this test on Mac/iOS with both on latest and I did not see the effect you are talking about. Steps I followed:

On Mac:

make notebook p1. Inside of p1 make notebook c1. Inside of c1 make note note-c1. Inside of p1 make notebook c2. inside of c2 make note note-c2. sync.

On iOS make new note in c1 note-c1-2. make new note in c2 note-c2-2. sync.

On ios edit notes. sync.

The folder structure remained.

At this point I think it's either an issue with Tencent COS or something outside of this syncing change, or 2.5 and 2.6 versions of joplin making different structures.

windows:2.6.10
mac:2.6.10
ios:12.6.2
I just tested it. There is no problem with the new and dragged notes. It's strange that only the previous notes have this problem.

[leematos]

@sbkg0002 and @vichbb if you can create a bucket for me to use on your platforms so I can test I should be able to take a look this week. My email is listed here:

#5875 (comment)

At the moment I haven't been able to reproduce either of your issues so I'm out of testing ideas without a repro case in hand

[leematos]

@laurent22 -- I think #5875 (comment) is a different problem than S3 syncing. Any recent changes come to mind that could be related?

[vichbb]

@sbkg0002 and @vichbb if you can create a bucket for me to use on your platforms so I can test I should be able to take a look this week. My email is listed here:

#5875 (comment)

At the moment I haven't been able to reproduce either of your issues so I'm out of testing ideas without a repro case in hand

It has been sent to you. Thank you for your inspection

[roman-r-m]

I suspect the issue @vichbb is seeing isn't related to S3. Seems that the top-level notebook failed to sync for some reason - can be a network blip or clocks going out of sync or something else.

You can try renaming the folder on the client where you have the folder (can even rename twice to keep the original name), then sync that client, and finally sync the other client.

[vichbb]

I suspect the issue @vichbb is seeing isn't related to S3. Seems that the top-level notebook failed to sync for some reason - can be a network blip or clocks going out of sync or something else.

You can try renaming the folder on the client where you have the folder (can even rename twice to keep the original name), then sync that client, and finally sync the other client.

I tried. The folder name was changed normally, but the directory level after synchronization was still wrong.

[sbkg0002]

@sbkg0002 and @vichbb if you can create a bucket for me to use on your platforms so I can test I should be able to take a look this week.

You got mail. Thanks in advance!