Cannot copy or move message that is not synchronized with the server (IMAP)

[Brackmeister]

Steps to reproduce:
Use "archive" for a message when you don't have an IMAP archive folder on the server.

Expected behaviour:
I only hit that function by accident. I have no idea what it is supposed to do.
But now that the email message is moved to a (local) archive folder, I expect to be able to move it back to the IMAP folder it was in before.

What happens instead:
When I select copy or move I get the message "Cannot copy or move message that is not synchronized with the server" (I use German language so that might not be the exact English phrase)
When I delete the message it goes to the Trash folder, but from there I cannot copy/move it either.
Effectively this email is kind of lost.

This is similar to a bug[*] in the old Google Code tracker at https://code.google.com/p/k9mail/issues/detail?id=861 which was never resolved.

[*]: I consider this a bug as there is no way of undoing this, no confirmation is required, but an IMAP message "archived" in this way is essentially gone for good

[diegobersanetti]

I confirm the existence of this bug, it happened to be also by accident: I did not mean to delete a message but it happened, and I did not have the Trash folder pointing correctly to the IMAP folder yet, so such message is buried in this local Trash folder and I cannot recover it.

[LongmanU]

This has been annoying me for years now. Accidently I use a gesture that moves a message to the Trash (so I think - I don't know what I do, but it is in Trash afterwards), and I cannot get it back.

So, when it worked this way, why doesn't it work the way back? I have to forward the mail to my own address each time this happens.

Please, please, make it work!

[bcraigie]

Me too. I found that messages sent from my phone (imap account) were being stored in the local sent folder of the phone and not the imap INBOX.Sent folder. I can't move these to the correct Sent folder. The whole point of imap is you see the same folders and emails everywhere. I found the setting to change the destination folders, but cannot move the locally sent ones.

A fix for this would be good. It should be easy to code the fix to move an email from a local folder to an imap folder. :-)

[vigna]

Same here. Sometimes I trash accidentally a message and there's no way to get it back to the inbox except by forwarding.

[kripper]

This bug is still present.
Messages are getting "lost" so this bug should be high prio.
I'm not sure if I triggered by accident a gesture, but some messages just disappear and are only visible in the "All messages" option.

[potyt]

What a horrible bug. How can is sit here untouched for 2+ years? Messages can be lost by an accidental glance of a finger, and therr is no way back. Forwarding is not a suitable solution to this either. I'm amazed k9 people aren't all over this.

[LongmanU]

Messages can be moved from Trash back to Inbox after synchronizing the Trash Folder (swype from top to bottom / drag message list down). But I'm still not happy about silently trashing mails by an unwanted swype action.

[EnviroEllen]

Adding my plea for a fix to this behavior.

I use K9 as a POP3 client. I believe that all of my K9 folders are local to my phone, except when I initiate a sync action. Why shouldn't I be able to move a message out of my [local] Trash folder into another local folder?

[tadam98]

I also have this problem on my Android phone. I cannot move messages between synchronized folders.

[Adambean]

Every time. When I see this message I just think "well sodding synchronise it then!?". This should only ever happen when you're offline.

[economicpygmy]

I can confirm this annoying bug is still present. I now have zombie messages I cant delete along with a folder. I didn't even want to archive, hit the button by mistake.

[dopefishh]

Every time. When I see this message I just think "well sodding synchronise it then!?". This should only ever happen when you're offline.

Well, sometimes it is not possible to synchronize. For example, I have an extra account setup just for sending. Email sent with that account appears in the Sent box for that account, but the IMAP information is bogus since I only use it for sending.
There are many other reasons why you might not be able to synchronize. Maybe your server is down, your quotum is reached, etc.

[rkoe]

So, this bug now exists ~10 years (with the previous bugreport on code.google.com), and nobody cares to fix it, and there's not even a workaroud. WTF? Seriously? Hello K-Mail-developers?

Throwing mails away (or locking them unmovable, as done here my K-9 Mails) is a no-go, which always must be prevented and fixed as soon as possible. Otherwise, the mail client is unusable.
But nothing happens here, and nobody cares.

Apparently, the developers don't give a shit about their users and their mails. That's sad, because otherwise K-9 Mail would be a nice app.

[Adambean]

Well, sometimes it is not possible to synchronize. For example, I have an extra account setup just for sending. Email sent with that account appears in the Sent box for that account, but the IMAP information is bogus since I only use it for sending.
There are many other reasons why you might not be able to synchronize. Maybe your server is down, your quotum is reached, etc.

I get that there will be situations where syncing isn't possible, but I don't think my situation quite falls into those you mentioned.

• I don't have any send-only accounts. -- IMAP details all correct.
• Server is definitely up, as is network connectivity. (Happens on Wi-Fi and 3G/4G equally.)
• No or controllable quotas on my own mail servers.

This impacts my own mail servers on the same network as well as other ones in data centres equally, and the problem is very easily reproducible. -- Each time I can start up Thunderbird on a PC in the same network and manage messages without any problem.

[rkoe]

Since the developers unfortunately don't care, I've tried to get some workarounds:

1. Forward the mail to yourself. Then, store the message and put it into your mail-directory again.
   Only stores the plain text; and if you can manually add mails to your IMAP-server;
   ignores attachments, formatting, html, mail headers.

2. Forward the mails as attachment to yourself. Then, store the attached file and put it into your mail-directory again.
   Works with simple plain-text mails (incl. mail header) and if you can manually add mails to your IMAP-server;
   does not work with html-mails and attachments but instead creates an empty mail.

3. Tell K-9 Mail to re-insert the mails via IMAP to the server, via raw access to the K-9 Mail database.
   Works with plain text mails and html mails; and probably only on rooted phones;
   but "deletes" attachments (attachments still exist, but with size=0), modifies the mail-header and maybe re-encodes the mail.

   Instructions (use with care, and always make backups before!):

   • close K-9 Mail (and probably stop background sync)
   • run Termux, and in Termux:

pkg install sqlite
su
cd /data/data/com.fsck.k9/databases
mkdir _backup
cp -a * _backup

/data/data/com.termux/files/usr/bin/sqlite3 *.db

# list all "local" mails
SELECT folder_id, uid FROM messages WHERE uid LIKE "K9LOCAL:%";
# tell K-9 Mail to add the messages to the IMAP server
# (probably has to be adapted if you're using several IMAP servers)
INSERT INTO pending_commands (command, data)
SELECT 'append', '{"databaseId":0,"folder":"' || name || '","uid":"' || uid || '"}' FROM
(SELECT (SELECT name FROM folders WHERE id=messages.folder_id) AS name, uid FROM messages WHERE uid LIKE "K9LOCAL:%");
# list all pending "add mail to server" operations
SELECT * FROM pending_commands;

• start K-9 Mail
• sync mails

---

Unfortunately, none of the workarounds really works well, so:
To the K-9 Mail developers: Please fix this or at least add a warning/confirmation dialog when a user tries to move a mail to a local folder (=before the uid in the messages-table is set to "K9LOCAL:...").

[LongmanU]

Did you also try to simply swype down when viewing the message list in the local folder? My understanding is, that the folder then is being synchronized with the server. After that, you should be able to to move the message back to the original folder in the normal way.

Since someone wrote that a few years ago in this thread, I was successful with it and hadn't got any more problems with this issue. But I'm not sure if this works if you're offline - will have to wait until a network connection is available, I think.

[dopefishh]

This workaround only works if you can sync to begin with.

[rkoe]

Did you also try to simply swype down when viewing the message list in the local folder?

Yes.

My understanding is, that the folder then is being synchronized with the server.

The folder is then synchronized -- except all "local" messages, which are not synchronized.
So, this does not help.

After that, you should be able to to move the message back to the original folder in the normal way.

No. As soon as K-9 Mail marks the mails as "local" (="K9LOCAL:..." in the uid-column of the messages-table in the SQLite-database), it will never sync them to the server again, and K-9 Mail even prevents moving the mails to an other folder for whatever reason (except for moving them to the Trash).

[rkoe]

This workaround only works if you can sync to begin with.

If you cannot sync with your IMAP server at all, you should check the network connection and/or account settings. The workarounds are only intended for the case of this issue -- when you can generally can sync your mails with the IMAP server, except for some mails which K-9 Mail (accidentally) stores locally; these workaround then can more or less sync these "locally stored" mails back to the IMAP server.

[dopefishh]

I cannot and do not want to sync because of what I said before: a send only mail account.

[Quibky]

Extremely frustrating, I deleted a returned mail sent to a bad address and apparently because it was in thread view it deleted the forwarded message as well. There is really no excuse for this to be a problem for 10 years.. how can I see it in my trash and not be able to move it?!

[potyt]

Moved to FairMail, after being annoyed by this for a while, and other quirks and oddities, a real breath of fresh air!

[heeplr]

Wow, almost 6 years. Not much hope but reporting anyway:

I stumbled upon this for a secondary IMAP account, that is only used to send mail on mobile, not receive. So I disabled syncing of folders years ago.
Now I noticed that sent messages were not stored on the server (of course). I learned that disabling syncing for the "sent" Folder was a bad idea. (Seemed unnecessary waste of data plan at the time and probably got the settings wrong).

This wouldn't be a big problem if I could manually sync the "Sent" messages from the phone to the server from time to time but that doesn't seem to be possible due to this bug.

[Player701]

I seem to be experiencing this issue as well, although I have set up my K9 so that all folders are supposed to be synchronized with the server ("Folders" -> "Move/copy destination folders" = "All", "Folders to display" = "All"). The problem is, just recently it seemed to have worked perfectly: moving, archiving messages, marking them as spam etc. would result in these actions being synchronized with the server as they should be. Now, however, it appears that whenever I move the message to any other folder except Trash, this action is not reflected on the server, and any further attempts to move the message (except to Trash) result in the "Cannot copy or move message..." error.

The thing is, I would synchronize it with the server if I knew how to do it, but I can't find this function anywhere. I tried swiping down in the message list, and it does seem to trigger an update of the folder's contents, but it does not actually synchronize the problematic message with the server. Restarting the app and even rebooting my device does not fix the problem. I can only make the message visible again in K9 by first deleting it on K9's side and then moving it to some other folder via another IMAP client (e.g. webmail).

This seems to be a very serious issue because it results in de-syncing of folder contents between K9 and the IMAP server, and what's worst is that it can happen unknowingly to the user (they won't notice unless they try to move the message again). All messages moved between IMAP folders should be synchronized by default, if it doesn't happen then it's a bug.

UPD: Just as I've written this, it seemed to have started working normally again. This is extremely werid. Something fishy is most certainly going on here.

[mstyp]

Echoing what others have said, this is a really serious bug. I now have 100+ emails that only exist in k9 mail that need to be re-synced back my server, I am going to have to go through and forward them to myself ONE AT A BLOODY TIME. gah.

This would not be so frustrating if It where not for the fact that I can see that this issue has been around for YEARS and gone unfixed.

[Player701]

I was able to reproduce this by switching my internet connection from Wi-Fi to 4G while K9 Mail was running (push was also enabled). The following steps were taken:

1. Rebooted the phone, K9 started up, 4G was enabled. Everything worked normally.
2. Turned off 4G and turned on Wi-Fi, everything still worked normally.
3. Turned off Wi-Fi and turned on 4G again, and K9 suddenly stopped synchronizing deleted/moved messages.
4. Switching back to Wi-Fi from 4G didn't help. But stopping K9 completely via the app manager and starting it again did, though.

I'm not sure if this is a reliable way to reproduce the issue, maybe I'll test again when I have sufficient time for that.

[gkhourii]

Hello, I would like to take a shot at this issue. I will be working with @dnicules on this as well as issue #7688 .

[gkhourii]

Update: here's the fix we've developed:

-Every time the user selects move/copy/archive/trash on a message, we sync the folder list.

Effects:
-clears out any "dead" folders (folders that no longer exist on the server but still exist locally)
-avoids the case where a user can potentially move data to a dead folder

Caveats:
-might slow down performance due to extra refreshes, particularly for accounts with many folders
-this fix only works online; can't refresh folders without a connection. At that point, if the user is confident the target folder still exists remotely, then we could allow them to send data there at their own risk, warning them before proceeding.

We're going to run more tests before submitting a PR. Please feel free to reach out for more details @Brackmeister

[Brackmeister]

I'm not very opinionated about the way this problem gets resolved. For people who actually want to archive a message it sounds reasonable. But I don't know what people expect to work offline or not.

Personally I guess I have around 100 folders but no clue how long a folder refresh would take, i.e. I can't say if I would be bothered by the time the extra refresh takes. But then, on mobile I tend to just read/write/delete emails from my inbox, moving or copying mails is something I would do on my desktop PC anyway.

I still have a small doubt: I stumbled over this problem when I accidentally hit the wrong action entry in the menu. So I could imagine it's possible to hit that additional "I'm sure that folder exists remotely, let me proceed" button by accident as well. And that would still result in a situation that can't be resolved by the user. But yeah, one would have to select the wrong action by accident and then also by accident proceed instead of canceling. Definitely an improvement I'd say.

[gkhourii]

Thanks for the feedback. We're currently implementing your suggestion of the offline warning, but we're having trouble figuring out whether or not we are online or offline (we need this info to determine whether or not to display the warning). How can we access this information from MessageViewFragment.kt and MessageListFragment.kt? @Brackmeister

[cketti]

Some background: All operations inside the app update the local database right away. Then an entry is added to the list of operations to be performed on the server (internally these are called "pending commands"). When there's network connectivity (and there are no authentication or other errors) these operations are performed right away. If not, the operations are performed at a later date. This allows managing mail offline.

Right now pending commands reference the server IDs of messages on which to perform an operation. With IMAP each message has a folder-specific server ID, i.e. if a message is moved from one folder to another, its server ID changes. For a move or copy operation the app can't know the new server ID of the message in the target folder until it has been successfully completed. This is why sometimes the "Cannot copy or move a message that is not synchronized with the server" is displayed.

An easy way to trigger this:

1. Switch device into airplane mode
2. Move a message to a different folder
3. Go to that folder
4. Try to move that message again, e.g. back to the original folder

There have been a bunch of bugs that could lead to a message accidentally ending up in a "not synchronized" state. These days most of them are fixed and the "Cannot copy or move a message that is not synchronized with the server" message should only be shown in the offline case described above. The way to resolve this right now, is to trigger processing the list of pending remote operations. Synchronizing the folder will do this. Of course this requires network connectivity.

---

The way to properly solve the problem of not being able to enqueue a pending command because the app doesn't know the server ID of a message is to not use the server ID in the list of pending commands. This was alluded to in the description of #7688.

Fixing this properly requires changing the way we store pending commands, i.e. operations that need to be performed on the server. Right now we reference the server ID of a message, not the database ID.

By using the local database ID, the app could move a message from folder A to folder B, then from folder B to folder C while being offline. We already create placeholder messages in the local database. After these local operations we would end up with three entries in the database. One (invisible) placeholder entry for the original message in folder A. This one also references the initial server ID in folder A. Then a placeholder message in folder B. This one does not have an associated server ID because the message doesn't exist on the server yet. The original local message will be in folder C. It is the full message the user can view. However, it will also not have a server ID because it doesn't exist in folder C on the server yet.

When network connectivity returns, the move operations will be performed in order. For the first move operation the app will read the server ID from the placeholder message in folder A. After a successful move on the server, the app will delete the placeholder message in folder A. The placeholder message in folder B will be updated with the new server ID in folder B. The same then repeats for the placeholder message in folder B and the message in folder C.

---

@gkhourii: This change is quite a bit of work and has a high potential to introduce new bugs. It's probably not the best project to get started as a new contributor. My suggestion would be to get to know the code base by fixing smaller bugs and working your way up to this one.

[dnicules]

Hi @cketti, I am working with @gkhourii on the issue. I see, we may leave this issue for someone with more experience with the codebase, at least for the case where a user is offline. However, we want to test our working solution for the case where a client is online; for testing purposes, what is the best way for us to verify that Thunderbird is connected to the internet and to the email server?
Thanks :)

[dnicules]

Also @Brackmeister what email client were you using that allowed you to not have a server-side archive folder?

[Brackmeister]

When I switched from POP3 to IMAP I was using the web frontend of GMX, Sylpheed on PC and K9 on Android.
I may have used the the GMX Android app at some point, but it was a long time ago so I'm not 100% sure.
Later I switched from Sylpheed to Thunderbird (but that was definitely after I had created this issue).