[Bug]: Some posts not federating, unable to be manually synced

[TeaHands]

Requirements

• Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support
• Did you check to see if this issue already exists?
• Is this only a single bug? Do not put multiple bugs in one issue.
• Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

I'm aware of the ongoing federation issues in general, and have been using the workaround of searching for specific post / comment URLs to force sync with my home instance. However in the last couple of days I've noticed a new variant of post that doesn't federate to lemmy.world specifically, and is also unable to be manually synced. See various examples here: https://lemmy.world/post/1090999

Not currently sure whether this is affecting all instances and is a Lemmy-wide issue of some kind (perhaps due to instances involved all running on different versions?), or if this is a lemmy.world specific bug that shouldn't be reported here. Apologies if it turns out to be the latter.

Steps to Reproduce

1. See that https://lemm.ee/post/703295 is a post that exists
2. Visit https://lemmy.world/c/[email protected] and see that the post is not federated
3. Attempt to manually search for https://lemm.ee/post/703295 from lemmy.world to force sync, = no results

Technical Details

The three problem posts so far originate from three different instances. Versions numbers are:

BE: 0.18.1-rc.4-48-gf5378d0b4-kt.1 (lemmy.blahaj.zone)
BE: 0.18.1-rc.10 (lemm.ee)
BE: 0.18.0 (reddthat.com)

Version

BE: 0.18.1-rc.9-14-ge71951309

Lemmy Instance URL

lemmy.world

[wereii]

Moved from different issue as this sounds exactly like the same problem, except that I did get partial sync for some posts and then some posts do not load/sync at all and posts are missing from communities.

UI: 0.18.1-rc.11 (image.revision: 02198f10)
BE: 0.18.1-rc.10 (image.revision: 22cdcc6)

I first loaded this post about an hour and something13hours ago, as of writing it has not synced yet.

• https://biglemmowski.win/post/23552 (1 vote, 0 comments)
• https://lemmy.world/post/936166 (original post, 8 votes, 1 comment)

Trying a different other instance yields the same thing:

• https://biglemmowski.win/post/23182 (first loaded about 12h ago, sitting at 0v. 0c.)
• https://lemm.ee/post/737999 (original post, instance has same UI/BE versions, 9v. 7c.)

Restarting services does not help it, purging made it worse as what was synced before is now gone.

Making a post from my instance on another does not seem to have this syncing problem - https://biglemmowski.win/post/23160

Edit:
Tried to downgrade to rc.4.
Manually putting links to comments to this post https://biglemmowski.win/post/23182 into search made the instance load them (without votes).

Edit2:
The only weird logs I get are these, not sure if it has to do with anything here.

lemmy-lemmy-1     | 2023-07-07T10:22:30.009487Z  INFO actix_web::middleware::logger: 54.36.178.108 'POST /inbox HTTP/1.0' 200 0 '-' 'Lemmy/0.18.1-rc.10; +https://lemmy.ml' 0.100993
lemmy-lemmy-1     | 2023-07-07T10:22:30.053759Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: data did not match any variant of untagged enum AnnouncableActivities
lemmy-lemmy-1     |    0: lemmy_apub::activities::community::announce::receive
lemmy-lemmy-1     |              at crates/apub/src/activities/community/announce.rs:46
lemmy-lemmy-1     |    1: lemmy_server::root_span_builder::HTTP request
lemmy-lemmy-1     |            with http.method=POST http.scheme="https" http.host=biglemmowski.win http.target=/inbox otel.kind="server" request_id=2e8ef58a-0911-4bb6-8b72-538b33c4a317 http.status_code=400 otel.status_code="OK"
lemmy-lemmy-1     |              at src/root_span_builder.rs:16
lemmy-lemmy-1     | LemmyError { message: None, inner: data did not match any variant of untagged enum AnnouncableActivities, context: SpanTrace [{ target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 46 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "\u{1b}[3mhttp.method\u{1b}[0m\u{1b}[2m=\u{1b}[0mPOST \u{1b}[3mhttp.scheme\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"https\" \u{1b}[3mhttp.host\u{1b}[0m\u{1b}[2m=\u{1b}[0mbiglemmowski.win \u{1b}[3mhttp.target\u{1b}[0m\u{1b}[2m=\u{1b}[0m/inbox \u{1b}[3motel.kind\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"server\" \u{1b}[3mrequest_id\u{1b}[0m\u{1b}[2m=\u{1b}[0m2e8ef58a-0911-4bb6-8b72-538b33c4a317 \u{1b}[3mhttp.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m400 \u{1b}[3motel.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"OK\"", file: "src/root_span_builder.rs", line: 16 }] }
lemmy-lemmy-1     | 2023-07-07T10:22:30.053836Z  INFO actix_web::middleware::logger: 54.36.178.108 'POST /inbox HTTP/1.0' 400 69 '-' 'Lemmy/0.18.1-rc.10; +https://lemmy.ml' 0.002368

lemmy-lemmy-1     | 2023-07-07T10:29:00.842905Z  INFO actix_web::middleware::logger: 34.199.116.47 'POST /inbox HTTP/1.0' 200 0 '-' 'Lemmy/0.18.1-rc.4-48-gf5378d0b4-kt.1; +https://lemmy.blahaj.zone' 1.978456
lemmy-lemmy-1     | 2023-07-07T10:29:01.165167Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: data did not match any variant of untagged enum AnnouncableActivities
lemmy-lemmy-1     |    0: lemmy_apub::activities::community::announce::receive
lemmy-lemmy-1     |              at crates/apub/src/activities/community/announce.rs:46
lemmy-lemmy-1     |    1: lemmy_server::root_span_builder::HTTP request
lemmy-lemmy-1     |            with http.method=POST http.scheme="https" http.host=biglemmowski.win http.target=/inbox otel.kind="server" request_id=65bea767-1690-4cec-aeac-672f25c3ac86 http.status_code=400 otel.status_code="OK"
lemmy-lemmy-1     |              at src/root_span_builder.rs:16
lemmy-lemmy-1     | LemmyError { message: None, inner: data did not match any variant of untagged enum AnnouncableActivities, context: SpanTrace [{ target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 46 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "\u{1b}[3mhttp.method\u{1b}[0m\u{1b}[2m=\u{1b}[0mPOST \u{1b}[3mhttp.scheme\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"https\" \u{1b}[3mhttp.host\u{1b}[0m\u{1b}[2m=\u{1b}[0mbiglemmowski.win \u{1b}[3mhttp.target\u{1b}[0m\u{1b}[2m=\u{1b}[0m/inbox \u{1b}[3motel.kind\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"server\" \u{1b}[3mrequest_id\u{1b}[0m\u{1b}[2m=\u{1b}[0m65bea767-1690-4cec-aeac-672f25c3ac86 \u{1b}[3mhttp.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m400 \u{1b}[3motel.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"OK\"", file: "src/root_span_builder.rs", line: 16 }] }

[wereii]

I did not notice the post I was trying to force sync is not local to that instance but is from a different one so disregard this.

---

With 0.18.1

When I try forcefully syncing a post that has not been visited yet I get no results (in the search) and always get this warning :

lemmy-lemmy-1     | 2023-07-07T15:09:48.437520Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: couldnt_find_object: Other errors which are not explicitly handled
lemmy-lemmy-1     |    0: lemmy_apub::fetcher::search::search_query_to_object_id
lemmy-lemmy-1     |              at crates/apub/src/fetcher/search.rs:19
lemmy-lemmy-1     |    1: lemmy_apub::api::resolve_object::resolve_object
lemmy-lemmy-1     |            with data=Query(ResolveObject { q: "https://lemmy.ml/post/1776178", auth: Sensitive })
lemmy-lemmy-1     |              at crates/apub/src/api/resolve_object.rs:15
lemmy-lemmy-1     |    2: lemmy_server::root_span_builder::HTTP request
lemmy-lemmy-1     |            with http.method=GET http.scheme="https" http.host=biglemmowski.win http.target=/api/v3/resolve_object otel.kind="server" request_id=fbcd5e90-a73d-4855-9d12-25c72186a8aa http.status_code=400 otel.status_code="OK"
lemmy-lemmy-1     |              at src/root_span_builder.rs:16
lemmy-lemmy-1     | LemmyError { message: Some("couldnt_find_object"), inner: Other errors which are not explicitly handled, context: SpanTrace [{ target: "lemmy_apub::fetcher::search", name: "search_query_to_object_id", file: "crates/apub/src/fetcher/search.rs", line: 19 }, { target: "lemmy_apub::api::resolve_object", name: "resolve_object", fields: "\u{1b}[3mdata\u{1b}[0m\u{1b}[2m=\u{1b}[0mQuery(ResolveObject { q: \"https://lemmy.ml/post/1776178\", auth: Sensitive })", file: "crates/apub/src/api/resolve_object.rs", line: 15 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "\u{1b}[3mhttp.method\u{1b}[0m\u{1b}[2m=\u{1b}[0mGET \u{1b}[3mhttp.scheme\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"https\" \u{1b}[3mhttp.host\u{1b}[0m\u{1b}[2m=\u{1b}[0mbiglemmowski.win \u{1b}[3mhttp.target\u{1b}[0m\u{1b}[2m=\u{1b}[0m/api/v3/resolve_object \u{1b}[3motel.kind\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"server\" \u{1b}[3mrequest_id\u{1b}[0m\u{1b}[2m=\u{1b}[0mfbcd5e90-a73d-4855-9d12-25c72186a8aa \u{1b}[3mhttp.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m400 \u{1b}[3motel.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"OK\"", file: "src/root_span_builder.rs", line: 16 }] }

This is for any post, tried https://lemmy.ml/post/1776178
Searching a comment link from that post also does not find it.

When force loading a community I do get a result but different warning got logged (tried loading ![email protected] which is on 0.18.0)

lemmy-lemmy-1     | 2023-07-07T15:14:02.103199Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: data did not match any variant of untagged enum AnnouncableActivities
lemmy-lemmy-1     |    0: lemmy_apub::activities::community::announce::receive
lemmy-lemmy-1     |              at crates/apub/src/activities/community/announce.rs:46
lemmy-lemmy-1     |    1: lemmy_server::root_span_builder::HTTP request
lemmy-lemmy-1     |            with http.method=POST http.scheme="https" http.host=biglemmowski.win http.target=/inbox otel.kind="server" request_id=ef5ead56-0066-48c9-89ea-29064c4a788c http.status_code=400 otel.status_code="OK"
lemmy-lemmy-1     |              at src/root_span_builder.rs:16
lemmy-lemmy-1     | LemmyError { message: None, inner: data did not match any variant of untagged enum AnnouncableActivities, context: SpanTrace [{ target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 46 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "\u{1b}[3mhttp.method\u{1b}[0m\u{1b}[2m=\u{1b}[0mPOST \u{1b}[3mhttp.scheme\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"https\" \u{1b}[3mhttp.host\u{1b}[0m\u{1b}[2m=\u{1b}[0mbiglemmowski.win \u{1b}[3mhttp.target\u{1b}[0m\u{1b}[2m=\u{1b}[0m/inbox \u{1b}[3motel.kind\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"server\" \u{1b}[3mrequest_id\u{1b}[0m\u{1b}[2m=\u{1b}[0mef5ead56-0066-48c9-89ea-29064c4a788c \u{1b}[3mhttp.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m400 \u{1b}[3motel.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"OK\"", file: "src/root_span_builder.rs", line: 16 }] }

[IAmCaligvla]

I think this might be related to what I'm experiencing as well. From both sh.itjust.works and lemmy.dbzer0.com I can't see any posts, comments or anything on certain kbin magazines e.g. [email protected]. If you access https://sh.itjust.works/c/[email protected] or https://lemmy.dbzer0.com/c/[email protected] you'll see that they're missing all the comments and posts.

I've managed to force the servers to sync specific posts from other magazines by redirecting from a post on kbin to either sh.itjust.works or lemmy.dbzer0.com via userscripts, but the comments are not synced in turn.

[SnipeAT]

i'm having a similar issue in that certain communities posts are populating at all. i have deleted and reinstalled my instance with the same user name. does this have anything to do with certificates of the ?

[TeaHands]

So after at least two of the servers involved upgraded to 0.18.1, this morning I found that all posts mentioned in my original issue are now able to be manually synced. I'm guessing this was indeed an incompatibility between the different versions?

Something to keep an eye on though, if anyone can figure out the technical reason for it, as reducing engagement on things posted elsewhere AND making existing remote communities look emptier from the pov of .world users is not doing us any favours in terms of decentralisation efforts.

@IAmCaligvla and @SnipeAT I'm not sure if your issues are the same. There are federation problems in general but are you able to manually search and sync posts that way? If so this was a different issue.

@wereii If you were in fact having the same issue, the upgrade might fix it!

[wereii]

I am at 0.18.1 and these are still stuck unsynced, putting them into search does not sync them

--

• https://biglemmowski.win/post/25272
• https://lemmy.world/post/936166

--

• https://biglemmowski.win/post/25178
• https://lemm.ee/post/737999

Both other instances are at 0.18.1 too.
I could try purging the posts later.
E: Purged, updated the links. Still nothing.

[wereii]

I will have to request a reopen of this issue as I can't resolve it.

Checked my nginx config many times as well as instance settings, even a bit of code around the search function.

My current assumptions

If I put a remote post link into the search bar of a local instance then the local instance:
a) returns if already found locally
b) fetch from remote if not
?) does it re-fetch/sync/check-in with the other instance if the post is already locally available and update the local, or otherwise, once a post is synced it is never directly fetched again and the only way it updates is thru subscribed updates