Add bidi / RTL support

[ahangarha]

Is your proposal related to a problem?

Considering the very nature of social media platforms (like Lemmy), people with different languages would create content. It becomes problematic when one creates content in RTL languages (like Persian, Arabic, Urdu,...). Lemmy is not ready to render RTL (or even mixed RTL/LTR) text properly.

Describe the solution you'd like

There are two approaches in general. One is through HTML modification which can be done by add dir="auto" attribute to elements containing user generated text or by wrapping it in <bdi> tags. The other approach would be through CSS by adding unicode-bidi: plaintext. In either of approaches, there should be avoidance in using absolute positioning or spacing. For example instead of text-align: left we should use text-align: start or instead of padding-left we should use padding-inline-start unless there be some strong UX reason.

Describe alternatives you've considered

--

Additional context

Just keep in mind that bidi (bidirectional text) is different from using RTL or LTR in general based on the language user choose for the interface.

I have opened this issue here not on back-end because I think this can be managed on front-end.

---

Todo

• Post Title
• Post Text (Add RTL/BIDI to post text #1963)
• Comments

[dessalines]

I have no familiarity with this, but anyone should feel free to give it a try.

[Nutomic]

I found this page which gives a lot ofdetails on how bidirectional text works: https://www.w3.org/International/articles/inline-bidi-markup/

[cadars]

Another great resource: https://rtlstyling.com/posts/rtl-styling

[ahangarha]

Another great resource: https://rtlstyling.com/posts/rtl-styling

Please notice that bidi is not about RTL. One can use RTL if it is known that the content is RTL. We are talking about a social media in which people may write text in different languages and even wrote mixed content. We don't have and we don't want to have rich text editor in which the user can specify which direction should be applied on a specific portion of the text. We want to handle this situation automatically which is called bidirectional text support.

In most of the cases, adding dir="auto" to the element which contains the text is the trick. In addition we need to use other values that absolute left|right for text alignment or padding/margin or even borders.

[cadars]

In most of the cases, adding dir="auto" to the element which contains the text is the trick.

Yes, to each element if you want true mixed content support. (And to the top element only for nested elements like ul or blockquote https://codepen.io/cadars/pen/poeXMXZ )

I did a quick review of what would be needed to fully support RTL languages in lemmy-ui:

1. Editor: add dir="auto" to all elements users can create, like @ahangarha suggested.
2. Add the proper lang attribute to the html element when switching the language. For now it's always lang="en".
3. Add dir="rtl" to the html element for RTL languages.
4. Switch to logical CSS properties:
   • Because lemmy-ui is using flexbox for layout at its core this is quite easy, see https://github.com/cadars/lemmy-ui/commit/4ded16218bd977787d9b7ea0f8d19b09a294f299 https://github.com/cadars/lemmy-ui/commit/025a058120df457f053af6a51b7a7c01902753e1
   • The hard part: most of the styling comes from the (bootstrap) themes CSS, and rewriting each theme by hand, or asking future theme writers to only use logical properties isn't really an option: a solution would be to use RTLCSS or similar(?) to convert each theme to their RTL counterpart. (of course this adds yet another dependency to the project…)

[ahangarha]

Things like RTLCSS wouldn't help as per my understanding. We need to modify certain styles. For example:

• text-align:left|right -> text-align:start|end
• padding-left|right:* -> padding-inline-start|end
• margin-left|right:* -> margin-inline-start|end
• and somehow similar properties for border,...

[cadars]

In an ideal world, yes, but lemmy-ui uses Bootstrap themes, and rewriting them is no small feat; so using a “converter”, while being a less elegant solution, sounds way more practical.

[ahangarha]

Let's say some bidifix. I don't know when I would find time to work on this, but it should easy to implement.

[dessalines]

Whoever takes this on, I want to mention that lemmy-ui uses bootstrap, and it has RTL instructions here: https://getbootstrap.com/docs/5.0/getting-started/rtl/

[ahangarha]

Whoever takes this on, I want to mention that lemmy-ui uses bootstrap, and it has RTL instructions here: https://getbootstrap.com/docs/5.0/getting-started/rtl/

Again, this is for explicitly define direction. Such solutions won't work for us. We need dynamic directions based on the content.

[jayabie]

Any update on this case guys?
We need the RTL ASAP please.

[dessalines]

PRs adding this welcome.

[ahangarha]

I checked dependencies. Lemmy uses markdown-it for handling markdown. I have success in convincing marked to add bidi support (they made a plugin for it) but still struggling with developers on markdown-it.

There are some workaround. If lemmy developers are ok with adding this feature in two steps, I think we can start working on it. We can make markdown sections to follow the first character's direction till markdown-it add bidi support or a plugin be made for it. then we can remove the relevant modification for these section since it will be fully handled by markdown parser.

If it is ok, I try to find some time to create local development environment.

[dessalines]

Sweet, let me know if you need any help setting up a development environment.

[ahangarha]

In few week I will come back again. First I try to do it on my own and if I failed, I will seek help.

[ahangarha]

I just wanted to inform you that we are working on a plugin for markdown-it which is used in this project for parsing md to html. This plugin called markdown-it-bidi and is under development. not ready to be used in production but I think it is on the track.

I decided to do this so that not only we can help lemmy to fix this issue but many more projects can benefit from. It is under LGPL which I think is very good :). So if I didn't follow up here, that was the reason. Hopefully in sometimes we can add that in this project and...

If anyone interested, please check the code and share your thoughts.

[jayabie]

@dessalines @ahangarha Friends, no progress with this yet?

[ahangarha]

@jayabie I am looking for an opportunity to open some time for it.

[jayabie]

@dessalines @ahangarha

No update my friends?

[ahangarha]

@jayabie I am going to try now.
As long as there is no nested elements (like list within list) the plugin is usable even now.

But let me try it on the actual project in development environment. If successful, I send PR. Would it be fine? I mean within 2 days I might ask to assign this issue to me. Would it be fine?

[jayabie]

Hello @ahangarha

I'm just a guy seeking for RTL, i think you should ask @dessalines if any extra permissions needed.

[Nutomic]

But let me try it on the actual project in development environment. If successful, I send PR. Would it be fine? I mean within 2 days I might ask to assign this issue to me. Would it be fine?

@ahangarha There is no rush, no one else is working on this. Just take your time and make a pull request when you are ready. Also feel free to ask if anything is unclear.

[ahangarha]

@Nutomic I have issue in building backend. How can I get help? I don't think here is a good place.

[Nutomic]

@ahangarha What error do you get? Make sure to run git submodule init && git submodule update before build. And follow the documentation.

https://join-lemmy.org/docs/en/contributing/contributing.html

[ahangarha]

This was and still is the error:

➜  lemmy git:(main) cargo check
   Compiling lemmy_utils v0.16.5 (/****/lemmy/crates/utils)
    Checking futures v0.3.21
    Checking actix-server v2.1.1
    Checking opentelemetry v0.17.0
error: failed to run custom build command for `lemmy_utils v0.16.5 (/****/lemmy/crates/utils)`

Caused by:
  process didn't exit successfully: `/****/lemmy/target/debug/build/lemmy_utils-53c738f32a60fcf3/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-changed=translations/email/en.json
  cargo:rerun-if-changed=translations/email/pt.json
  cargo:rerun-if-changed=translations/email/ko.json
  cargo:rerun-if-changed=translations/email/fi.json

  --- stderr
  Error: Fmt(Os { code: 2, kind: NotFound, message: "No such file or directory" })
warning: build failed, waiting for other jobs to finish...
error: build failed

It seems it cannot get translations.

[Nutomic]

So you are missing the submodule. Did you run the commands from my previous comment? And what does git status show?

[ahangarha]

Yes. I ran.

I ran clone with --recursive and also I ran the commands you gave me. Nothing happened. see:

➜  lemmy git:(main) git submodule init && git submodule update
➜  lemmy git:(main)
➜  lemmy git:(main) git submodule status                      
 3f86b5c40796fa83054e2226e36effff3b93198a crates/utils/translations (remotes/origin/lemmy-ml-registration-message-252-g3f86b5c)

[dessalines]

We also saw issues with that when people were running old versions of rust, or not properly using rustup to get new rust versions.

If the files are there, make sure you're at least on rust 1.60.

What does rustc --version say?

[dessalines]

Apparently they paid someone to code this, and keep the source code private. They would have done much better to pay someone with RTL experience to update lemmy-ui itself, then they could still get code updates. As it they'll be stuck on old versions of lemmy-ui.

[jayabie]

Hello guys @dessalines @Nutomic ,

should i move with the signup, or we wait for the response to @Nutomic ?

Yes, after a further check found they didn't share their code.

[dessalines]

Lets wait for them to respond to nutomic first.

[xximj]

Hi guys
Sorry to let you wait so long .. The code is available here :
https://github.com/xximj/dwanya
Works nice with v0.16.6.

I wish i've seen this conversation before working on the code , very interesting information shared by @ahangarha. Thanks for all your efforts guys.

[dessalines]

Thanks! Again I personally don't have time to work on this, but hopefully that code can be helpful to someone who does.

[jayabie]

No update made to this? we need the RTL from lemmy`s master branch very much so we can get the updates.

[Nutomic]

If you want this to be implemented, you can work on it yourself (or hire someone to work on it), and then make a pull request. We would definitely like to merge this functionality.

[jayabie]

No progress made on this @dessalines @Nutomic ?

[dessalines]

No. Read the comments above.

[jayabie]

I couldn't find anyone to make it this is why i'm asking.

[xximj]

Hey @jayabie
Feel free to say hi at @mjxtb:matrix.org - we can team up on this.

[jayabie]

@xximj Thanks for your invitation, i will join soon.

[lorem10]

Hi guys, we had a draft PR that was supposed to work on other sections last week. is it in progress, I also want to contribute to this case

[hamidrezakp]

Hi guys, we had a draft PR that was supposed to work on other sections last week. is it in progress, I also want to contribute to this case

Hey, do you mean #1963 or something else you were working on?
Anyways, we love to see progress on this part, especially on comments section.

Last week i have done some testing with comments section and it's kinda impossible to make HTML uls bidirectional. only one direction can be chosen.(maybe i wasn't trying hard).

@dessalines
When can we expect the new UI to be ready?
Is it worth it to make effort and fix this UI?

[ahangarha]

For lists, only we can add direction to ul and not to li. Don't waste time on fixing issues with li.

[hamidrezakp]

For lists, only we can add direction to ul and not to li. Don't waste time on fixing issues with li.

Yup, i've tested it, and if we can make it happen, it's gonna look ugly.
For comments section, we should avoid using ul.

[ahangarha]

May you share some screenshots of what you see as ugly?

[hamidrezakp]

May you share some screenshots of what you see as ugly?

I couldn't make it happen, it was hypothetically.
one comment on left and another on right, inside chain of replies, should be hard to follow.

[ahangarha]

That is the nature of the issue. It happens the same if we do it here. See:

این متن فارسی است و در به صورت یک متن RTL با تراز راست‌چین نمایش داده می‌شود.

[hamidrezakp]

Any updates on this?
@ahangarha did you get to work on comments section?
At least we can merge the parts that is working(#1963)

[ahangarha]

I haven't worked on it yet.

Since the current partial solution doesn't harm the project, I would say we can ask for merging #1963

For lists, I need to make a new release in markdown-it-bidi. But I don't think this is a crucial issue for now.

[hamidrezakp]

Let's have this issue open as it it not totally completed, but partially.
The comments section needs BIDI too.

[ahangarha]

@hamidrezakp Can you make a checklist of what is done and what are remaining?

[hamidrezakp]

@hamidrezakp Can you make a checklist of what is done and what are remaining?

Sure:

• Post Title
• Post Text (Add RTL/BIDI to post text #1963)
• Comments

Although we should discuss if we want to have post title be bidi even if users layout is different than title direction (e.g. User with en_us language opens a Persian post with Persian title).

Other than post title, imo we just need comments to be bidi for now.

And please update your first comment and mention current status.

[ahangarha]

if we want to have post title be bidi even if users layout is different than title direction
We should have bidi by I think what you mean is that if we should enforce text alignment with left|right rather than start.

This is important but it is a design decision.

[Nutomic]

I just submitted a backend PR which allows Arabic (and Cyrillic) in usernames and community names: LemmyNet/lemmy#4083