Handle non-UTF8 files

[kirawi]

Not sure what's causing the newlines, I'm guessing it's because of \n within the PDF. Definitely still buggy overall (see below).

Resolves #18

[kirawi]

This is the plan for this PR:

• Provide the encoding and fileencoding commands inspired by Vim. I am unaware of their equivalents in kakoune.
• Support encoding + decoding for writing and reading respectively.
• Display the file encoding on the status line.
• Automatically detect encoding.

@archseer You said this, #18 (comment)

So far I've made no effort to support anything other than UTF-8. I think moving forward we could use encoding_rs + hsivonen/chardetng for encoding detection, but I'd probably maintain a whitelist of encodings we'd allow. I've gotten hit by esoteric edge cases before where the encoding was under-specified.

AFAIK, encoding_rs only supports encodings specified by the encoding standard. Do you still want a whitelist?

[kirawi]

Haha, oops. The buffer seems glitched right now. I think I may need to drop the while loop with just a loop loop.

[archseer]

By the way, there's also https://docs.rs/encoding_rs_io/0.1.7/encoding_rs_io/ by burntsushi. Let's look at how that's doing the reads internally, there might be some patterns we're able to copy.

[kirawi]

[kirawi]

By the way, there's also https://docs.rs/encoding_rs_io/0.1.7/encoding_rs_io/ by burntsushi. Let's look at how that's doing the reads internally, there might be some patterns we're able to copy.

I took a quick look, but I don't think there's anything from it that would improve the code here significantly.

[kirawi]

I removed the files for the benchmark for now so it's easier to review.

[kirawi]

Last thing left to do is commands and the status bar.

[kirawi]

I think the encoding had this thing where it'll change encoding on the fly. Can someone double check that? If so, do we want to disable that if encoding is manually selected?

[archseer]

What are you testing this with? Make sure to download some Shift-JIS encoded files.

[kirawi]

Tentatively, I'd consider the functionality fully implemented. I'll be working on getting benchmarks and exhaustive tests before this PR is ready to be merged.

This bug should be fixed at some point, but it would require large changes to rendering:

The text I'm editing on line 131 is rendered at line 128.

Edit:
I realized I forgot to re-decode the actual text when the encoding command is used.

[kirawi]

Got this error while trying to set encoding of helix-core/src/lib.rs to UTF-16. It happens when you scroll horizontally past the viewport.

     Running `target\debug\hx.exe`
thread 'main' panicked at 'Trying to access position outside the buffer: x=362, y=0, area=Rect { x: 0, y: 0, width: 120, height: 30 }', helix-tui\src\buffer.rs:217:9
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\/library\std\src\panicking.rs:515
   1: std::panicking::begin_panic_fmt
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\/library\std\src\panicking.rs:457
   2: helix_tui::buffer::Buffer::index_of
             at .\helix-tui\src\buffer.rs:217
   3: helix_tui::buffer::Buffer::get_mut
             at .\helix-tui\src\buffer.rs:184
   4: helix_tui::buffer::Buffer::set_style
             at .\helix-tui\src\buffer.rs:358
   5: helix_term::ui::editor::EditorView::render_buffer
             at .\helix-term\src\ui\editor.rs:320
   6: helix_term::ui::editor::EditorView::render_view
             at .\helix-term\src\ui\editor.rs:69
   7: helix_term::ui::editor::{{impl}}::render
             at .\helix-term\src\ui\editor.rs:709
   8: helix_term::compositor::Compositor::render
             at .\helix-term\src\compositor.rs:137
   9: helix_term::application::Application::render
             at .\helix-term\src\application.rs:92
  10: helix_term::application::Application::handle_terminal_events
             at .\helix-term\src\application.rs:151
  11: helix_term::application::{{impl}}::event_loop::{{closure}}
             at .\helix-term\src\application.rs:109
  12: core::future::from_generator::{{impl}}::poll<generator-0>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\future\mod.rs:80
  13: helix_term::application::{{impl}}::run::{{closure}}
             at .\helix-term\src\application.rs:331
  14: core::future::from_generator::{{impl}}::poll<generator-0>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\future\mod.rs:80
  15: hx::main::{{closure}}
             at .\helix-term\src\main.rs:96
  16: core::future::from_generator::{{impl}}::poll<generator-0>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\future\mod.rs:80
  17: tokio::park::thread::{{impl}}::block_on::{{closure}}<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\park\thread.rs:263
  18: tokio::coop::with_budget::{{closure}}<enum$<core::task::poll::Poll<enum$<core::result::Result<tuple<>, anyhow::Error>, 1, 18446744073709551615, Err>>>,closure-0>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\coop.rs:106
  19: std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::try_with<core::cell::Cell<tokio::coop::Budget>,closure-0,enum$<core::task::poll::Poll<enum$<core::result::Result<tuple<>, anyhow::Error>, 1, 18446744073709551615, Err>>>>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\std\src\thread\local.rs:400
  20: std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::with<core::cell::Cell<tokio::coop::Budget>,closure-0,enum$<core::task::poll::Poll<enum$<core::result::Result<tuple<>, anyhow::Error>, 1, 18446744073709551615, Err>>>>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\std\src\thread\local.rs:376
  21: tokio::coop::with_budget
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\coop.rs:99
  22: tokio::coop::budget
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\coop.rs:76
  23: tokio::park::thread::CachedParkThread::block_on<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\park\thread.rs:263
  24: tokio::runtime::enter::Enter::block_on<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\runtime\enter.rs:151
  25: tokio::runtime::thread_pool::ThreadPool::block_on<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\runtime\thread_pool\mod.rs:71
  26: tokio::runtime::Runtime::block_on<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\runtime\mod.rs:452
  27: hx::main
             at .\helix-term\src\main.rs:98
  28: core::ops::function::FnOnce::call_once<fn() -> enum$<core::result::Result<tuple<>, anyhow::Error>, 1, 18446744073709551615, Err>,tuple<>>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\ops\function.rs:227
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[kirawi]

When trying to insert a character into UTF-16. Probably also need to encode on insert.

thread 'main' panicked at 'byte index 5 is not a char boundary; it is inside '㩤' (bytes 3..6) of `瑳㩤攺癮㨺慶⡲䠢䱅塉剟乕䥔䕍⤢笠 †††爠瑥牵楤⹲湩潴⤨਻††੽ †挠湯瑳删彔䥄㩒☠瑳⁲‽爢湵楴敭㬢 †氠瑥挠湯彦楤⁲‽潣普杩摟物⤨樮楯⡮呒䑟剉㬩 †椠⁦潣普摟物攮楸瑳`[...]', helix-core\src\syntax.rs:503:29
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\/library\std\src\panicking.rs:515
   1: core::panicking::panic_fmt
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\/library\core\src\panicking.rs:92
   2: core::fmt::Arguments::new_v1
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\/library\core\src\fmt\mod.rs:338
   3: core::str::slice_error_fail
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\/library\core\src\str\mod.rs:114
   4: core::str::traits::{{impl}}::index
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\str\traits.rs:365
   5: core::str::traits::{{impl}}::index<core::ops::range::RangeFrom<usize>>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\str\traits.rs:64
   6: helix_core::syntax::{{impl}}::parse::{{closure}}
             at .\helix-core\src\syntax.rs:503
   7: core::ops::function::impls::{{impl}}::call_mut<tuple<usize, tree_sitter::Point>,closure-1>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\ops\function.rs:269
   8: tree_sitter::{{impl}}::parse_with::read<slice<u8>,closure-1>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tree-sitter-0.19.5\binding_rust\lib.rs:451
   9: ts_lexer__get_chunk
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tree-sitter-0.19.5\src\lexer.c:59
  10: ts_lexer_start
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tree-sitter-0.19.5\src\lexer.c:306
  11: ts_parser__lex
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tree-sitter-0.19.5\src\parser.c:423
  12: ts_parser__advance
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tree-sitter-0.19.5\src\parser.c:1407
  13: ts_parser_parse
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tree-sitter-0.19.5\src\parser.c:1890
  14: tree_sitter::Parser::parse_with<slice<u8>,closure-1>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tree-sitter-0.19.5\binding_rust\lib.rs:465
  15: helix_core::syntax::LanguageLayer::parse
             at .\helix-core\src\syntax.rs:497
  16: helix_core::syntax::LanguageLayer::update
             at .\helix-core\src\syntax.rs:685
  17: helix_core::syntax::{{impl}}::update::{{closure}}
             at .\helix-core\src\syntax.rs:342
  18: std::thread::local::LocalKey<core::cell::RefCell<helix_core::syntax::TsParser>>::try_with<core::cell::RefCell<helix_core::syntax::TsParser>,closure-0,enum$<core::result::Result<tuple<>, enum$<helix_core::syntax::Error>>, 0, 2, Err>>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\std\src\thread\local.rs:400
  19: std::thread::local::LocalKey<core::cell::RefCell<helix_core::syntax::TsParser>>::with<core::cell::RefCell<helix_core::syntax::TsParser>,closure-0,enum$<core::result::Result<tuple<>, enum$<helix_core::syntax::Error>>, 0, 2, Err>>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\std\src\thread\local.rs:376
  20: helix_core::syntax::Syntax::update
             at .\helix-core\src\syntax.rs:341
  21: helix_view::document::Document::_apply
             at .\helix-view\src\document.rs:552
  22: helix_view::document::Document::apply
             at .\helix-view\src\document.rs:594
  23: helix_term::commands::insert::insert_char
             at .\helix-term\src\commands.rs:2042
  24: helix_term::ui::editor::EditorView::insert_mode
             at .\helix-term\src\ui\editor.rs:537
  25: helix_term::ui::editor::{{impl}}::handle_event
             at .\helix-term\src\ui\editor.rs:646
  26: helix_term::compositor::Compositor::handle_event
             at .\helix-term\src\compositor.rs:112
  27: helix_term::application::Application::handle_terminal_events
             at .\helix-term\src\application.rs:145
  28: helix_term::application::{{impl}}::event_loop::{{closure}}
             at .\helix-term\src\application.rs:109
  29: core::future::from_generator::{{impl}}::poll<generator-0>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\future\mod.rs:80
  30: helix_term::application::{{impl}}::run::{{closure}}
             at .\helix-term\src\application.rs:331
  31: core::future::from_generator::{{impl}}::poll<generator-0>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\future\mod.rs:80
  32: hx::main::{{closure}}
             at .\helix-term\src\main.rs:96
  33: core::future::from_generator::{{impl}}::poll<generator-0>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\future\mod.rs:80
  34: tokio::park::thread::{{impl}}::block_on::{{closure}}<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\park\thread.rs:263
  35: tokio::coop::with_budget::{{closure}}<enum$<core::task::poll::Poll<enum$<core::result::Result<tuple<>, anyhow::Error>, 1, 18446744073709551615, Err>>>,closure-0>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\coop.rs:106
  36: std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::try_with<core::cell::Cell<tokio::coop::Budget>,closure-0,enum$<core::task::poll::Poll<enum$<core::result::Result<tuple<>, anyhow::Error>, 1, 18446744073709551615, Err>>>>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\std\src\thread\local.rs:400
  37: std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::with<core::cell::Cell<tokio::coop::Budget>,closure-0,enum$<core::task::poll::Poll<enum$<core::result::Result<tuple<>, anyhow::Error>, 1, 18446744073709551615, Err>>>>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\std\src\thread\local.rs:376
  38: tokio::coop::with_budget
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\coop.rs:99
  39: tokio::coop::budget
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\coop.rs:76
  40: tokio::park::thread::CachedParkThread::block_on<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\park\thread.rs:263
  41: tokio::runtime::enter::Enter::block_on<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\runtime\enter.rs:151
  42: tokio::runtime::thread_pool::ThreadPool::block_on<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\runtime\thread_pool\mod.rs:71
  43: tokio::runtime::Runtime::block_on<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\runtime\mod.rs:452
  44: hx::main
             at .\helix-term\src\main.rs:98
  45: core::ops::function::FnOnce::call_once<fn() -> enum$<core::result::Result<tuple<>, anyhow::Error>, 1, 18446744073709551615, Err>,tuple<>>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\ops\function.rs:227
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: process didn't exit successfully: `target\debug\hx.exe` (exit code: 101)

[kirawi]

thread 'main' panicked at 'assertion failed: self.len_after == other.len', helix-core\src\transaction.rs:128:9
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\/library\std\src\panicking.rs:515
   1: core::panicking::panic_fmt
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\/library\core\src\panicking.rs:92
   2: core::panicking::panic
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\/library\core\src\panicking.rs:50
   3: helix_core::transaction::ChangeSet::compose
             at .\helix-core\src\transaction.rs:128
   4: helix_view::document::{{impl}}::apply::{{closure}}
             at .\helix-view\src\document.rs:605
   5: helix_view::document::take_with::{{closure}}<helix_core::transaction::ChangeSet,closure-0>
             at .\helix-view\src\document.rs:91
   6: std::panic::{{impl}}::call_once<helix_core::transaction::ChangeSet,closure-0>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\std\src\panic.rs:347
   7: std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure-0>,helix_core::transaction::ChangeSet>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\std\src\panicking.rs:401
   8: std::panicking::try::do_catch<std::panic::AssertUnwindSafe<closure-0>,tuple<>>
   9: std::panicking::try<helix_core::transaction::ChangeSet,std::panic::AssertUnwindSafe<closure-0>>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\std\src\panicking.rs:365
  10: std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure-0>,helix_core::transaction::ChangeSet>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\std\src\panic.rs:434
  11: helix_view::document::take_with<helix_core::transaction::ChangeSet,closure-0>
             at .\helix-view\src\document.rs:91
  12: helix_view::document::Document::apply
             at .\helix-view\src\document.rs:604
  13: helix_term::commands::insert::insert_char
             at .\helix-term\src\commands.rs:2042
  14: helix_term::ui::editor::EditorView::insert_mode
             at .\helix-term\src\ui\editor.rs:537
  15: helix_term::ui::editor::{{impl}}::handle_event
             at .\helix-term\src\ui\editor.rs:646
  16: helix_term::compositor::Compositor::handle_event
             at .\helix-term\src\compositor.rs:112
  17: helix_term::application::Application::handle_terminal_events
             at .\helix-term\src\application.rs:145
  18: helix_term::application::{{impl}}::event_loop::{{closure}}
             at .\helix-term\src\application.rs:109
  19: core::future::from_generator::{{impl}}::poll<generator-0>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\future\mod.rs:80
  20: helix_term::application::{{impl}}::run::{{closure}}
             at .\helix-term\src\application.rs:331
  21: core::future::from_generator::{{impl}}::poll<generator-0>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\future\mod.rs:80
  22: hx::main::{{closure}}
             at .\helix-term\src\main.rs:96
  23: core::future::from_generator::{{impl}}::poll<generator-0>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\future\mod.rs:80
  24: tokio::park::thread::{{impl}}::block_on::{{closure}}<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\park\thread.rs:263
  25: tokio::coop::with_budget::{{closure}}<enum$<core::task::poll::Poll<enum$<core::result::Result<tuple<>, anyhow::Error>, 1, 18446744073709551615, Err>>>,closure-0>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\coop.rs:106
  26: std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::try_with<core::cell::Cell<tokio::coop::Budget>,closure-0,enum$<core::task::poll::Poll<enum$<core::result::Result<tuple<>, anyhow::Error>, 1, 18446744073709551615, Err>>>>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\std\src\thread\local.rs:400
  27: std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::with<core::cell::Cell<tokio::coop::Budget>,closure-0,enum$<core::task::poll::Poll<enum$<core::result::Result<tuple<>, anyhow::Error>, 1, 18446744073709551615, Err>>>>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\std\src\thread\local.rs:376
  28: tokio::coop::with_budget
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\coop.rs:99
  29: tokio::coop::budget
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\coop.rs:76
  30: tokio::park::thread::CachedParkThread::block_on<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\park\thread.rs:263
  31: tokio::runtime::enter::Enter::block_on<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\runtime\enter.rs:151
  32: tokio::runtime::thread_pool::ThreadPool::block_on<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\runtime\thread_pool\mod.rs:71
  33: tokio::runtime::Runtime::block_on<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\there\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\tokio-1.6.1\src\runtime\mod.rs:452
  34: hx::main
             at .\helix-term\src\main.rs:98
  35: core::ops::function::FnOnce::call_once<fn() -> enum$<core::result::Result<tuple<>, anyhow::Error>, 1, 18446744073709551615, Err>,tuple<>>
             at /rustc/0a8629bff642c3c3b84bb644c0099194f063b627\library\core\src\ops\function.rs:227

[archseer]

When trying to insert a character into UTF-16. Probably also need to encode on insert.

The in-memory representation is always UTF-8 s and we always want to insert UTF-8 encoded text. I don't think you're finding encoding specific behavior, but rather bugs with our UTF-8 handling.

[archseer]

To clarify: the only valid rope representation is UTF-8. You need to decode from other encodings to UTF-8 on file load, and re-encode to other encodings on file save. So _set_encoding really shouldn't re-encode the string on the fly, it simply sets the encoding variable and modifies what encoding we re-encode to on file save.

[kirawi]

I'll be moving the reload functionality into a separate PR once this PR is more or less complete.

[kirawi]

At some point, we should find the cause for this error: thread 'main' panicked at 'internal error: entered unreachable code', helix-core\src\graphemes.rs:210:22. But since it seems to only happen on binary files, I'll defer it to another PR at some point in the future.

[archseer]

Just some minor comments this round, great work 👍

[cessen]

@kirawi Your latest commit message says "wip", but after your explanation this looks feature complete to me. I still have minor concerns about code readability, but I don't want to block merging on it.

What is it that you're still working on?

Edit: nevermind, brain fart. Didn't check to see if there were new commits since the last time I looked. Ha ha.

[kirawi]

@kirawi Your latest comment message says "wip", but after your explanation this looks feature complete to me. I still have minor concerns about code readability, but I don't want to block merging on it.

What is it that you're still working on?

Edit: nevermind, brain fart. Didn't check to see if there were new commits since the last time I looked. Ha ha.

There appears to be a bug with line indentation in Document::open, which caused the merge commit to fail tests. I'll be debugging that and returning to the nested loops tomorrow.

[cessen]

I believe I've fixed the bug that was causing the merge commit tests to fail. It was totally my fault, from a different PR.

So hopefully all that's left is to switch back to the nested loop construction (if you still want to--I won't block merging on that), and fix the small merge conflict that my own fix introduced.

[archseer]

Great work!

Addressed